SlowGuess/NAACL_2022_OCR huggingface dataset:
dialog context representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836-3845, Florence, Italy. Association for Computational Linguistics.
Nikita Moghe, Mark Steedman, and Alexandra Birch. 2021. Cross-lingual intermediate fine-tuning improves dialogue state tracking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1137–1150, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nikola Mrkšić, Diarmuid Řeaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017a. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788, Vancouver, Canada. Association for Computational Linguistics.
Nikola Mrkšić, Ivan Vulic, Diarmuid Řeaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017b. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Linguistics, 5:309-324.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, Online. Association for Computational Linguistics.
Osman Ramadan, Paweł Budzianowski, and Milica Gašić. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 432-437, Melbourne, Australia. Association for Computational Linguistics.
Evgeniia Razumovskaia, Goran Glavas, Olga Majewska, Anna Korhonen, and Ivan Vulic. 2021. Crossing the conversational chasm: A primer on multilingual task-oriented dialogue systems. CoRR, abs/2104.08570.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online. Association for Computational Linguistics.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496-505, Vancouver, Canada. Association for Computational Linguistics.
Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building task-oriented dialogue systems for online shopping. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 4618-4625. AAAI Press.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computational Linguistics.
Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu, and Minlie Huang. 2020. CrossWOZ: A large-scale Chinese cross-domain task-oriented dialogue dataset. Transactions of the Association for Computational Linguistics, 8:281-295.
Lei Zuo, Kun Qian, Bowen Yang, and Zhou Yu. 2021. Allwoz: Towards multilingual task-oriented dialog systems for all. CoRR, abs/2112.08333.
A Annotation Guidelines: Post-editing of the Translation
1 Task Description
Multi-domain Wizard-of-Oz dataset (Multi-WOZ) (Budzianowski et al., 2018) is introduced as a fully-labeled collection of human-to-human written conversations spanning over multiple domains and topics.
Our project aims to translate the monolingual English-only MultiWOZ dataset to four linguistically diverse major world languages, each with a different script: Arabic (AR), Chinese (ZH), German (DE), and Russian (RU).
In this annotation task, we resort to the revised version 2.1 (Eric et al., 2020) and focus on the development and test portions of the English MultiWOZ 2.1 (in total of 2,000 dialogs containing a total of  $29.5\mathrm{K}$  utterances). We first automatically translate all the utterances and the annotated slot values to the four target languages, using Google Translate. Next the translated utterances and slot values (i.e., fix the translation errors) will be post-edited with manual efforts.
For this purpose, a JSON file for development or test set will be provided to each annotator. There are two tasks: (1) Fix the errors in automatic translations of translated utterances and the translated slot values. (2) Check the alignment between each translated utterance and the slot value annotations for that utterance.
2 JSON Representation
The JSON file will be structured as follows, feel free to use any JSON editor tools (e.g., JSON Editor Online) to annotate the files.
Annotation data
- dialogID: An unique ID for each dialog.
- turnID: The turn ID of the utterance in the dialog.
- services: Domain(s) of the dialog.
- utterance: English utterance from Multi-WOZ.
- SlotValues: English annotated slot values from MultiWOZ.
- transUtterance: Translated utterance from Google Translate.
- transSlotValues: Translated slot values from Google Translate.
Annotation Task
- fixTransUtterance: The revised translated utterance with manual efforts.
- fixTransSlotValues: The revised translated slot values with manual efforts.
- changedUtterance: Whether the translated utterance is changed. Annotate as 1 if the translated utterance is revised, 0 otherwise.
- changedSlotValues: Whether the translated slot values is changed. Annotate as 1 if the translated slot values are revised, 0 otherwise.
3 Annotation Example
Example 1: Name Correction and Mismatch
The following example in Chinese shows the error fixed with the translated name issue, and also the correctness of the mismatch case between the translated utterance and translated slot values.
dialogID: MUL0484.json
turnID: 6
services:train,attraction
utterance: No hold off on booking for now. Can you help me find an attraction called cineworld cinema?
slotValues: {attraction-name: cineworld cinema}
transUtterance: 目前暂无预订。您能帮我找到一个名为cineworld Cinema的景点吗？
transSlotValues: {attraction-name: Cineworld电影}
fixTransUtterance: 目前暂无预订。您能帮我找到一个名为电影世界电影院的景点吗？
fixTransSlotValues: {attraction-name: 电影世界电影院}
changedUtterance: 1
changedSlotValues: 1
Example 2: Grammatical Error
The following example in German shows the error corrected based on the grammatical issue of the translated utterance.
4 Additional Notes
There might be some cases of synonyms. For example, in Chinese 周五 and 星期五 both have the same meaning as Friday in English, also similarly in Russian regarding the weekdays. In this case, just pick the most common one and stays consistent among all the translated utterances and slot values. Besides there might be some language variations across different regions, please ignore the dialects and metaphors while fixing the translation errors.
If there are any open questions that you think are not covered in this guide, please do not hesitate to get in touch with me or post the questions on Slack, so these issues can be discussed together with other annotators and the guide can be improved.
B Annotation Guidelines: Quality Control
Multi-domain Wizard-of-Oz dataset (MultiWOZ) (Budzianowski et al., 2018) is introduced as a fully-labeled collection of human-to-human written conversations spanning over multiple domains and topics. Our project is aimed to translate the monolingual English-only MultiWOZ dataset to four linguistically diverse major world languages, each with a different script: Arabic (AR), Chinese (ZH), German (DE), and Russian (RU). In the previous annotation task, we resorted to the revised version 2.1 (Eric et al., 2020) and focused on the development and test portions of the English MultiWOZ 2.1.
According to the translation process, it was processed in two steps: we first automatically translated all the utterances and the annotated slot values to the four target languages, using Google Translate. Next the translated utterances and slot values (i.e., fix the translation errors) were post-edited with manual efforts from native speakers of each language.
Additionally, a quality assurance step is required to check the quality of the post-edited translation. For this purpose, a JSON file for a random sample 200 dialogs (100 from the development and test set each), containing 2,962 utterances in total will be provided to two annotators for each target language to judge the correctness of the translations. Each annotator has to independently answer the following questions for each translated utterance from the sample: (1) Is the utterance translation acceptable? (2) Do the translated slot values match the translated utterance?
- dialogID: An unique ID for each dialog.
- turnID: The turn ID of the utterance in the dialog.
- utterance: English utterance from Multi-WOZ.
- SlotValues: English annotated slot values from MultiWOZ.
- fixTransUtterance: The revised translated utterance with manual efforts.
- fixTransSlotValues: The revised translated slot values with manual efforts.
- UtteranceAcceptable: Is the utterance translation acceptable? Annotate as 1 if the translated utterance is acceptable, 0 otherwise.
- SlotValuesMatchAcceptable: Do the translated slot values match the translated utterance? Annotate as 1 if the translated slot values are acceptable, 0 otherwise.
- NOTE: Extra notes of judgement.
2 Annotation Example
Small grammatical errors, but still catch the meaning will be considered acceptable. However, if the whole meaning regarding the translation change, it will then be considered as not acceptable.
Example 1: Ambiguity
The following example shows the ambiguity issues regarding the translated utterance. In German, table can be translated into Tabelle as a table form or Tisch as a table for reservation. Regarding the contextual information from the utterance, the correct translation should be Tisch instead of Tabelle in this case. Therefore, the translated utterance will be considered as not acceptable, and annotated as 0.
dialogID: PMUL2464.json  
turnID: 9
utterance: Yes, Bedouin is a restaurant that serves African food in the Centre. It is in the expensive range. Would you like to book a table?
slotValues: {restaurant-name: bedouin}
fixTransUtterance: Ja, Beduinist ein Restaurant,das afrikanisches Essen im Zentrum serviert. Es liegt im teuren Bereich. Mochten Sie eine Tabelle reservieren?
fixTransSlotValues: {restaurant-name: Beduine}
UtteranceAcceptable: 0
SlotValuesMatchAcceptable: 1
The following example shows a slight grammatical issue regarding the translated utterance. This is mainly with the synonym case in Chinese, where
the place can be translated into地方 or位置, while 位置 will be more appropriate in this scenario. However, 地方 still keep the semantic meaning. Therefore, the translated utterance will be considered as acceptable, and annotated as 1. And further checking with the translated slot values, all are correct, and should be annotated as 1.
dialogID: PMUL0400.json
turnID: 12
utterance: Please book the place for 7 people at 11:30 on the same day.
slotValues: {restaurant-people: 7, restaurant-time: 11:30, restaurant-day: Monday}
fixTransUtterance: 请于当天11:30预订7人的地方。
fixTransSlotValues: {restaurant-people: 7, restaurant-time: 11:30, restaurant-day: 周一}
UtteranceAcceptable: 1
3 Additional Notes
Please ignore the slot values with "dontcare", "not mentioned" and "none", while checking the translation quality. If there are any open questions that you think are not covered in this guide, please do not hesitate to get in touch with me or post the questions on Slack, so these issues can be discussed together with other annotators and the guide can be improved.
Lang	Model	DST	RR
1%	5%	10%	50%	100%	1%	5%	10%	50%	100%
DE	TOD-XLMR	7.68	19.26	28.08	33.17	34.10	10.25	32.47	35.56	45.39	49.46
MLM on Mono-CC	13.75	25.15	34.12	38.01	38.26	34.37	42.13	43.51	49.10	52.80
TLM on OS	14.17	19.45	21.62	27.28	29.91	47.21	48.59	48.96	53.01	55.30
TLM+RS-Mono on OS	15.88	24.14	28.38	32.57	35.45	46.08	48.94	49.98	53.43	55.72
AR	TOD-XLMR	1.48	1.57	6.18	15.62	17.63	6.36	18.72	23.57	36.04	42.69
MLM on Mono-CC	4.41	5.74	7.02	14.10	17.22	28.54	31.50	32.82	41.09	44.26
TLM on OS	4.18	6.33	6.89	13.60	17.77	32.19	35.04	37.02	41.39	47.04
TLM+RS-Mono on OS	4.42	6.79	8.27	14.39	21.48	33.45	37.09	38.01	41.89	47.15
ZH	TOD-XLMR	8.63	12.55	16.40	23.45	25.49	15.69	31.10	33.22	41.97	48.14
MLM on Mono-CC	11.64	19.73	25.46	34.93	35.61	34.40	37.65	39.65	48.01	50.97
TLM on OS	11.48	17.43	21.95	28.52	32.51	38.17	42.82	42.91	49.29	51.63
TLM+RS-Mono on OS	11.63	14.90	17.97	22.81	28.84	38.45	43.71	45.27	48.50	51.81
RU	TOD-XLMR	4.34	21.89	30.01	37.58	37.61	8.90	31.31	34.51	43.33	47.45
MLM on Mono-CC	12.70	16.56	19.45	24.58	25.90	37.43	42.80	46.19	52.43	53.73
TLM on OS	12.45	14.26	16.10	21.13	27.04	42.23	44.40	44.78	49.43	53.76
TLM+RS-Mono on OS	13.74	17.44	18.63	24.33	29.15	41.97	45.44	46.02	49.90	53.16
Table 6: Full per-language few-shot cross-lingual transfer results for Dialog State Tracking and Response Retrieval. Results shown for different sizes of the training data in the target-language (i.e., different number of shots):  $1\%$ ,  $5\%$ ,  $10\%$ ,  $50\%$  and  $100\%$  of the MULTI $^2$ WOZ development sets (of respective target languages).