SlowGuess/NAACL_2022_OCR huggingface dataset:
Monolingual fine-tuning The pretrained model is tuned for a downstream task using the training data for a specific language and evaluated on the test set for the same language.
Multilingual fine-tuning The pretrained model is jointly fine-tuned with data in all languages for a specific task. Different from the monolingual fine-tuning setting, there is only one model for each downstream task, which can serve all languages.
Cross-lingual generation Since MTG is multiway parallel, it can be reorganized to create input-output pairs that belong to different languages. In this paper, we make use of the multiway parallel data to do the supervised cross-lingual training, e.g., for English centric cross-lingual training, we take the English source as the input and the parallel German, French, Spanish, Chinese target as the out
put. Then we evaluate the model on same setting (en->de, en->es, en->fr, en->zh). The cross-lingual generation performances on all  $5*4$  directions are evaluated.
Zero-shot transfer We also try to explore the zero-shot ability of multilingual pretrained models on the four tasks. The model is fine-tuned on a specific task with English input and output. Then it is used to generate output in other languages with a given language tag.
5.1 Monolingual and Cross-lingual
This section displays the monolingual and cross-lingual model comparison to explore their performances in different tasks and languages. Figure 1 contains the five language-centric cross-lingual and monolingual results. Several conclusions can be drawn from the results:
The performance of Cross-lingual is better than monolingual in some cases. As shown in Figure 1, model performances on ensemble scores in cross-lingual setting exceed those in monolingual setting frequently (e.g., the monolingual result of French underperforms the English to French cross-lingual result in Figure 1(b)). This is because the cross-lingual models are trained with more data (e.g., the English centric cross-lingual model is trained with en->de, en->fr, en->es, en->zh data), and the data from different cross-lingual directions can sometimes benefit from each other thus improving the model performance.
Chinese text generation is challenging in cross-lingual setting. As illustrated in Figure 1, nearly all models obtain inferior scores when generating Chinese text. Also, model results on Chinese inputs are usually worse than results on inputs in other languages. The wide discrepancies in grammar and vocabulary between Chinese and other languages lead to the poor performance of cross-lingual generation when either the target language or source language is Chinese.
Multilingual pretrained models obtain lower scores on the Summarization task. Compared with other tasks, summarization task requires longer output, which increases the difficulty of text generation, thus causing poor performance both in cross-lingual and monolingual settings.
Figure 2: The different stage performances averaged across five languages of XLM in summarization under various settings. Here stage1 represents models trained only on rough training data while stage2 represents models further trained on human-annotated training data based on models in stage1.
5.2 Monolingual and Multilingual
In addition to cross-lingual analysis, we also explore the performance difference between models trained in monolingual and multilingual settings. Table 4 displays the monolingual and multilingual training results for four models in four tasks.
In most cases, multilingual training can improve model performance on relevance. As shown in Table 4, 75 out of 96 multilingual results outperform the monolingual counterparts on various relevance metrics in different tasks. The reason is that the multilingual data in MTG is fully parallel across all five languages and every sample has semantically aligned counterparts in other languages. It makes better semantic fusion among different languages, thus boosting the multilingual training performance.
The advantages of multilingual training are not obvious on diversity measured by distinct-1. Especially in the story generation task, 3 out of 4 models obtain better distinct-1 scores in monolingual setting than in multilingual one. Diversity can not be improved by semantic sharing across languages especially when the samples of them are multiway parallel. This is because the multiway parallel dataset with the semantic aligned samples repeating in different languages encourages models to generate similar texts to some extent.
5.3 Zero-shot results
To test the cross-lingual generation ability of multilingual pretrained models when no direct cross-lingual training data are provided, we evaluate the zero-shot cross-lingual generation performance.
Table 5 presents the zero-shot results for XLM in four tasks. It demonstrates that the multilin
Task	Language	BLEU	ROUGE-1	ROUGE-2	ROUGE-L	METEOR	BERTScore	Distinct-1	Distinct-2	Ensemble
SG	en->de	0.02/3.20	7.20/27.20	0.20/4.00	7.20/25.80	0.05/0.14	0.63/0.73	0.47/0.96	0.50/1.00	18.90/29.70
en->fr	0.02/4.23	5.90/28.10	0.20/6.30	5.90/26.40	0.04/0.20	0.63/0.74	0.38/0.95	0.41/0.99	14.30/27.70
en->es	0.09/3.38	8.70/26.30	0.40/4.60	8.50/24.80	0.04/0.14	0.65/0.74	0.52/0.96	0.55/0.99	16.90/28.40
en->zh	0.00/5.79	0.00/28.80	0.00/8.80	0.00/26.80	-	0.45/0.67	0.61/0.99	0.57/0.34	16.60/26.70
QG	en->de	1.96/10.41	18.10/38.70	2.40/14.70	17.60/37.20	0.10/0.25	0.73/0.78	0.94/0.97	0.98/1.00	29.80/29.30
en->fr	2.16/14.70	16.80/42.80	2.90/19.00	16.20/39.60	0.08/0.35	0.74/0.80	0.94/0.95	0.99/0.99	28.60/29.80
en->es	7.46/16.93	25.50/49.50	8.70/22.40	23.90/46.80	0.18/0.37	0.76/0.83	0.94/0.95	0.99/1.00	28.50/29.10
en->zh	0.00/16.07	0.00/43.10	0.00/22.90	0.00/37.90	-	0.44/0.73	0.10/1.00	0.08/1.00	16.40/28.60
TG	en->de	2.58/9.15	13.40/26.90	4.40/11.10	12.50/24.30	0.12/0.22	0.67/0.73	0.83/0.95	0.88/0.99	26.30/30.60
en->fr	3.26/11.54	13.90/33.80	4.50/14.70	12.70/29.00	0.12/0.30	0.69/0.75	0.89/0.91	0.93/0.99	25.20/28.50
en->es	4.90/12.45	21.20/36.30	7.40/15.70	18.50/31.10	0.17/0.31	0.71/0.76	0.88/0.91	0.94/0.99	24.50/29.50
en->zh	0.01/15.44	0.00/34.50	0.00/19.40	0.00/29.90	-	0.45/0.69	0.37/0.98	0.22/0.58	16.70/27.10
Summ	en->de	1.85/8.36	15.40/34.70	2.90/11.70	14.50/31.10	0.08/0.20	0.65/0.72	0.61/0.81	0.78/0.97	18.50/21.50
en->fr	1.29/11.79	13.70/39.90	2.60/15.80	13.00/35.50	0.07/0.29	0.68/0.75	0.64/0.75	0.82/0.94	18.60/20.30
en->es	4.18/11.93	22.50/41.00	5.80/15.60	20.30/36.60	0.14/0.29	0.69/0.75	0.64/0.74	0.82/0.95	17.30/20.70
en->zh	0.00/14.58	0.00/42.20	0.00/20.40	0.00/38.70	-	0.42/0.71	0.68/0.84	0.27/0.94	12.80/19.60
gual pretrained model XLM still lacks the ability to generate high-quality cross-lingual output in zero-shot scenario. Moreover, English to Chinese and French zero-shot generation shows inferior performance. $^{9}$  The performance decline is rather salient when generating Chinese text. This is because Chinese and French (especially Chinese) are distant from English in the language family tree. On the other hand, zero-shot results underperform cross-lingual results which further emphasizes the importance of direct cross-lingual training data for cross-lingual text generation.
5.4 Pseudo and Annotated Data
To answer the question "Does the 400k annotated training data help the model generate better?", we use the rough training data filtered by back translation for the first stage fine-tuning and the annotated training data for the second stage. The ablation study results on the two-step fine-tuning in summarization under all evaluation scenarios with XLM are illustrated in Figure 2.
The extra human-annotated data boost model performance by at least  $3.8\%$  on the ensemble metric. We also make a T-test and prove that the improvement of annotated training data is significant in all settings.[10] It demonstrates that although the number of annotated data is small, it can significantly improve the performance. It also highlights the necessity of human-annotated multilingual data compared with pseudo-parallel data via machine translation.
Table 5: English centric zero-shot and cross-lingual results for XLM on four tasks. Scores on the left and right side of each cell represent the zero-shot and cross-lingual results respectively.
Setting	Model	Gram.	Flu.	Rel.	lang fuse	task spec.
SG	mono	4.69	4.81	3.75	1.00	3.79
multi	4.71	4.80	3.67	1.00	4.02
cross	4.18	4.23	3.49	0.95	2.53
zero	4.15	4.18	3.27	0.18	3.00
QG	mono	4.66	4.69	3.03	0.99	3.95
multi	4.69	4.67	3.06	0.97	4.11
cross	4.30	4.30	2.70	0.95	2.64
zero	3.35	4.26	3.18	0.19	3.09
TG	mono	4.53	4.51	3.09	0.96	3.71
multi	4.66	4.65	3.18	0.93	3.17
cross	3.73	3.64	2.63	0.90	1.85
zero	3.52	4.15	3.51	0.18	1.43
Summ	mono	4.19	3.99	3.71	0.68	3.71
multi	4.19	4.02	3.78	0.64	3.60
cross	2.14	2.22	2.23	0.68	2.05
zero	1.57	1.54	1.58	0.03	1.59
Table 6: Human evaluation scores averaged on five languages for mBART on four tasks. 'Gram', 'Flu', 'Rel', 'Lang Fu', 'Task Spec.' indicates Grammar, Fluency, Relevance, Language Fusion and Task Specific scores respectively.
5.5 Human evaluation
Table 6 presents the human evaluation scores for mBART in four tasks. Multilingual training results can surpass the monolingual results in QG, TG and Summ on relevance. In terms of task-specific score, multilingual results are also superior in SG and QG. This is consistent with the conclusion in Sec. 5.2. On the other hand, language fusion scores in zero-shot setting are extremely low, indicating the pretrained models still lack the ability to generate texts in correct language in zero-shot setting.
6 Leaderboard
We build a leaderboard for  $\mathrm{MTG^{11}}$ . It provides an overall evaluation of models in two scores:
MTGScore MTGScore is designed to evaluate the multilingual model. It is the average of ensem
<sup>9</sup>Zero-shot results show the same trend as shown in Table 18 in Appendix.
10The t-test details are shown in Appendix C.
11The address of MTG leaderboard is https://mtg-benchmark.netlify.app/
Models	MTGScore	MTGScore-XL
M-BERT	28.24	27.72
XLM	27.07	26.99
mBART	29.37	25.63
mT5	29.07	28.63
Table 7: MTGScore and MTGScore-XL for the four multilingual pretrained models.
ble scores over all languages and tasks.
MTGScore-XL MTGScore-XL is a special score for MTG. It enbales better evaluation of cross-lingual generation ability by testing model in 25 cross-lingual directions. It is the average of ensemble scores over all tasks and all cross-lingual language directions.
The MTGScore and MTGScore-XL for the four multilingual pretrained models are shown in Table 7.
Considering the annotation cost, it is not realistic to construct a multiway text generation dataset with all data annotated by human. As a consequence, most of the non-English data in MTG are automatically translated from their English counterparts. Although the n-gram consistency check when round-trip translating the data can guarantee the quality of them to some extent, some translation errors are inevitable. MTG with more annotated data and with data filtered by more reliable methods will be explored in the future.
On the other hand, human often gives an overall evaluation of a generated text rather than measuring it in fine-grained aspects of grammar, fluency and relevance. Thus we try to propose a metric measuring how a text resembles human writing and consider grammar, fluency and relevance as a whole. This metric may not be perfect, but it is a promising direction as there does not exist a really reliable text generation metric nowadays.
In this paper, we propose a multilingual multiway benchmark MTG for text generation. It contains four typical generation tasks: story, question, title generation and text summarization. The key feature of MTG is that it has multiway parallel data across five diverse languages: English, German, French, Spanish and Chinese. It provides the benchmark with the ability to create cross-lingual data between
any two languages and makes the semantic fusion between languages easier. On the other hand, it provides more evaluation scenarios, such as multilingual training, cross-lingual generation and zero-shot transfer. We also benchmark state-of-the-art multilingual pretrained models on our MTG from different metrics (including a newly proposed ensemble metric) to explore its features and promote research in multilingual text generation.
9 Ethics Consideration
Since we propose a new multilingual text generation benchmark MTG, we solve some possible ethnic considerations in this section.
English dataset We choose ROCStories, SQUAD 1.0, ByteCup and CNN/DailyMail as the English datasets for story, question, title generation and text summarization tasks. All of them are available for research use under their licenses. They can be downloaded free from their websites<sup>12</sup>. We ensure that these datasets are only used for academic research and the dataset construction process complies with the intellectual property and privacy rights of the original authors. Also, our proposed benchmark suite MTG should only be used for academic research purposes.
Annotation process As described in Sec. 3.3, we hire some full-time and part-time language experts to do the annotation. Full-time experts are paid  $40 per day and part-time annotators are paid$ 0.2 per example<sup>13</sup>. Their salary is higher than the local average hourly minimum wage. All annotators are aware of any risk of harm associated with their participation. The annotation process is in compliance with the intellectual property and privacy rights of the recruited annotators. The annotation protocol is proved by the legal department inside the company.
Risk Concern In this paper we propose a new ensemble metric measuring to what degree is the generated text close to human-level. The further pursue for more human-like multilingual generation will possibly raise safety concerns.
$^{12}$ ROCStories requires for some necessary contact information
13Full-time employees work at most 8 hours per day, and the local minimum hourly wage is $3.7. The part-time annotators can produce at least 20 examples per hour.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Leo Breiman. 1996. Bagging predictors. Machine learning, 24(2):123-140.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 866-874.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint arXiv:2102.01672.
Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6473-6480.
Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. 2020. Wiki-40b: Multilingual language model dataset. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2440-2452.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. Xtreme: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411-4421. PMLR.
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-factr: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943-5959.
Rong Jin and Alexander G Hauptmann. 2002. A new probabilistic model for title generation. In COLING 2002: The 19th International Conference on Computational Linguistics.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Yichong Leng, Xu Tan, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu. 2019. Unsupervised pivot translation for distant languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 175-183.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark dataset for cross-lingual pretraining, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-6018.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. Mkqa: A linguistically diverse benchmark for multilingual open domain question answering. arXiv preprint arXiv:2007.15207.
Inderjeet Mani. 2001. Automatic summarization, volume 3. John Benjamins Publishing.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ca glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL 2016, page 280.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. arXiv preprint arXiv:2105.09501.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu, and Lei Li. 2021. Contrastive aligned joint learning for multilingual summarization. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 2739-2750.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
Rong Ye, Wenxian Shi, Hao Zhou, Zhongyu Wei, and Lei Li. 2020. Variational template machine for data-to-text generation. arXiv e-prints, pages arXiv-2002.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. Ncls: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3054-3064.
A Back Translation Threshold Testing
The detailed data sizes of back translation filtered datasets for different tasks are presented in Table 8.
The overall statistics for multilingual pretrained models are presented in Table 9 and the detailed descriptions for them are as follows:
1M-BERT Multilingual BERT (M-BERT) (Devlin et al., 2019) is a single language model pretrained from monolingual corpora in 104 languages using Masked Language Modeling (MLM) task. M-BERT leverages a shared vocabulary of 110k WordPiece tokens and has 12 layers with 172M parameters totally.
XLM The Cross-Lingual Language Model (XLM) (Lample and Conneau, 2019) is pre-trained simultaneously with Masked Language Model (MLM) task using monolingual data and Translation Language Model (TLM) task using parallel data. XLM has a shared vocabulary of  $200\mathrm{k}$  byte-pair encoded (BPE) subwords (Sennrich et al., 2016) and 16 layers totaling 570M parameters.
1mBART Multilingual BART (mBART) (Liu et al., 2020) is a pre-trained encoder-decoder model using denoising auto-encoding objective on monolingual data over 25 languages. mBART has a shared vocabulary of  $250\mathrm{k}$  tokens leveraging Sentence Piece tokenization scheme. mBART consists of 12-layer encoder and 12-layer decoder with a total of 680M parameters.
mT5 Multilingual T5 (mT5) (Xue et al., 2020) is a multilingual variant of T5 (Raffel et al., 2020) leveraging a text-to-text format. mT5 is pre-trained on a span-corruption version of Masked Language Modeling objective over 101 languages. It is composed of 24-encoder layers and 24 decoder layers with 13B parameters.
We use the encoder-decoder architecture for our generation tasks. Among the models described
above, mBART and mT5 have been pretrained for generation tasks, but M-BERT and XLM are only pretrained for encoder representations. Therefore, we initialize the decoder with the encoder parameters for M-BERT and XLM. During the pretraining phase, there are no language tags in M-BERT and mT5. Thus we manually add the language tag at the beginning of the source and target for M-BERT and add the target language tag to the beginning of source for mT5.
We adjust the input format for each task. For QG, we append the answer to the passage and insert a special token to separate them. For SG, we take the beginning four sentences as the source and make the last sentence as the target.