SlowGuess/NAACL_2022_OCR huggingface dataset:
Yiran Chen $^{1}$ , Zhenqiao Song $^{1*}$ , Xianze Wu $^{1}$ , Danqing Wang $^{1}$ , Jingjing Xu $^{1}$ , Jiaze Chen $^{1}$ , Hao Zhou $^{2\dagger}$ , Lei Li $^{3\dagger}$
$^{1}$  ByteDance AI Lab  $^{2}$  Institute for AI Industry Research, Tsinghua University  $^{3}$  University of California, Santa Barbara
{chenyiran. robin, songzhenqiao, wuxianze.0}@bytedance.com
{wangdanqing.122,chenjiaze}@bytedance.com
jingjingxu@pku.edu.cn zhouhao@air.tsinghua.edu.cn
We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation. It is the first-proposed multilingual multiway text generation dataset with the largest human-annotated data (400k). It includes four generation tasks (story generation, question generation, title generation and text summarization) across five languages (English, German, French, Spanish and Chinese). The multiway setup enables testing knowledge transfer capabilities for a model across languages and tasks. Using MTG, we train and analyze several popular multilingual generation models from different aspects. Our benchmark suite fosters model performance enhancement with more human-annotated parallel data. It provides comprehensive evaluations with diverse generation scenarios. Code and data are available at https://github.com/zide05/MTG.
Natural language generation (NLG) aims to automatically generate meaningful texts with the input in different formats, such as images (Anderson et al., 2018), tables (Ye et al., 2020) or texts (Guan et al., 2019). The generated texts generally target at realizing an underlying communicative goal while remaining coherent with the input information and keeping grammatically correct. Multilingual text generation extends the natural language generation task to produce texts in multiple languages, which is important to overcome language barriers and enable universal information access for the world's citizens (Artetxe et al., 2020; Arivazhagan et al., 2019; Pan et al., 2021).
To achieve this goal, various multilingual text generation datasets have been proposed. Some of them do not incorporate cross-lingual pairs (Liang et al., 2020; Ladhak et al., 2020). This limits the
knowledge transfer from one language to another. Others involve cross-lingual pairs while English is included on either source or target side in most cases (Zhu et al., 2019; Ladhak et al., 2020), leading to difficult transfer between low-resource or distant language pairs. Constructing a multilingual text generation dataset that can directly transfer knowledge between any two languages is still under-explored.
To this end, we propose MTG, a human-annotated multilingual multiway dataset. Multiway means that the same sample is expressed in multiple languages. It covers four generation tasks (story generation, question generation, title generation and text summarization) across five languages (English, German, French, Spanish and Chinese). We do not include multilingual machine translation because MT itself is a standard task. The multiway parallel feature enables cross-lingual data construction between arbitrary language pairs. Such direct parallel signal promotes knowledge transfer and cross-lingual generation between any language pairs (even distant pairs such as Spanish-Chinese) without involving an intermediate language such as English (Leng et al., 2019).
The multilingual multiway feature also enables various training and test scenarios. In this paper, we design four scenarios to verify the advantages of our MTG from different aspects. Several representative pretrained multilingual models are employed to test these scenarios, including multilingual BERT (M-BERT) (Devlin et al., 2019), XLM (Lample and Conneau, 2019), mBART (Liu et al., 2020) and mT5 (Xue et al., 2020). We leverage various metrics to assess the coherence and diversity of the outputs generated by these models. Besides, we also propose an ensemble metric, which mainly focuses on relevance, measuring to what degree is the generated text close to human-level. Human evaluation is also conducted to validate models' performances.
$\dagger$  Work is done while at ByteDance.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2508 - 2527
In summary, the contributions of this paper are listed as follows:
(i) We propose a new human-annotated multilingual multiway text generation benchmark suite MTG.
(ii) We design a new evaluation metric measuring how a text resembles human writing and prove that it has higher correlation scores with human scores compared with other automatic relevance metrics.
(iii) We evaluate several representative pretrained multilingual models on our proposed MTG and make a rigorous analysis to verify its advantages.
A significant body of works have been committed to the construction of multilingual datasets covering diverse tasks (Hu et al., 2020; Jiang et al., 2020; Longpre et al., 2020). XTREME (Hu et al., 2020) is a multilingual understanding benchmark across 40 languages and 9 tasks, but it does not cover any generation task. Jiang et al. (2020) propose X-FACTR, which is a cross-lingual factual retrieval benchmark. Longpre et al. (2020) propose MKQA, an open-domain question answering evaluation dataset covering 26 diverse languages. Ladhak et al. (2020) present WikiLingua, which is a large-scale, multilingual dataset for cross-lingual abstractive summarization systems. MLSUM (Wang et al., 2021) is a dataset for text summarization in 12 languages. Wiki-40B (Guo et al., 2020) is a multilingual language model dataset across  $40+$  languages. Although these datasets cover multiple languages, they either belong to natural language understanding tasks or a single, specific generation task, which limits researchers to obtain general findings incorporating a set of generation tasks.
XGLUE (Liang et al., 2020) is a cross-lingual benchmark dataset for nine understanding tasks and two generation tasks. GEM (Gehrmann et al., 2021) is a newly-presented vision-language dataset covering 11 image-language and video-language tasks and 32 languages. These two datasets encompass multiple tasks and languages. However, a remarkable difference of our MTG from XGLUE and GEM is that MTG focuses on text-to-text generation tasks and is parallel across all languages, which facilitates easier knowledge transfer.
3 Dataset Collection and Methodology
This section will introduce how to create the benchmark suite for multilingual text generation (MTG).
In order to construct multiway parallel dataset, the initial dataset is translated into other languages by an off-the-shelf translation model. Part of the translated data is randomly selected for further human annotation to increase data quality. The selection of tasks, initial datasets and languages are based on several principles as described below.
3.1 Task and Dataset Selection
It is important to select suitable tasks for our MTG benchmark to make it diverse and challenging. Thus, we define several criteria during the task selection procedure:
Task Definition Tasks should be well-defined, which means that humans can easily determine whether the generated results meet the task requirements.
Task Difficulty Tasks should be solvable by most college-educated speakers. In the meantime, they should be challenging to current models, the performance of which in various test scenarios falls short of human performance.
Task Diversity Tasks should cover a wide range of generation challenges that allow for findings to be as general as possible.
Input Format The input format of the tasks needs to be as simple as possible to reduce the difficulty of data processing. Besides, it should not contain anything but text (e.g., without any images or videos).
In order to meet the above criteria, 8 domain experts are asked to vote from 10 typical generation tasks<sup>1</sup>. Finally, four generation tasks are selected for MTG, which are story generation, question generation, title generation and text summarization. Story generation (SG) aims to generate the end of a given story context, which requires the model to understand the story context and generate a reasonable and fluent ending (Guan et al., 2019). Question generation (QG) targets at generating a correct question for a given passage and its answer (Duan et al., 2017). For the same passage with different answers, the system should be able to generate different questions. Title generation (TG) converts a given article into a condensed sentence while preserving its main idea (Jin and Hauptmann, 2002). The title should be faithful to the original document and encourage users to read the news
<sup>1</sup>These generation tasks are story generation, commonsense generation, style transfer, question generation, question answering, dialogue generation, title generation, text summarization, image caption, and data-to-text generation.
Task	Corpus	Domain	Format	Goal
Story Generation	ROCStories	Daily life	<story>	Generate the end of the story
Question Generation	SQUAD 1.0	Wikipedia	<passage,answer, question>	Generate the question of the answer
Title Generation	ByteCup	News	<article, title>	Generate the title of the document
Text Summarization	CNN/DailyMail	News	<article, summary>	Generate the summary of the document
at the same time. Text summarization (Summ) aims to condense the source document into a coherent, concise, and fluent summary (Mani, 2001). It is similar to title generation but the output of text summarization is relatively longer. These four tasks focus on different generative abilities and realize different goals.
After confirming the tasks, the next step is to choose the dataset for each task. The two selection principles are listed as follows:(1) License: Task data must be available under licenses that allow using and redistributing for research purposes. The dataset should be free and available for download. (2) Quality: The dataset size should be as large as possible and the quality should be checked.
English datasets are chosen as the initial datasets because they are more accessible in all four tasks and have relatively larger size compared with datasets in other languages. We choose ROCStories (Mostafazadeh et al., 2016) for story generation, SQUAD 1.0 (Rajpurkar et al., 2016) for question generation, ByteCup  $^{2}$  for title generation and CNN/DailyMail (Nallapati et al., 2016) for text summarization. These datasets are popular in the corresponding fields and have been verified to be high-quality by many works. Moreover, they are all under a permissive license. An overview of all task datasets is shown in Table 1.
3.2 Language Selection
The original datasets are in English (en) only and we want to extend them into a multiway parallel form. This means that all English texts should be translated into other languages, which will lead to high annotation costs. Thus, a state-of-the-art translator is leveraged to do the translation and then annotators are asked to correct the translated text. Considering this construction method, MTG should contain languages that (1) have good English-to-X translators and (2) are diverse in language family. Finally, German (de), French (fr), Spanish (es) and Chinese (zh) are chosen. German is from the
same language branch as English while French and Spanish are from different ones. Chinese is more distant from the rest of languages in the language family tree.
Table 1: The description of tasks and English datasets included in MTG. For story generation, we use the last sentence as story end to be generated and the rest as input.
Task	SG, QG, TG, Summ
For each language
Rough training size	76k/61k/270k/164k
Annotated training size	15k/15k/15k/15k
Annotated development size	2k/2k/2k/2k
Annotated test size	3k/3k/3k/3k
For five languages (en, de, fr, es, zh)
Total Annotated size	400k
Total dataset size	6.9m
Table 2: The number of samples in MTG. MTG consists of four subsets: rough training, annotated training, development and test set. The rough training set is filtered by back translating across five languages. The annotated training, development and test sets are corrected by human experts.
3.3 Data Collection
After determining the tasks and languages, we introduce the data collection process to get the MTG. The Google Translate $^3$  is used to translate the English datasets to the selected languages. To control the quality of translated texts, we back translate the text to English and filter the samples whose n-gram overlap ratios with the original English texts are lower than a certain threshold. Different threshold values (from 0.3 to 0.6 with 0.1 as step length) are tested and if it is set to 0.6, the training data size of QG will drop more than  $60\%$ . Thus we decide to use 0.5 as the threshold number to improve the quality of the filtered data while still maintaining more than  $70\%$  of the original training data. $^4$  Samples in four languages are aligned to ensure that the dataset is multiway parallel.
20,000 samples of each task and language are randomly selected for annotation under the premise
$^{2}$ https://www.biendata.xyz/competition/bytecup2018/
<sup>3</sup>https://translate.google.com/
4The detailed sizes of the filtered datasets with respect to different thresholds are included in appendix A.
Correlation	AdaBoost	DecisionTree	ExtraTree	GradientBoosting	KNeighbors	Linear	RandomForest	SVR	Bagging
Pearson	0.100	0.133	0.190	0.215	0.192	0.173	0.208	0.113	0.240
Correlation	BLEU	ROUGE-1	ROUGE-2	ROUGE-L	METEOR	BERTScore-P	BERTScore-R	BERTScore-F1	Bagging
Pearson	0.180	0.142	0.163	0.144	0.122	0.142	0.176	0.162	0.344
Table 3: The correlation scores between automatic metric scores and human-annotated scores (the average scores of grammar, fluency and relevance). Upper part of the table shows the correlation scores of different regression algorithms in test set of all languages. The lower part demonstrates correlation scores of our ensemble score (the bagging regressor) and other classic automatic scores in test set without Chinese results because Meteor does not support Chinese.
of ensuring inter-language alignment. The annotators are required to further check the translated results based on the following rules: (1) Semantic aligned Whether the target text is meaningful and is fully semantic aligned with the source text. (2) Fluency Whether the translated text is grammatically correct. (3) Style Whether the translation follows the norms of local culture, language conventions, and gender-related words. If the translated text contradicts any of the above rules, annotators will correct it accordingly. The annotated data is then split to  $15\mathrm{k} / 2\mathrm{k} / 3\mathrm{k}$  as training/development/test subsets.
A team of 10 full-time experts<sup>5</sup> are hired to do the annotation, who are paid daily. Some part-time workers<sup>6</sup> are also employed to increase the annotation throughput, who are paid by the number of annotations. Each annotator is an expert in at least two languages (English and another target language). They are trained to correct translation errors according to the above rules, first a small number of samples for trial, these annotation results are re-checked by us and feedback is given to the annotators to help them understand the tasks better. After this annotation training process, the annotators start to annotate the dataset. For quality control, we sample  $2\%$  from the annotations and arrange for 9 experts to double-check them. Each example is assigned to two other experts and the data is qualified only if both of them agree on the annotation<sup>7</sup>. If more than  $5\%$  of the annotations fail, then all the data of that annotator for that day will be re-checked.
Then the multiway parallel generation benchmark MTG is finally completed. It contains four
different generation tasks in five languages and its quality is improved by the incorporation of human annotation. However, the number of human-annotated data is still small due to cost concerns. Introducing more human-annotated data or carrying out extra filtering for machine-translated data can be future directions to further improve the quality of MTG. The statistics of MTG is shown in Table 2.
In this section, we conduct extensive experiments to benchmark the difficulty of our proposed MTG via several state-of-the-art multilingual models under different scenarios.
The performance of the following four popular multilingual pretrained models is explored:
M-BERT Multilingual BERT (M-BERT) (Devlin et al., 2019) is a language model pretrained from monolingual corpora in 104 languages using Masked Language Modeling (MLM) task.
XLM The Cross-Lingual Language Model (XLM) (Lample and Conneau, 2019) is pretrained with Masked Language Modeling (MLM) task using monolingual data and Translation Language Modeling (TLM) task using parallel data.
mBART Multilingual BART (mBART) (Liu et al., 2020) is a pretrained encoder-decoder model using denoising auto-encoding objective on monolingual data over 25 languages.
mT5 Multilingual T5 (mT5) (Xue et al., 2020) is a multilingual variant of T5 (Raffel et al., 2020) formatting all tasks as text-to-text generation problems. mT5 is pretrained on a span-corruption version of Masked Language Modeling objective over 101 languages.
Detailed descriptions for models are included in Appendix B.
5There are 3 language experts for German, 3 for French, 4 for Spanish and 4 for Chinese
There are 16 part-time workers who are participated in the German annotation, 39 for French, 4 for Spanish and 15 for Chinese.
7The grammar, expressions, and punctuation of the annotated text are completely correct and the expressions are in accordance with the foreign language.
(a) SG M-BERT
(c) SG mBART
(d) SG mT5
(e) QG M-BERT
(g) QG mBART
(h) QG mT5
(i) TG M-BERT
(m) Summ M-BERT
Figure 1: The cross-lingual ensemble metric results for four models in four tasks. Every cell of row lang1 and column lang2 means the result when the languages of input and output are lang1 and lang2 respectively. Deeper red represents better cross-lingual performance while deeper gray indicates worse performance.
(n) Summ XLM
(k) TG mBART
(o) Summ mBART
(1) TG mT5
(p) Summ mT5
In order to fully understand the model performance, the quality of generated texts is evaluated from different aspects, including metrics measuring the relevance between outputs and references (e.g., BLEU, ROUGE, and BERTScore) and metrics measuring the diversity of the generated texts (e.g., Distinct). Moreover, we propose a new ensemble metric leveraging relevance metrics to measure how close the generated text is to human writing. It not only has higher correlation scores with human judgments but also is capable of measuring model performances fairly between languages.
N-gram based Metrics N-gram based metrics evaluate the text-overlapping scores between the outputs and references. The following three metrics are used: (1) BLEU (Papineni et al., 2002) is a popular metric that calculates the word-overlap scores between the generated texts and gold-standard ones. We use the BLEU-4, which is the average score for unigram, bigram, trigram, and 4-gram. (2) ROUGE (Lin, 2004) is a recall-oriented metric that counts the number of overlapping units such as n-gram and word sequences between the produced texts and gold-standard ones. (3) ME
TEOR (Banerjee and Lavie, 2005) relies on semantic features to predict the similarity scores between system hypotheses and human references.
**Embedding based Metrics** The embedding-based metrics can, to a large extent, capture the semantic-level similarity between the generated text and the ground truth. BERTScore (Zhang et al., 2019) computes the similarity of candidate and reference as a sum of cosine similarities of tokens using BERT contextual embeddings.
Diversity Metrics We also employ the distinct metric (Li et al., 2016), which calculates the proportion of the distinct n-grams in all the system hypotheses and can be used to evaluate the diversity of the generated texts.
Human Evaluation Human evaluation is also leveraged to better estimate the quality of model outputs. Specifically, 30 cases are randomly sampled from the test set for each task and language while ensuring all 30 cases are aligned among five languages, and then they are presented to human annotators with the model outputs. The generated texts are evaluated under task-agnostic and task-specific aspects. Task-agnostic aspects include Grammar, Fluency, Relevance and Language
Task	Model	BLEU	ROUGE-L	METEOR	BERTScore	Distinct-1	Ensemble
mono	multi	mono	multi	mono	multi	mono	multi	mono	multi	mono	multi
SG	M-BERT	2.486	2.836	16.680	17.240	0.139	0.140	0.741	0.743	0.952	0.959	30.891	30.987
XLM	4.026	2.992	24.520	22.820	0.145	0.144	0.754	0.744	0.967	0.967	28.364	28.449
mBART	4.514	4.880	19.320	19.920	0.149	0.156	0.759	0.762	0.985	0.983	31.430	31.907
mT5	2.668	3.832	16.280	18.620	0.126	0.145	0.751	0.759	0.976	0.974	31.623	31.482
QG	M-BERT	8.266	9.980	27.340	29.520	0.240	0.262	0.778	0.785	0.938	0.944	30.553	30.526
XLM	16.472	15.264	41.100	40.600	0.305	0.298	0.810	0.809	0.966	0.967	30.072	29.979
mBART	16.256	17.624	36.640	38.140	0.298	0.315	0.811	0.817	0.981	0.983	32.522	32.961
mT5	15.792	17.700	34.100	37.680	0.294	0.313	0.806	0.818	0.977	0.979	32.257	32.944
TG	M-BERT	9.524	10.550	25.440	26.360	0.214	0.228	0.749	0.754	0.930	0.957	28.971	29.422
XLM	11.144	11.926	26.960	28.660	0.236	0.248	0.752	0.759	0.946	0.941	28.808	29.063
mBART	14.726	14.786	31.680	32.120	0.257	0.260	0.773	0.775	0.966	0.968	30.556	30.322
mT5	11.336	13.546	26.460	29.400	0.223	0.257	0.753	0.767	0.959	0.956	29.556	30.010
Summ	M-BERT	9.766	10.956	31.280	32.220	0.221	0.232	0.748	0.751	0.787	0.815	22.122	22.018
XLM	9.486	11.830	30.160	34.740	0.235	0.235	0.729	0.755	0.814	0.772	19.281	20.770
mBART	12.858	12.792	32.940	32.920	0.256	0.257	0.750	0.750	0.796	0.803	21.972	22.292
mT5	5.022	6.090	25.060	27.980	0.145	0.162	0.724	0.741	0.826	0.870	20.499	21.826
Table 4: Automatic scores averaged across five languages for four models on four tasks. Mono and multi mean models are trained in monolingual and multilingual setting respectively. Higher scores between monolingual and multilingual results are bolded.
Fusion. The former three aspects are scored from 1 to 5 while the language fusion score is set to 1 if all tokens of a model-generated text are in the target language and 0 otherwise.
Besides task-agnostic aspects, the generated text is also evaluated under task-specific aspects. For title generation and summarization, coverage measures the degree to which the generated text covers the main content of the document. Correspondence for question generation measures the extent to which the generated question is matched with both document and answer. For story generation, we further evaluate whether the generated story is logically feasible. All task-specific aspects are scored from 1 to 5.
Ensemble Metric Some N-gram based metrics such as BLEU and ROUGE largely depend on the tokenizer for specific languages. For example, BLEU scores for Chinese outputs are relatively high because it simply uses a character-level tokenizer. This causes unfair comparison between different languages. To this end, we propose an ensemble metric that evaluates the degree to which a piece of text resembles manual writing. It not only enables fair comparison between languages but is also proved to have a better correlation with human-annotated scores at the end of this subsection. We first average the grammar, fluency and relevance scores as targets, then normalize the automatic metrics and human scores among every language to eliminate the score discrepancy between languages. Three relevance metrics (BLEU, ROUGE-L, and BERTScore-F1) are gathered as features. The sam
plies are split into training, development and test sets.
After comparing different regression models' performance as shown in the upper part of Table 3, we finally choose bagging regression model (Breiman, 1996) as the ensemble metric. Moreover, the bagging ensemble metric shows a higher correlation with human-annotated scores compared with other relevance automatic metrics as shown in the lower part of Table 3.
4.3 Evaluation Scenarios
To validate the effect of different experimental settings on model performance, several state-of-the-art multilingual models are studied under four evaluation scenarios.