CCEval: A Representative Evaluation Benchmark for the Chinese-centric Multilingual Machine Translation
Authors: Lianzhang Lou
Xi Yin
Yutao Xie
Yang Xiang
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingual Machine Translation
Low-resource Languages
Evaluation Benchmark
Evaluation Dataset
Chinese-centric
Translation Evaluation
Test Set

Abstract: The Chinese-centric Multilingual Machine Translation (MMT) has gained more importance recently due to increasing demands from international business development and cross-cultural exchanges. 
However, an important factor that limits the progress of this area is the lack of highly representative and high-quality evaluation benchmarks. 
To fill this gap, we propose CCEval, an impartial and representative Chinese-centric MMT evaluation dataset. 
This benchmark dataset consists of 2500 Chinese sentences we meticulously selected and processed, and covers more diverse linguistic features as compared to other MMT evaluation benchmarks.
These sentences have been translated into 11 languages of various resource levels by professional translators via a rigorously controlled process pipeline to ensure their high quality.
We conduct experiments to demonstrate our sampling methodology's effectiveness in constructing evaluation datasets strongly correlated with human evaluations.
The resulting dataset enables better assessments of the Chinese-centric MMT quality.
Our CCEval benchmark dataset is available at https://bright.pcl.ac.cn/en/offlineTasks.

CCEval: A Representative Evaluation Benchmark for the Chinese-centric Multilingual Machine Translation
Lianzhang Lou'! Xi Yin Yutao Xie², Yang Xiang',
'Peng Cheng Laboratory, Shenzhen, China
International Digital Economy Academy, Shenzhen, China
1zlou1 eeailm993mmmal. com, "yxthuêqq. com, yutayxieaadeeeeeeoee edu. cn, xiangy@pcl. wwtpryypcl.ac.cm
Abstract
attention compared to the MMT study centered on English (Aharoni et al., 2019; Zhang et al., 2020a). A critical challenge hindering the progress of the Chinese-centric MMT is evaluation, mainly due to the lack of representative and human- correlated evaluation datasets. Recently, a few evaluation benchmarks for MMT have emerged, such FLORES-101 (Goyal al., 2022) and Tatocba (Tiedemann, 2020). However, they only be used for English-centric translation evalu- ations (Goyal et al., 2022: Thu et al., 2016), have not undergone rigorous human expert annota- tion and quality inspection process to ensure the dataset's high quality (Tiedemann, 2020), or are only limited to specific domains (Anastasopoulos et al., 2020). Furthermore, an ideal MMT evalua- tion dataset shall provide system-level evaluations that correlate closely with human assessments. To achieve a high correlation, Zhan et al. (Zhan et al., 2021) selected samples yielding substantial perfor- mance fluctuations of evaluated systems, while Liu et al. (Liu et al., 2022) allocated greater weights to samples with elevated translation chunk entropy. The restrictions of these works are their depen- dence on hypothesis text in the source sentence sampling process.
The Chinese-centric Multilingual Machine Translation (MMT) has gained more impor- tance recently due to increasing demands from international business development and cross- cultural exchanges. However, important factor that limits the progress of this area is the lack highly representative and high- quality evaluation benchmarks. To fill this gap, we propose CCEval, an impartial and rep- resentative Chinese-cene centric MMT evaluation dataset. This benchmark dataset consists of 2500 Chinese sentences we meticulously se- lected and processed, and covers more diverse linguistic features as compared to other MMT evaluation benchmarks. These sentences have been translated into 11 languages of various resource levels by professional translators via rigorously controlled process pipeline to en- sure their high quality. We conduct experi- ments to demonstrate our sampling method- ology's effectiveness in constructing evalua- tion datasets strongly correlated with human evaluations. The resulting dataset enables bet- ter assessments of the Chinese-centric MMT quality. Our CCEval benchmark dataset is available https: /tbrig:/bbigtt...../ pcl. ac. coeen//e/////// offlineTaske Tasks.
1 Introduction
To overcome previous limitations mentioned above, propose highly representative evalu- ation benchmark dataset, named CCEval, which specifically constructed enable evaluation of the Chinese-centric MMT models/systems. Each language pair in the CCEval contains 2,500 Chi- nese source sentences we carefully selected and processed using diversity and translation diffi- culty based approach. Utilizing human-annotate Multidimensional Quality Metrics (MQM) (Freitag et al., 2021a) corpus from recent Workshop Ma- chine Translation (WMT) evaluation campaigns, we conduct experiments to demonstrate the valid- ity of this approach in building evaluation datasets correlated better with human assessments. These
Multilingual Machine Translation (MMT) is hot research area that has made big progress recently (Lin et al., 2020; Fan et al., 2021; Costa-jussà et al., 2022). As the language spoken by the largest popu- lation in the world, Chinese has become more and more influential globally, and China has been grow- ing its worldwide impact. For example, in 2021, China's total import and export trade amount has reached 21% of the entire global trade a which is significant proportion. Despite this, the Chinese-centric MMT research has received less
'quual contribution. "Corresponding author. hhtps:///cc worddenrppfill//ouunntti

---

well selected Chinese source sentences have been professionally translated into eleven languages of various resource levels, i.e., Mongolian, Lao, Por- tuguese, Polish, Czech, Turkish, Italian, Russian, Arabic, Greek, and Croatian. Note that about half of them are low-resource languages. We employ systematic annotation and translation quality in- spection workflow to guarantee the dataset's s high quality.
correlation between dataset evaluation results and human judgments.
Diversity-basee selection In terms of the lin- guistic diversity of the Chinese source sentences, we focus on the distributions of topic domain, word frequency, and grammar complexity. Our selection strategy is as follows:
1) We picked sentences from multiple domains and topics, evenly covering six major domains: News, Politics, Travel, Biomedical, Technology, and Daily Life, as well as 60 topics, as shown in the Appendix 1.1.
The CCEval dataset is accessible via BRIGHT", a Chinese-centriy MMT evaluation platform. To foster relevant research using our benchmark, we validate the effectiveness of CCEval by conducting evaluation tests on actual MMT baseline models. The contributions of our work are:
2) We obtain the appearing frequency of each candidate sentence according to word-level fre- quency accumulation using unigram language model. Our sampling distribution is similar the obtained frequency distribution of the candidate sentences. The details are dictated in Appendix 1.3. 3) As an indicator of grammar complexity, we conducted syntactic analysis on the candidate sen- tences. Based on the results, we extracted sentences obeying an approximately normal distribution of the syntax tree depth. The details are also described in Appendix 1.3.
We establish and publicly release a representa- tive and high-quality MMT evaluation dataset. To the best of our knowledge, this is the first evaluation benchmark for the Chinese-centric MMT research community.
We propose Chinese source sentence selec- tion criteria and sampling strategy based on the linguistic diversity and translation diffi- culty, facilitating the dataset's higher evalua- tion correlations with human assessments.
selection Represen- tativeness of the evaluation dataset means the de- gree of correlation with human assessments for the system ranking evaluations. A test set with more consistent evaluation results with human judgments is considered to have higher representativeness, and vice versa.
Representativeness-
2 Dataset Construction
The overall construction process of CCEval dataset includes three steps. In the first stage of source text screening, we consider various linguis- tic features of the Chinese sentences, and screen appropriate sentences from the initial text pool to satisfy the corresponding source diversity require- ments. Subsequently, we propose method to cal- culate the translation difficulty of each sentence, and further select highly representative samples based on this method. Finally, in the target sen- tence translation phase, we adopt a strict translation quality inspection pipeline to obtain a high-quality dataset.
In this stage, we selected samples with the top 20% translation difficulty to improve the represen- tativeness the dataset. We verify the validity of this difficulty-based strategy in picking high- representativeness samples by performing experi- ments, as shown in Sec. 3.1.
In light of the sentence translation characteris- ties disclosable by back translations, borrow the idea of denoising autoencoder (DAE) to character- ize the degree of translation difficulty of a sentence. Specifically, we employ a DAE to perform cross- lingual corruption of the source sentence, and then use its degree of recovery to measure the trans- lation difficulty of the sample. The formula for calculating the translation difficulty d of Chinese sentence is follows:
2.1 Source text selection
In the source text screening stage, we first filtered sentences with sensitive words and meaningless text from the initial text pool. Then we focus on the linguistic diversity and the representativeness of the source sentences. The former requirement is important for the authority and fairness of the dataset, while the latter is essential enhance the
d = d= 1=
(1)
where x is the representation of the source sen- tence, and x, is the representation of the restored
"https: http://tttggttppte pcl, ac. cn/en/

---

Post edit
Source sentences
Dataset corpus
Translation
Review
Quality check
Few sampling
Pliot translation
Qualification check
MQM Quality standard
C
Figure I:
Workflow of Translation Quality Control in CCEval.
Language
p1
68
#Sentences
2500
Quality
91.8%
90.7%
91.1%
92.7%
93.3%
Average Characters per Sentence
30
Language
it
tr
hr
Quality
91.8%
90.7%
91.1%
92.7%
95.0%
#Topics
60
% of Sentences with URL
85%
Table I: Translation Quality Score of All Languages. lo, mn. pt, pl, cs, ru, it, tr, ar, el. and hr denote Lao, Mongolian, Portuguese, Polish, Czech, Russian, Italian, Turkish, Arabic, Greek, and Croatian, respectively.
Domain
# Sentences
2500
Technology
416
Biomedical
416
Travel
416
source sentence. The similarity metric s is com- puted using the BERT-score Recall (Zhang et al., 2020b). Since we use this approach to further se- lect source sentences for each language pair in- volved in CCEval, in the DAE process we utilized open-sourced corpora train two separate Chi- Mese2Many and Many2Chinese MMT models.
Daily Life
416
News
418
Politics
418
Table 2: Statistics of CCEval Source Text.
generally lived for 5 years more in the tar- get language-speaking countries regions. They check the quality of each translated sentence pro- vided by the LSPs based on Direct Assessments (DA). The translation quality evaluated using a 5-point scale according to the fidelity and fluency, as detailed in Appendix 1.2, where translations re- ceiving a score of or greater are deemed correct.
2.2 Translation annotation and quality management
Quality of the annotated translation from the Chi- nese source text other languages is crucial to ensure the fairness of MMT evaluations. Therefore, strict translator qualification review and translation quality control process are applied to guarantee the translation quality of CCEval.
The final translation quality scores of all eleven target languages are shown in Table 1. It is ob- served that all languages have quality score above 90%, with a maximum of 95%.
The whole workflow of translation annotation and quality control is shown in Fig. 1.
In the source translation phase, we follow the translation annotation workflow of FLORES- 101 (Goyal et al., 2022). For the selected Chinese sentences, we have engaged 11 different Language Service Providers (LSPs) to annotate the transla- tions in their respective languages. Detailed infor- mation about the annotators of different languages is shown in Appendix 1.6.
2.3 Dataset overview
The following is detailed breakdown CCEval's composition from multiple perspectives.
Sources Table 2 provides overview of the CCEval source text. includes 2,500 diverse and representative sentences. The mean length of our selected sentences is set moderate value 30 characters, with long sentences exceeding 60 char- acters filtered. All samples contain rich metadata
After that, our linguistic experts implement the human quality inspection process. These profes- sional linguistic experts are well qualified and have

---

Dataset
#SPs
#Langs
cc
Div
Domain
#Topics
Quality
Source
FLORES-10) 101 (Goyal al., 2022)
3001
101
10
>90%
WikiNeesWWiJJomoo//iii
ALT (Thu al., 2016)
20000
x
x
News
WikiNews
1100-19 (Anastasopoulos et al., 2020)
3071
Biomedical
>00%
PubMed/W ct.al
CCEval
2500
12
6
60
>00%
Chinese Webs/Books
Table 3: Comparison of C'Eval with Other MMT Evaluation Benchmarks. #SPs, #Langs, CC, and Div denote the number of Sentence Pairs, the number of Languages, Chinese-centric, and Diverse, respectively. To the best of our knowledge, CCEval is the first Chinese-centric multilingual machine translation test set, which shows a diverse distribution in domain, topic, and other dimensions, as well as an equally tight control in dataset quality.
information, such as domain, topic, URL (except for the Daily Life domain), and created date and time, to facilitate source text tracing and in-depth research. This traceability is enabled by our se- lection of mainstream Chinese websites (detailed in Appendix 1.4) as the initial corpus pool of the source sentences.
3 Experiments and Results
3.1 Empirical Verification: Validity of the difficulty-based strategy
We experimentally verify the effectiveness of using difficulty-based approach in selecting high- rep- resentativeness samples. Ideally the verification should have been done on the CCEval dataset di- rectly, but it is very costly to do MQM/DA anno- tation of different systems' performance on CCE- val, since this must be done by senior linguistic experts mastering both Chinese and our eleven lan- guages. Alternatively, we validate the hypothesis by performing experiments two publicly avail- able WMT datasets with rich MQM annotation data, as well as an in-house test set with DA anno- tations of different systems.
Domains The dataset covers representative domains and 60 distinct topics, as mentioned in Sec. 2.1. These domains are widely recognized, with News, Technology. and Dailylife domains adopted in WMT evaluations, Travel domain used in FLORES-101 (Goyal et al., 2022), and Biomed- ical domain specifically focused on by TICO- 19 (Anastasopoulos et al., 2020).
Languages Our CCEval contains sentence pairs between Chinese and eleven languages (Table 1) of different families, including Slavie languages, Ara- bic languages, and Southeast Asian languages, with different levels of resources available. About half of these languages are low-resource languages, and they exhibit significant differences from Chinese in terms of morphology and grammar structure.
Specifically, we first use Eq. 1 to calculate the sample difficulty from the two open-sourced translation evaluation sets of WMT 2021 and WMT 2022, and one in-house Chinese-to-Mongollann translation test set. Then obtain the statistical difference between two sampling strategies using different subsets with varied difficulty levels, in terms of system ranking correlation with human assessments.
In Appendix 1.5, we provide some data examples of CCEval.
Table shows the comparison of CCEval with other available MMT evaluation benchmarks. Over- all, CCEval is the first Chinese-centnn centric MMT test set with high quality. Compared to other main- stream benchmarks, our dataset has a comparable size and shows more diverse distributions in regard to domain, topic, and other linguistic features.
Figure. 3 shows the three translation evalua- tion datasets' system ranking correlations with the MQM/DA human evaluations under different sub- set selection strategies. For all three datasets, our proposed difficulty-based subset sampling strategy yields a higher correlation than random subset sam- pling. In addition, the selection of more difficult samples further enhances the correlations.
In particular, as shown in Fig. 2, we compare a typical source sentence distribution in CCEval with FLORES-101 101, in terms of sentence length, word frequency, and grammar complexity. It is observed that the source sentences in CCEval have approxi- mately normal distributions of word frequency and grammar complexity. In contrast, their distribu- tions in FLORES-I01 are unbalanced and biased to the low ranges.
3.2 Evaluation of Multilingual Baseline Models using CCEval Benchmark
Baselines a) M2M-100 (Fan al., 2021) is the first many-to-many machine translation model devel- oped by Meta, which was trained on CCAligned corpus (EI-Kishky et al., 2020) and is capable of arbitrary bilingual translations within more than

---

1600
CCtval
700
1200
@CCEvv
1200
FLORES-101
FLOREG-181
1000
PLORES-IOO
1000
800
Sertbence
800
400
600
400
400
100
200
(0,20)
[20,40]
(40,60)
(60,80)
(0.00.2)
(0....))
Sentence length
Normalized word frequency
Grammar
complexity
(a)
(b)
(c)
Figure 2:
Comparisons of typical source sentence distribution in CCEval with FLORES-I01. CCEval
possesses better linguistic diversity in terms of sentence length, word frequency, and grammar complexity.
Resource level
Low
Medium
High
Language
lo
hr
pl
pt
M2M-100
from Chinese
0.69
0.81
10,40
11.75
9.76
8.87
12.95
14.09
7.45
11.38
2021)
Chinese
3.89
5.09
14.09
12.73
11.95
12.36
12.61
12.33
13.43
mBART50
from Chinese
0.32
1.65
0.85
1.39
2.12
2.70
1.80
0.66
(Tang et al., 2020)
to Chinese
4.42
0.87
4.03
1.82
1.92
3.27
1.01
3.58
2.08
Table 4: Multilingual translation evaluation results (SacreBLEUU using CCEval benchmark.
100 languages. It covers all 11 languages in CCE- val. b) mBART50 (Tang et al., 2020) is another multilingual translation model obtained by pre-
Kendall-tau
0.65
Diffcully--aaadddd shZen, WMT 2022
zhßen,
0.60
Difficully based,
2021
Random, th2en, WHT 2021
training mBART (Liu ct al., 2020) on 25 more languages and further finetuning on multilingual corpus, which covers 9 languages in CCEval ex- cept for Lao and Greek.
0.55
Difficultyba based, zh2mn,
Random, zh2ms, in-ouue
0.50
0.45
Metric We use SacreBLEU (Post, 2018) the evaluation metric. For Chinese, we perform full- width to half-width conversion.
0.40
0.35
Top%
Results The evaluation results are shown in Ta- ble 4. The main findings are: Firstly, for both models and most languages, the SacreBLEU from Chinese to other languages is lower than that of the reverse direction, i.e., from other languages to Chinese. Furthermore, for the M2M-100 model, the SacreBLEUs between low-resource languages and Chinese are much lower than those berween high- or medium-resoure languages and Chinese.
Figure Three translation evaluation datasets' sys- tem ranking correlations with the MQM/DA human evaluations under different subset selection strate- gies. Horizontal axis is the top percentage for the se- lected subset based on the sample's difficulty score rank- ing. Vertical axis is the Kendall's tau-like correlation coefficient (Freitag al., 2021b).
also welcome the community to build up support for more languages and more valuable evaluation samples.
4 Conclusion
this paper, we construct and publicly release CCEval, a Chinese-centric MMT evaluation bench- mark dataset covering twelve languages of vary- ing resource levels, about half of which are low- resource languages. The dataset is highly repre- sentative, and the linguistic diversity of Chinese outperforms existing MMT benchmarks. We hope that the open source of this dataset will benefit the community. the future, we will continue ex- pand the number of languages covered in CCEval and increase the dataset's sample size. We would
Limitations
It is very costly to do MQM/DA annotation of dif- ferent systems' performance on CCEval, since this must be done by senior linguistic experts master- ing both Chinese and 11 languages. Therefore, cannot verify the effectiveness of the difficulty- based strategy on the CCEval dataset. Rather, we could only validate this hypothesis using one in- house test set with DA annotations of different

---

systems and two publicly available WMT datasets with rich MQM annotation data.
Wang. 2022. No language left behind: Scal- ing cuuman-centered machine translation. CoRR. abs/2207. 0ts/2227.04662.
As a MMT evaluation dataset with diverse distri- bution, we believe that a larger amount of data sam- ples would increase the evaluation confidence and robustness of CCEval. However, due to the cost and time constraints. currently only 2,500 sentence pairs of eleven languages each can be provided, and the coverage in some linguistic dimensions may be still low.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Kochn. 2020. Ccaligned: A massive collection of cross- loos-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020. Ouline, November 16-20, 2020, pages 5960-5969. Association for Computational Linguis- tics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky. Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary. Naman Goyal, Tom Birch, Vitaliy Liptchinsky. Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centric multi- lingual machine translation. J. Mach. Learn, Res., 22: 12:107:1-
A Acknowledgements
This work was supported by the Major Key Project of Peng Cheng Laboratory, PCL2022D01, the National Key R&D Program of China, 20227D01 15305, and the National Natural Sci- ence Foundation of China, 62106115. We thank all of the translators and linguistic experts, as well as the translation and quality assurance agencies we worked with, for helping create CCEval, with special thanks to the Translation Institute of the China International Communications Group.
Markus Freitag. George F. Foster, David Grangier. Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey, 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Comput. Linguistics, 9: 9:1460-1474
Trans. Assoc.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George F. Foster, Alon Lavie, and Ondrej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Pro- ceedings of the Sixth Conference on Machine Transla- tion, WMT @ EMNLP 2021, Online Event, November 10-11, 2021, pages 733-774. Association for Com- putational Linguistics.
References
Roce Aharoni. Melvin Johnson, and Orhan Firat. 2019, Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLI' 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume (Long and Short Papers), pages 3874- 3884. Association for Computational Linguistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation. Trans, Assoc. Comput. Linguistics, 10:522-538.
Antonios Anastasopoulos, Alessandro Cattelan, Zi- Yi Dou, Marcello Federico, Christian Federmann. Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Mac- duff Hughes, Philipp Koehn, Rosie Lazar, William Lewis. Graham Neubig, Mengmeng Niu, Alp Ök- tem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. TICO-19: the translation initiative for covid-19. In Proceedings of the 1st Workshop NLP for COVID- 19@ EMNLP 2020, Online, December 2020, Associ- ation for Computational Linguistics.
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pre- training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 2649-2663. Associa- tion for Computational Linguistics.
Marta R. Costa-jussă, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffer- nan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers, Safiyyah Saleem, Holger Schwenk, and Jeff'
Yilun Liu, Shimin Tao, Chang Su. Min Zhang, Yanqing Zhao, and Hao Yang. 2022. Part represents whole: Improving the evaluation of machine translation sys- tem using entropy enhanced metries. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, Online only, November 20-23, 2022, pages 296-307. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunoy, Marjan Ghazvininejad, Mike Lewis, and

---

Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726-742.
1.3 Details of grammar complexity and word frequency
We employ HanLP for the syntactic analysis of the Chinese texts and count the number of subject-verb- object structures in the text as a measure of gram- matical complexity. Regarding word frequency in the text, we first calculate the frequency of all Chi- nese words in the corpus, and then use the Jieba tokenizer to segment the text. The logarithm of the joint probability of all words is taken as the text's word frequency.
Matt Post. 2018. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation: Research Pupers, pages 186- 191. Belgium, Brussels. Association for Computa- tional Linguistics.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning. CoRR, (as/2008.0040)
1.4 Chinese websites for source collection
Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew M. Finch, and Eiichiro Sumita. 2016. Introducing the
We select the Chinese source texts of all evalua- tion datasets from the following official websites:
asian language treebank (ALT), Proceedings of the Tenth Interaatinall Conference Language Re- sources and Evaluation LREC 2016, Portorvř, Slove-
http:/wwww.news.cn/ https:/cn.nytimes.comm http://www.stdaily.cmm http://fim.llyoul66.. www.:lloooool......//..m
nia, May 23-28, 2016. European Language Resources Association (ELRA).
https://www.yaozhh
Jörg Tiedemann. 2020. The tatoeba translation chal- lenge realistic data for low resource and mul- tilingual MT. In Proceedings of the Fifth Confer- ence on Machine Translation, WMT@ EMNLP 2020, Online, November 19-20, 2020, pages 1174-1182. Association for Computational Linguistics.
1.5 CCEval's data examples
Table 7 demonstrates some data samples of CCEval dataset. All of our evaluation samples contain rich metadata information, such as language, domain, topic, etc.
Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S. Chao. 2021. Variance-awarc machine translation test sets. In Proceedings of the Neural Information Pro- cessing Systems Truck on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, Decem- ber 2021, virtual.
1.6 Annotator details
Table 8 shows the detailed information about the translation annotators.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020a. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Pruceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 1628-1639, Associa- tion for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A
Appendix
1.1
CCEval's topic distribution
CCEval's topic distribution in typical language pair is shown in Table 5.
1.2 Translation quality scoring standard
Table 6 shows the translation quality scoring stan- dard used by our linguistic experts during the hu- man quality inspection process.

---

Topie
#Sentenees
Topic
#Sentences
current jorent_affair affairs
273
diet
24
trave_nottes
238
emotion
22
internal interia_meeicinee medicine
110
gonzo
22
chat government
107
agriculture
obstetrics _nnd gynecologgy
militury
conference
pritese_culture
98
aerospace
common senso
86
local_news news
17
internationality
79
financial management
17
physical education
66
mechanical
16
surgical
64
duily_lid life
15
culture and entertainment
73
philosophy
travel_guid
water_conservange
life perception
environment
popular_scicne science
dermatology
financial
55
health_ane audvisory
13
naavl_exhbbtioonnnfeee information
39
leader_speech
13
material
38
genome_seet _eeuueeceeeeceeeeeeee
13
ENT
37
trip psychiatry
12
chemical industry
36
10
physics
35
leader_ visit
information Technology
shopping
logal
33
traditional Chinese medicine
pediatrics
32
Hong Kong Kong_Macau Macau_and Bateu___nd_Tawaan
life_ sec_scinness
30
character
biomedical oohee depurtments
29
math
education
29
dress
civil civil_enginee eelgnnnnegiing
28
drug
industry unalysis
28
leader lealer_meet______ and _nne_greettt great
health
others
Table 5: Sentence Count of Each Topic in Typical Language Pair.
Score
Description
4
The translation faithfully reflects source semantics and flows smoothly. minor errors that will not affect the understanding of the original meaning.
2
The meaning generally correct. but there are some localized error that may cause certain difficulties in understanding.
sigmificunt errons that have a severe impact on understanding. Almost entirely incorrect completely incomprehensible.
0
Table 6: Translation Quality Scoring Standard.
Chinese source bod
langnags
Tarpst
Tarast leat
Domain
Topic
A.E Dr "o 5:50 RoRben Cu
Hobert Cuabson sumnoouuniioo Iioooarnpen
Tachenkayy
Amsrqpac
Portguee
rquistte pan. lippous pectifuaçios
Desennlstmeno Retorna Minissério Eoologia Ambieme progseeran
Technology
Cannervaccy
Water
ALL
Greek
Tripyn Eve: tapgammeii micoic mmp dopyy nou opipos tre his "pafiir mponagy moddee mvppinus anos onolo dve wohho eepookoweet spoccerec or leoplginn cwoollore dooy zolyo.
Travel
Notes
Haw veru smpoerea 79a byourhh comm awrign morrysudd
Mergelia
rapa eyyse sostacuur vecras foduss.
Trovel
Canikc
Tukáb
Alfela inis, lnceajimm roool satidgim siry loyloyciils misiai?!
Daily
Shopping
Life
AIRER LAN AMM
dalla uticiu illezali
base alla
News
Legil
gaepairennn bobtetoni
tebeomanizzggee paiss waheadsowgg
nortstyfone
porkrellajy nelegine proeitapenoo
Legil
mocio pieécis, oo opinis publicam mocao odosieniells.
Mlsnnootte alisee projinuddl mike vou zaridini gemmoiiiltte inko na projimaiinn
EIME. R9 #28 08655
Cncch
zichwrech Netipli je postitat projmnads afddeé neho je reporttor sibee c eyjimnnn detsorich
Bienedical
Pedvrriee
naddurich pipali.
REA SE FULEE 3530
Ardri
Zaage 12244 aiattais Va dla Duacs
33 lan Uemagn jomy do shyi uplno
Polrice
Uinermmnn
Iilx
Chontias
lon cootebvaree podriio.
melomnoon rata cata Vitwwasg
Poltics
Unverment
Tile
Table 7: Data Examples of CCEval.

---

#Years of Professional Translation Experience
Language
#Annotators
Country of Residence
Age Range
Mongolian
6
Mongolia
29-42
5-16
Lao
6
Laos
29-36
5-12
Portuguese
4
China & Portugal
34-44
5-9
Polish
3
Poland China
34-37
4-11
Czech
4
Czech Republic
27-54
3-21
Italian
4
China
29-42
5-17
Russian
5
Russia
32-38
7-14
Greek
Greece
31-37
Turkish
Turkey
33-38
4-10
Arabic
5
China
28-35
8-13
Croatian
3
Croatia
30-39
4-11
Table Detailed Information about Translation Annotators.


Reviews:
[{"review_id": "ZpgkECspEa", "reviewer": "Reviewer_DEN2", "summary": "", "questions": "1. I would like to know the average number of words/tokens per sentence.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are very grateful for your constructive comments and suggestions. They are very helpful for us to improve the quality of our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first restated and then followed by our point-by-point responses.\n\n> In L101, the authors employed various linguistic features but didn’t discuss which linguistic features were used in this study.\n\nWe mentioned and discussed the specific linguistic features to use in the “Diversity-based selection” section of Sec. 2.1, i.e., domain, topic, word frequency, and grammar complexity.     \n\n> In Equation 1, I didn’t really understand the relation between the source sentence and the restored source sentence by the BERT model. The similarity of these two sentences really matters for the translation of a sentence by humans? In my observance, difficulties in translation might happen for long or complex sentences, because the translator lost the context sometimes.\n\nIntuitively, long or complex sentences are indeed difficult to translate, since humans need to focus on more linguistic aspects during translations. However, there is currently no quantitative metric to measure the degree of \"difficulty\" in terms of sentence length or complexity. Therefore, in light of the sentence translation characteristics disclosable by back translations, we borrow the idea of denoising autoencoder (DAE) to characterize the degree of translation difficulty of a sentence.      \n\n> Why were long sentences over 60 characters removed?\n\nThree reasons: \n1. In Chinese, the proportion of sentences longer than 60 characters is very small (no more than 5%).\n2. The translation annotation of super long sentences is difficult and costly.\n3. In our early pilot assessments of translation quality, the quality scores of these super-long sample sentences were too low.\n\n> LSP’s translation can be modified while verifying the quality of the translation. Why the authors didn’t fix the translation or authors keep them on purpose is not discussed properly.\n\nThere is not a 100% accurate dataset, since \"professional\" is always relative, meaning that theoretically we can always find more professional language experts to discover translation mistakes that are not able to be revealed by less professional experts, hence there is no absolute 100% for the quality score. Therefore, our goal for the dataset construction can only be set as achieving as high as possible quality scores given by highly qualified professionals. In fact, from the experience of previous datasets inspected by our group of linguistic experts, \\>90% quality score given by these experts does indicate a very high-quality dataset.\n\n> 8 topics have 1 to 3 individual sentences, would it be insightful to draw a conclusion based on 1 to 3 translations for a topic?\n\nIt is true that some topics have too few sentences, but our dataset does not intend to compare the evaluation results of different topics. In addition, in the revised paper, we will merge the longtail low-frequency topics into a single category, and will name it as the \"other\" category.\n\n> 2,500 sentences are not enough to evaluate an MMT model.\n\nWhen constructing an evaluation dataset, its scale is the first issue that needs to be considered. We have thoroughly investigated the current open-sourced evaluation sets, such as FLORES-101 and the evaluation sets of WMT, and we found that the scale of these datasets is mostly in the range of 2000~3000, so we took 2500 as a comparable scale. In addition, our intention of constructing *CCEval* is to hope that as few evaluation samples as possible can be used to achieve reliable and human-like evaluation results. Therefore, we were not keen on expanding the scale of the evaluation set, but rather focused on the overall diversity of the evaluation set and carefully picked each sample, as we described in Section 2. However, we will continue to mine more valuable evaluation samples and supplement them appropriately into *CCEval*.\n\n> I would like to know the average number of words/tokens per sentence.\n\n| **Language** | zh | lo | mn | pt | pl | cs | ru | it | tr | ar | el | hr |\n| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| **Mean #tokens/sentence** | 30 | 36 | 31 | 27 | 33 | 31 | 33 | 29 | 26 | 35 | 29 | 28 |\n\n> There are many grammatical errors in the article. The authors should use a grammatical tool for correction.\n\nThank you very much for the suggestion. We have used “Grammarly” to check over the full paper carefully, and have corrected all of the found mistakes. Here are three examples:\n- Line 031: “the most population” has been corrected to “the largest population”\n- Line 187: “work flow” has been combined to “workflow”\n- Line 245: “as regard to” has been corrected to “in regard to”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors proposed a Chinese centric multilingual machine translation benchmark evaluation dataset consisting 2500 Chinese sentences. The authors also conducted experiments on the evaluation dataset to compare with existing multilingual datasets.", "reasons_to_reject": "1. In L101, the authors employed various linguistic features but didn’t discuss which linguistic features were used in this study.\n2. In Equation 1, I didn’t really understand the relation between the source sentence and the restored source sentence by the BERT model. The similarity of these two sentences really matters for the translation of a sentence by humans? In my observance, difficulties in translation might happen for long or complex sentences, because the translator lost the context sometimes.\n3. Why were long sentences over 60 characters removed?\n4. LSP’s translation can be modified while verifying the quality of the translation. Why the authors didn’t fix the translation or authors keep them on purpose is not discussed properly.\n5. 8 topics have 1 to 3 individual sentences, would it be insightful to draw a conclusion based on 1 to 3 translations for a topic?\n6. 2,500 sentences are not enough to evaluate an MMT model.", "questions_to_authors": "1. I would like to know the average number of words/tokens per sentence.", "paper_topic_and_main_contributions": "This paper presents a Chinese-centric multilingual machine translation evaluation dataset. The dataset consisting 11 languages which covers 6 domains and 67 topics.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are many grammatical errors in the article. The authors should use a grammatical tool for correction."}, "extra_text": {}}, {"review_id": "9WzawQCUgV", "reviewer": "Reviewer_fSg8", "summary": "", "questions": "A: Did you evaluate the linguistic diversity of the target sentences? It appears that the claim of high linguistic diversity is based on the diversity-based selection applied to the Chinese source only. If only the Chinese source text is verified to be diverse, then the\nclaim on line 287 that \"The dataset is highly representative and outperforms existing MMT benchmarks in terms of linguistic diversity\" must be rephrased to clarify that it applies to Chinese language only.\n\nB: Why do you consider it desirable to transition from English-centric evaluation to another evaluation setting centered around a single language, in this case Chinese? Have you considered the alternative of truly multiparallel evaluation without a central pivot language?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are very grateful for your constructive comments and suggestions. They are very helpful for us to improve the quality of our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first restated and then followed by our point-by-point responses.\n\n> The data set covers a fairly small number of languages, only 12. In particular the Flores-101, Flores-200 [1], and Tatoeba [2] datasets greatly outscale the proposed dataset.\n\nIt is true that our *CCEval* only includes 12 languages, which is not something to show off compared to FLORES-101. However, please note that the initial version of FLORES-101 only contains 2 languages (FLORES-2, [3]), and both FLORES-101 and FLORES-200 are extended versions of FLORES-2. In a similar manner, the current paper is only an initial effort of us. In the future, our *CCEval* will be expanded to include more languages according to the demands from the Chinese translation evaluation. \n\nOn the other hand, the Tatoeba corpus is similar to the OPUS corpus, meaning that its portion of evaluation datasets are split from the training datasets, and the evaluation sets have not undergone a strict human annotation and quality control process, just like our *CCEval* or FLORES series evaluation sets have done. Hence, the Tatoeba dataset is not included in our comparison.\n\nRef: [3] [The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English](https://aclanthology.org/D19-1632) (Guzmán et al., EMNLP-IJCNLP 2019)\n\n> A: Did you evaluate the linguistic diversity of the target sentences? It appears that the claim of high linguistic diversity is based on the diversity-based selection applied to the Chinese source only. If only the Chinese source text is verified to be diverse, then the claim on line 287 that \"The dataset is highly representative and outperforms existing MMT benchmarks in terms of linguistic diversity\" must be rephrased to clarify that it applies to Chinese language only.\n\nIn fact, our target sentences are partially evaluated for diversity, such as domains and topics, which are inherited from the Chinese source. Theoretically, when we decide to choose Chinese as the centric source language, with high-quality translation annotations, the linguistic characteristics of the target languages are determined. In addition, due to the characteristics of different languages, such as the distribution and amount of available corpus, the distribution of Chinese linguistic features may not be completely equivalent to the linguistic feature distribution of the target languages. Therefore, we agree that the usage of “outperforms” in line 287 is inappropriate, since the lead only refers to Chinese and should not include the diversity of other (target) languages. This sentence has been rephrased to “The dataset is highly representative, and the linguistic diversity of Chinese outperforms existing MMT benchmarks.” \n\n> B: Why do you consider it desirable to transition from English-centric evaluation to another evaluation setting centered around a single language, in this case Chinese? Have you considered the alternative of truly multiparallel evaluation without a central pivot language?\n\nJust like FLORES-101 chose English as the centric language, we chose Chinese as the center to construct the multilingual machine translation evaluation set. This is because Chinese is the language with the highest resource level among all of the 12 languages in *CCEval*, thus we can have more optimization flexibility when constructing the diversity of source sentences. In addition, the reason why we cannot use FLORES-101 to evaluate the translation performance between Chinese and other languages is that the sentences of the non-English languages in FLORES-101 are all translationese, hence their text quality might have some problems. \n\nWe didn’t consider the circumstance of multi-parallel evaluation without a central language, for two reasons: a) the translation annotation cost of building a multi-parallel dataset is on the order of $n^2$ versus the cost of *n* in our case, where *n* is the number of total languages. b) It is very difficult, if not impossible, to pick and translate enough sentences for the language pairs between two low-resource languages. \n\n> Missing References: [1] FLORES-200 dataset; [2] Tatoeba dataset \n\nWe have already cited FLORES-200 [1] and will include Tatoeba [2] in the revised paper version.\n\n> 166: \"Difficulty\" should be formatted as text, rather than a sequence of variables. Currently the text is italic and the kerning is not correct.\n\nThanks for pointing out this mistake. We will correct it in the revised paper version.\n\n> Table 3: Please add the number of sentence pairs in each data set. To fit the additional column, you can either abbreviate or rotate the column headings.\n\nSure. We will add this additional column/row in the revised paper. Below please find the number of sentence pairs for each dataset listed in Table 3:\n| **Dataset** | FLORES-101/200 | ALT | TICO-19 | CCEval |\n| ------ | :------: | :------: | :------: | :------: | \n| **#Sentence pairs** | 3001 | 20000 | 3071 | 2500 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A high-quality machine translation evaluation data set such as this is valuable to the research community focusing on the Chinese language.", "reasons_to_reject": "The data set covers a fairly small number of languages, only 12.\nIn particular the Flores-101, Flores-200 [1], and Tatoeba [2] datasets greatly outscale the proposed dataset.", "questions_to_authors": "A: Did you evaluate the linguistic diversity of the target sentences? It appears that the claim of high linguistic diversity is based on the diversity-based selection applied to the Chinese source only. If only the Chinese source text is verified to be diverse, then the\nclaim on line 287 that \"The dataset is highly representative and outperforms existing MMT benchmarks in terms of linguistic diversity\" must be rephrased to clarify that it applies to Chinese language only.\n\nB: Why do you consider it desirable to transition from English-centric evaluation to another evaluation setting centered around a single language, in this case Chinese? Have you considered the alternative of truly multiparallel evaluation without a central pivot language?", "paper_topic_and_main_contributions": "The work presents a machine translation evaluation data set.\nThe data set is Chinese-centric, paired with 11 other languages: Mongolian, Lao, Portuguese, Polish, Czech, Turkish, Italian, Russian, Arabic, Greek, and Croatian.\nThe dataset evenly covers 6 domains, and is filtered by hand to ensure quality.\nTranslations are performed by professional translators, followed by a quality control procedure.\nWhile the resulting dataset is 12-way multiparallel, it can be considered Chinese-centric due to the direction of translation: the source text is always Chinese.", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The Flores-200 dataset is one of multiple contributions in the NLLB paper [1]\n[1] Costa-jussà, Marta R., et al. \"No language left behind: Scaling human-centered machine translation.\" arXiv preprint arXiv:2207.04672 (2022).\n\n[2] Tiedemann, Jörg. \"The Tatoeba Translation Challenge–Realistic Data Sets for Low Resource and Multilingual MT.\" Proceedings of the Fifth Conference on Machine Translation. 2020.", "Typos_Grammar_Style_and_Presentation_Improvements": "166: \"Difficulty\" should be formatted as text, rather than a sequence of variables. Currently the text is italic and the kerning is not correct.\n\nTable 3: Please add the number of sentence pairs in each data set. To fit the additional column, you can either abbreviate or rotate the column headings."}}, {"review_id": "lmwlQS1Nbx", "reviewer": "Reviewer_Ucq3", "summary": "", "questions": "- It is suggested to validate the effectiveness of this benchmark on actual machine translation models.\n- It is recommended to provide some sample data as evidence.\n- Please provide more annotation details, such as the number of annotators, educational background, salary, and other relevant information.\n- More analysis could be provided characterizing the dataset's diversity and comparability to existing benchmarks. The representativeness for intended use cases could be clarified.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors sincerely appreciate your effort and time spent reviewing our paper. Thanks a lot for your constructive comments and encouraging words. They are very helpful in improving the quality of our paper. We have carefully incorporated them in the revised paper. Our point-by-point responses to your comments are given below.\n\n> The scale of the dataset (2,500 sentences) is still quite small for thoroughly evaluating MMT models. Expanding to more target languages and data would be beneficial.\n\nWe have thoroughly investigated the current open-sourced evaluation sets, such as FLORES-101 and the evaluation sets of WMT, and we found that the scale of these datasets is mostly in the range of 2000~3000, so we took 2500 as a comparable scale. In addition, our intention of constructing *CCEval* is to hope that as few evaluation samples as possible can be used to achieve reliable and human-like evaluation results. Therefore, we were not keen on expanding the scale of the evaluation set, but rather focused on the overall diversity of the evaluation set and carefully picked each sample, as we described in Section 2. However, we will continue to mine more valuable evaluation samples and supplement them appropriately into *CCEval*.\n\nRegarding the number of included languages, the current paper is only an initial effort of us. In the future, we will continue to expand the dataset and include more languages according to the demands of the Chinese translation evaluation. \n\n> - The author did not actually use this evaluation benchmark to test the machine translation model. \n> - It is suggested to validate the effectiveness of this benchmark on actual machine translation models.\n\nWe have actually conducted translation evaluation experiments on our *CCEval* benchmark, using two well-known multilingual translation models, but we were not able to incorporate this section into the current paper due to the page limit of the initial version (4 pages in total). In the revised paper version, given an additional page, we will add this part.\n\nWe use SacreBLEU as the evaluation metric, and our evaluation results of SacreBLEU are shown in the table below:\n| **Language** | lo | mn | pt | pl | cs | ru | it | tr | ar | el | hr |\n| ------ | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | \n| **M2M-100, from Chinese** | 1.13 | 0.97 | 18.41 | 13.62 | 13.19 | 17.47 | 17.73 | 15.75 | 15.99 | 16.94 | 13.33 | \n| **M2M-100, to Chinese** | 6.59 | 7.14 | 19.18 | 18.96 | 19.85 | 23.71 | 22.26 | 23.79 | 21.17 | 21.87 | 22.92 | \n| **mBART, from Chinese** | — | 1.03 | 16.31 | 10.24 | 9.98 | 15.45 | 14.21 | 14.21 | 14.22 | 13.22 | 13.01 | \n| **mBART, to Chinese** | — | 8.87 | 15.77 | 13.87 | 17.24 | 21.98 | 18.93 | 18.44 | 19.43 | 19.7 | 19.89 |\n\nThe main findings of the evaluation above are: \n- In general, the translation SacreBLEUs from Chinese to other languages are significantly lower than the reverse direction, i.e., from other languages to Chinese; \n- The translation SacreBLEUs between low-resource languages and Chinese are generally far lower than those between high- or medium-resource languages and Chinese.\n\n> The empirical verification of the difficulty metric's effectiveness is indirect, performed on different public/private datasets rather than CCEval itself. It lacks sufficient persuasiveness and needs to provide more evidence.\n\nAs we have already mentioned in Sec. 3.1 and the “Limitations” section, there is indeed a lack of human-annotated DA/MQM data to support our effectiveness verification of the difficulty-based sampling method on our *CCEval* evaluation datasets. Currently, we have tried our best and verified it on all available DA/MQM data, including two open-sourced WMT evaluation sets, as well as our own Chinese-to-Mongolian test set.\n\n> - No data examples were provided.\n> - It is recommended to provide some sample data as evidence.\n\nWe provide one data sample for each of the 11 language pairs. As shown in the table below, all of our evaluation samples contain rich metadata information, such as language, domain, topic, etc.\n| Chinese source text | Target language | Target text  | Domain | Topic |  \n| ------ | :------: | ------ | :------: | :------: |  \n| 来自加州喷气推进实验室的Robert Carlson表示，一种更好的混合原料是氨和碳氢乙炔。| ru | \"Robert Carlson из калифорнийской Лаборатории реактивного движения сказал, что лучшим сырьём для смеси является аммиак и ацетилен.\" | Technology | Aerospace | \n| 发展改革委、生态环境部有关司局负责同志对清理整改工作提出了要求。 | pt | \"Os responsáveis relevantes de gabinetes e departamentos, tais como a Comissão Nacional de Desenvolvimento e Reforma e o Ministério da Ecologia e Ambiente, propuseram os requisitos para a limpeza e rectificação.\" | Technology | Water Conservancy | \n| 嘿嘿，路边有一个不大引人注意的广告牌，上面有个女生正专注地看着墙上的招租广告呢。 | el | \"Υπάρχει ένας διαφημιστικός πίνακας στην άκρη του δρόμου που δεν τραβάει την προσοχή πολλών ανθρώπων, στον οποίο ένα κορίτσι παρακολουθεί προσεκτικά τη διαφήμιση ενοικίασης στον τοίχο.\" | Travel | Travel Notes | \n| 建议大家带报纸去坐在店外面吃，别有一番风味。 | mn | Ийм нэгэн онцлогтой тул та бүхнийг сонин авчирч дэлгүүрийн гадаа сууж хооллохыг зөвлөж байна. | Travel | Travel Guide | \n| 打扰一下，您能告诉我黄油在哪儿卖吗？ | tr | \"Affedersiniz, tereyağının nerede satıldığını söyleyebilir misiniz? \" | Daily Life | Shopping |\n| 按营业额百分比进行罚款可以提高法律对相关企业违法行为的威慑力。 | it | La messa a punto in base alla percentuale di fatturato può aumentare il potere deterrente della legge alle attività illegali delle imprese interessate.  | News | Legal |\n| 重拳打击涉黑涉恶、电信诈骗、涉枪涉爆、黄赌毒、拐卖妇女儿童等人民群众反映强烈的突出违法犯罪。 | pl | \"Uderzanie w gangsterstwo i zło, oszustwa telekomunikacyjne, broń palną i wybuchową, żółte i hazardowe, narkotykowe, handel kobietami i dziećmi oraz inne, podkreślają nielegalne przestępstwa mocno pięścią, co opinia publiczna mocno odzwierciedla.\" | News | Legal | \n| 长时辰运用泻药，可使胃肠道对泻药发作依靠性，除了为解一时之急，最佳仍是少用或不用泻药。 | cs | \"Dlouhodobé užívání projímadel může vést k závislosti gastrointestinálního traktu na projímacích záchvatech. Nejlepší je používat projímadla střídmě nebo je nepoužívat vůbec, s výjimkou dočasných naléhavých případů.\" | Biomedical | Pediatrics | \n| 建议可以将蔬菜和水分一起榨汁饮用，这样既能够摄取蔬菜中的养分，同时又能补充水分。 | lo | \"ແນະນຳວ່າຜັກ ແລະ ນ້ຳສາມາດເອົາຄັ້ນນ້ຳມາດື່ມໄດ້, ເຊິ່ງໃຫ້ສານອາຫານໃນຜັກສາມາດດູດຊຶມໄດ້ ແລະ ສາມາດເພີ່ມນ້ຳໄດ້ພ້ອມໆກັນ.\" | Biomedical | Obstetrics and Gynecology | \n| 对在保护传承工作中作出突出贡献的组织和个人，按照国家有关规定予以表彰、奖励。 | ar | تقديم التقديرات والمكافآة لأفراد أو منظمات التي تقدم مساهمة بارزة في أعمال الحماية والميراث وفقا للوائح الوطنية ذات الصلة. | Politics | Government File | \n| 完善最高人民法院巡回法庭工作机制，健全综合配套措施。 | hr | Unaprijediti mehanizam rada okružnog suda Vrhovnog narodnog suda kao i sveobuhvatne mjere podrške. | Politics | Government File | \n\n> - Lacks detailed information about data annotation.\n> - Please provide more annotation details, such as the number of annotators, educational background, salary, and other relevant information.\n\nWe have obtained a complete profile of the annotator details, and will add the required information for each language into the revised paper version. As an example, for each language, we have 3-6 translation annotators.\n\n> More analysis could be provided characterizing the dataset's diversity and comparability to existing benchmarks. The representativeness for intended use cases could be clarified.\n\nWe have actually conducted extensive analysis on this, comparing the diversity of our *CCEval* benchmark with that of FLORES-101 dataset from multiple linguistic dimensions, including the distributions of sentence length, word frequency, and grammar complexity. But we were not able to incorporate this part into the current paper due to the page limit of the initial version (4 pages in total). In the revised paper version, given an additional page, we will add this part, including the comparison graph.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- CCEval fills an important gap as the first open benchmark tailored for Chinese-centric MMT, which is useful.\n- The paper provides sensible methods and analysis to construct a high-quality diverse dataset. The difficulty-based sample selection is empirically shown to improve human correlation.\n- Releasing the dataset and providing an online evaluation platform is valuable for the research community. The dataset has rich metadata for traceability.", "reasons_to_reject": "- The scale of the dataset (2,500 sentences) is still quite small for thoroughly evaluating MMT models. Expanding to more target languages and data would be beneficial.\n- The author did not actually use this evaluation benchmark to test the machine translation model.\n- The empirical verification of the difficulty metric's effectiveness is indirect, performed on different public/private datasets rather than CCEval itself. It lacks sufficient persuasiveness and needs to provide more evidence.\n- No data examples were provided.\n- Lacks detailed information about data annotation.", "questions_to_authors": "- It is suggested to validate the effectiveness of this benchmark on actual machine translation models.\n- It is recommended to provide some sample data as evidence.\n- Please provide more annotation details, such as the number of annotators, educational background, salary, and other relevant information.\n- More analysis could be provided characterizing the dataset's diversity and comparability to existing benchmarks. The representativeness for intended use cases could be clarified.", "paper_topic_and_main_contributions": "This paper proposes a new Chinese-centric multilingual machine translation (MMT) evaluation benchmark dataset called CCEval. The main contributions are:\n\n- CCEval is a dataset designed specifically for evaluating Chinese-centric MMT models. It contains 2,500 Chinese source sentences translated into 11 target languages.\n- The authors propose methods to select a diverse and representative set of Chinese source sentences based on linguistic features like topic, word frequency, grammar complexity, and translation difficulty.\n- The dataset covers a wide range of domains and low/high resource target languages. Quality control measures were taken to ensure high translation quality.\n- Experiments show that their proposed translation difficulty metric is effective at selecting sentences that correlate better with human assessments for system ranking.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper work presents a Chinese-centric machine translation evaluation dataset paired with 11 other languages. It covers 6 domains and is filtered by hand to ensure quality. This is a valuable resource since most other evaluation datasets are either not curated or not Chinese-centric, but it remains unclear if the proposed resource provides any practical advantages over other evaluation datasets like Flores or Tatoeba.

During the author response period, the authors showcased the effectiveness of their benchmark on actual machine translation models, and they also promised to provide a comparison between their dataset and Flores, which hopefully addresses some criticisms made by the reviewers.
Decision: Accept-Findings