Evaluating Factuality in Cross-lingual Summarization
Year: 2023

Abstract:
Cross-lingual summarization aims to help people efficiently grasp the core idea of the document written in a foreign language. Modern text summarization models generate highly fluent but often factually inconsistent outputs, which has received heightened attention in recent research. However, the factual consistency of cross-lingual summarization has not been investigated yet. In this paper, we propose a crosslingual factuality dataset by collecting human annotations of reference summaries as well as generated summaries from models at both summary level and sentence level. Furthermore, we perform the fine-grained analysis and observe that over 50% of generated summaries and over 27% of reference summaries contain factual errors with characteristics different from monolingual summarization. Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of crosslingual summarization and perform differently at different tasks and levels. Finally, we adapt the monolingual factuality metrics as an initial step towards the automatic evaluation of summarization factuality in cross-lingual settings. Our dataset and code are available at https: //github.com/kite99520/Fact_CLS.

heading: 1 Introduction
text: Cross-lingual summarization, the task of generating a summary in different languages from the source documents, aims to help people efficiently grain the main point of the original document. It is recognized as a challenging task that combines the difficulties of text summarization as well as machine translation. Traditional pipeline methods first translate the document and then summarize it in the target language or vice versa (Leuski et al., 2003; Orǎsan and Chiorean, 2008; Wan et al., 2010; Wan, 2011; Yao et al., 2015). Currently, modern neural cross-lingual summarization models have
*Equal contribution.
witnessed rapid growth in recent research (Shen et al., 2018; Duan et al., 2019; Zhu et al., 2019, 2020; Cao et al., 2020).
Factuality, a crucial dimension, is absent from the current evaluation of cross-lingual summarization approaches. ROUGE (Lin, 2004) is the main automatic evaluation metric. Informativeness, fluency, and conciseness are the dimensions of human evaluation. However, many case studies have pointed out that the summaries generated by neural cross-lingual summarization models have factual errors (Zhu et al., 2019, 2020; Bai et al., 2021). Table 1 also shows the state-of-the-art cross-lingual summarization models generate factually incorrect summaries. A variety of factuality evaluation metrics have drawn close attention in monolingual summarization (Kryscinski et al., 2020; Maynez et al., 2020; Goyal and Durrett, 2021), yet so far no study has comprehensively studied the factuality of crosslingual summarization.
To fill the gap, we collect summaries from six models on a cross-lingual summarization dataset
12415
proposed by Zhu et al. (2019) and obtain the human judgments of fine-grained factuality. The result of human evaluation suggests that over half of the generated summaries and over 27% of reference summaries contain at least one factual error. During the annotation process, we identify the peculiarity of factual errors in cross-lingual summaries, such as translation-related errors. Further, since the existing monolingual factuality metrics require the aid of translation to use in cross-lingual settings, after analyzing their performance, we explore the challenging automatic evaluation of factuality in cross-lingual summarization. In summary, our contributions are as follows:
• We propose a cross-lingual factuality dataset by collecting fine-grained human annotations over references as well as the outputs of six cross-lingual summarization systems at both summary level and sentence level. The dataset will be released and contribute to future crosslingual summarization research.
• We introduce a typology of factual errors and conduct a fine-grained analysis of the factuality of the summaries and the performance of existing metrics. To the best of our knowledge, this is the first work to analyze the factuality of cross-lingual summarization.
• We adapt the monolingual factuality metrics as an initial step towards automatic factuality assessment in cross-lingual summarization.

heading: 2 Related Work

heading: 2.1 Cross-lingual Summarization
text: Early explorations on cross-lingual summarization are pipeline methods that simply integrate machine translation into monolingual summarization and achieve some improvement through incorporating bilingual parallel information. (Leuski et al., 2003; Orǎsan and Chiorean, 2008; Wan et al., 2010; Wan, 2011; Yao et al., 2015). Recently, neural-based methods have been applied to crosslingual summarization (Shen et al., 2018; Duan et al., 2019; Zhu et al., 2019, 2020; Cao et al., 2020). Shen et al. (2018) first propose the neuralbased cross-lingual summarization system with a teacher-student framework. Similarly, Duan et al. (2019) improve the teacher-student framework by using genuine summaries paired with the translated pseudo source sentences for training. Zhu et al.
(2019) propose a multi-task learning framework, which incorporates monolingual summarization or machine translation into cross-lingual summarization training process. A concurrent work by Zhu et al. (2020) improves the performance by combining the neural model with an external probabilistic bilingual lexicon. Cao et al. (2020) propose a multi-task framework with two encoders and two decoders that jointly learns to summarize and align context-level representations.

heading: 2.2 Factuality Evaluation in Summarization
text: There are many analyses and meta-evaluations for factuality in monolingual summarization (Maynez et al., 2020; Pagnoni et al., 2021; Gabriel et al., 2021). Cao et al. (2017) reveal nearly 30% of the outputs from a state-of-the-art neural summarization system contain factual errors. Similarly, Falke et al. (2019) conduct the initial crowdsourcing of binary factual annotations and find that nearly 25% of the generated summaries are factually inconsistent.
In terms of evaluation metrics, the most commonly used ones based on n-gram overlap like ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and METEOR (Agarwal and Lavie, 2008) are insufficient to measure the factual consistency of summaries and fail to correlate with the human judgments of factuality (Falke et al., 2019; Kryscinski et al., 2019). Yuan et al. (2021) convert the evaluation task to a conditional generation task and utilize the generation probability of the pre-trained language model BART (Lewis et al., 2020) to estimate the quality of system output, including faithfulness. Further, several works have explored using natural language inference (NLI) models to evaluate the factuality of summaries (Falke et al., 2019; Kryscinski et al., 2020; Maynez et al., 2020). In addition, Durmus et al. (2020) and Wang et al. (2020) evaluate factual consistency through question generation and question answering models. All the above metrics can not be used directly in crosslingual settings.

heading: 3 Typology of Factual Errors
text: We define a typology of ten factual errors by analyzing both reference summaries and generated summaries. An example for each error type is shown in Table 2. Hallucination Error (HalE): This occurs when the events not directly inferable from the input doc-
ument are added to the summaries. Particulars Error (ParE): This occurs when the summary contains the major events of the source document, but some details are inaccurate or mistaken, like time, location and direction. Predicate Error (PreE): This occurs when the predicate in the summary is contradictory to the source document. Entity Error (EntE): This occurs when the entity of an event is wrong, including substitution, addition and deletion cases. Coreference Error (CorE): This occurs when pronouns and other references to the aforementioned entities are either incorrect or ambiguous. Incompleteness Error (IncE): This occurs when the word [UNK] is presented in the summary.
The above factual error types are from monolingual summarization (Pagnoni et al., 2021; Wang et al., 2022). Considering the specificity of crosslingual summarization, three error types in machine translation (denoted as MTE) are added after some case studies. Tense Error (TenE): This occurs when the tense of the summary is inconsistent with the source document, which is common in machine translation as the natural differences in tense expression between Chinese and English. Tenses in English can be directly indicated through predicate verbs, while they are not clearly marked in Chinese (Shi, 2021). Plural Error (PluE): This occurs when the summary changes the singular or plural forms of nouns in the source document. English nouns focus on the concept of singular and plural, while nouns in Chinese are usually replaced by flexible and fuzzy quantitative expressions (Xiao, 2013).
Terminologies Error (TerE): This occurs when the terminologies in the source document cannot be expressed professionally or accurately in summary. Li and Feng (2020) find that terminologies error ranks first among the high-frequency errors in machine translation.
Finally, we add the additional type Other Error (OthE) to ensure the completeness of the typology. This occurs when the error does not correspond to any of the above types.

heading: 4 Data Annotation

heading: 4.1 Dataset and Model
text: We annotate samples from the cross-lingual summarization datasets released by Zhu et al. (2019), which includes an English-to-Chinese (En-toZh) dataset and a Chinese-to-English (Zh-to-En) dataset. The Chinese summaries of the En-to-Zh dataset are translated correspondingly from the union set of CNN/DM (Hermann et al., 2015) and MSMO (Zhu et al., 2018). The Zh-to-En dataset is constructed from the Chinese LCSTS dataset (Hu et al., 2015).
Based on the dataset, we collect generated summaries from six models. Since Wan et al. (2010) have shown that summarize-then-translate is preferable to avoid both the computational expense of translating more sentences and sentence extraction errors caused by incorrect translations, we first use PGN (See et al., 2017), a monolingual summarization model to generate the summaries 1, and a
1CNN/Dailymail (https://github.com/abisee/ pointer-generator) and LCSTS (https://github.com/ LowinLi/Text-Summarizer-Pytorch-Chinese). Prior studies had trained PGNs on the original monolingual
translation model 2 is then applied to translate the summary. To compare the impact of different translation systems on summarization, we also use a commercial translator Youdao 3, during the process of translation. We refer to these two pipeline methods as Pipe-ST and Pipe-ST* respectively. We also collect outputs from four neural cross-lingual summarization models. TNCLS (Zhu et al., 2019) trains a standard Transformer model on the parallel corpora in an end-to-end manner. CLSMS (Zhu et al., 2019) combines the cross-lingual summarization task with monolingual summarization and calculates the total losses. Similarly, CLSMT (Zhu et al., 2019) combines cross-lingual summarization with machine translation. They both use one encoder and multiple decoders for multi-task frameworks. ATS (Zhu et al., 2020) is another Transformer-based model that utilizes a pointergenerator network to exploit the translation patterns in cross-lingual summarization.
100 documents are randomly sampled from the test set of En-to-Zh and Zh-to-En corpus respectively with the corresponding model-generated summaries. Each summary is manually split into sentences.

heading: 4.2 Annotation Procedure
text: We recruit eight college students with qualification certificates who are fluent in both English and Chinese languages, with Chinese as their mother tongue. They are provided with an annotation guideline. Further, we design a qualification test consisting of 10 document-summary pairs, only annotators who pass the test are considered to be qualified and are allowed to continue annotation. To ensure the annotation quality, we set the number of tasks each annotator needs to complete each day. After receiving the results of the day, we check the results and provide feedback.
In the annotation interface, we show the full source document on the left and a summary sentence by sentence on the right. Seven summaries are listed for each document in random order, including one translated reference summary and six model-generated summaries. The fine-grained annotations are a three-step process: For each sentence in a summary, annotators first determine
summarization datasets. We just got the generated summaries. 2https://huggingface.co/Helsinki-NLP/ opus-mt-zh-en/tree/main and https://huggingface. co/Helsinki-NLP/opus-mt-en-zh/tree/main
3https://fanyi.youdao.com
whether it is factual or not. If a sentence is marked as not factual, annotators identify the error types based on our typology. A sentence can be annotated with multiple types. Finally, a Likert Scale from 1-5 is used to rate the overall factuality of the summary.
Each sample is annotated by two distinct annotators. For the sentence-level binary label, a third annotator from us makes the final decision if they are in disagreement. For the error types of each sentence, the intersection and union of two annotators are both collected. For the summary-level annotation, we take the average score of two annotators as the final result.

heading: 4.3 Inter-annotator Agreement
text: Table 3 shows the nearly perfect inter-annotator agreement on two datasets. For sentence-level annotation, we obtain an average agreement of Cohen’s kappa (Cohen, 1960) with κ=0.891. For the summary level, we obtain the agreement of Krippendorff’s alpha (Krippendorff, 1970) with α=0.903 on average. More annotation details can be found in Appendix F.

heading: 5 Fine-grained Factuality Analysis

heading: 5.1 Factuality of Reference Summaries
text: The two cross-lingual summarization datasets were originally constructed in a two-step way: (1) Given a source document, a reference summary in the same language was written by humans or crawled from their titles. (2) Then the summary was translated into another language by an automatic translator 4, maybe followed by a manual correction. The following analysis shows that the constructed datasets are error-prone and the factual errors can be introduced at both steps. Error Proportion. Table 4 reports the annotation results on reference summaries from cross-lingual summarization datasets. We discover that 27%- 50% single sentences contain at least one factual
4http://www.anylangtech.com
error. However, the most popular n-gram based evaluation metrics like ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002) only utilize the reference summary to evaluate the quality of the modelgenerated summary. The reference with poor quality will undermine the reliability of their evaluation under cross-lingual settings. We encourage future researchers to be aware of the issue, especially for evaluation. Error Types. Both the intersection and union of two annotators are in a similar distribution that HalE and ParE occur most frequently. Since references are abstractive and do not simply copy sentences from the documents, it is natural to incorporate the author’s background knowledge (van Dijk and Kintsch, 2014; Brown and Day, 1983), e.g.:
Document: 失踪的女生身高158厘米左右，体重 约90斤，皮肤白皙。9月3日失踪当天手提两大袋东 西。当日她从厦门去福州，中午12点36分和她通话 后再也联系不上。警方调取福州金山公交总站附近 监控发现，她在附近上了一辆出租车。(The missing girl is about 158 cm tall, weighing about 90 pounds, with fair skin. She was carrying two large bags of stuff on the day she went missing, September 3. That day she went from Xiamen to Fuzhou, but could not be reached after calling her at 12:36 PM. The police retrieved the surveillance near Fuzhou Jinshan bus terminal and found that she got into a cab in the vicinity.) Reference: Xiamen 23-year-old girl disappeared after she went to Fuzhou to find her classmates with a taxi.
(Zh-to-EnSum, HalE)
Particularly, such hallucination may not always be erroneous. The information not entailed by the source in the above example is consistent with the relevant introduction in Wikipedia. It remains controversial whether this kind of hallucination should be allowed (Maynez et al., 2020) because it is difficult to verify whether it is factual outside of the source document. Task Comparisons. We notice the difference between the two tasks of summarization: The factuality of references in En-to-Zh task is better than Zh-to-En task on both average score and er-
ror proportion. The reasons are two-fold: (1) In En-to-Zh task, the references of original English dataset are manually-written highlights offered by the news providers (Hermann et al., 2015; Zhu et al., 2018). While in Zh-to-En task, the Chinese dataset is constructed from a microblogging website and the crawled references are headlines or comments, which are generally more error-prone as they usually contain rhetoric to attract readers. An example is shown in Table 10 in Appendix G. (2) In En-to-Zh task, the dataset belongs to the news domain and existing machine translation for news reports has reached human parity (Hassan et al., 2018). While the dataset in Zh-to-En task comes from social media, the proportion of abbreviations, omitted punctuation, and catchphrases in the text is much higher than in news, resulting in lower translation quality. An example is shown in Table 11 in Appendix G.

heading: 5.2 Factuality of System Outputs
text: Error Proportion. Figure 2 visualizes the proportion of summaries with factual errors for each model and dataset. We observe that over 50% summaries contain factual errors, with the best model (CLSMT) generating 52.6% inconsistent summaries in En-to-Zh and 53.0% in Zh-to-En task. Similar observations have been made in monolingual summarization where 30%-80% of generated texts are factually inconsistent (Cao et al., 2017; Pagnoni et al., 2021). Error Types. Error distributions in system outputs are shown in Figure 1. As in the reference, models also produce HalE and ParE error types with highest proportion.
For three error types that occur frequently in machine translation, we notice the proportion in Zh-to-En task (18.49%) is higher than that of Ento-Zh task (3.24%) showing the natural differences between the two languages. The comparison in IncE is more apparent. In Zh-to-En task, models
seldom generate UNK because the subword-based tokenization makes the vocabulary list smaller (Zhu et al., 2019). Additionally, OthE makes up a very small percentage (less than 3%) of errors showing that our typology is relatively complete.
It is worth noting that compared with the reference, the model-generated summaries contain more EntE, suggesting that models tend to confuse the role of each entity in an event, such as the subject and object, e.g.: Four entities mismatch in the generated summary. Since the document contains 40 entities in total, it is challenging for models to accurately locate the logical correlation between different parts when multiple entities appear.
Document Fragment: (CNN) – Rafa Benitez’s turbulent reign as Chelsea manager took another battering on the day his supposed successor, Pep Guardiola, agreed a deal to become the new manager of Bayern Munich. Benitez, who was appointed as Chelsea interim manager in November following the dismissal of Roberto Di Matteo, was left stunned after his team squandered a two-goal lead to draw 2-2 with lowly Southampton. European champion Chelsea is now 13 points adrift of Premier League leader Manchester United with 16 games remaining and has failed to win any of their past three games at Stamford Bridge. Guardiola agrees three-year deal with Bayern. [. . . ] Summary: 拉法·贝尼特斯同意与拜仁慕尼黑签署 一份为期三年的合同。切尔西以2 - 2 战平南安普 顿，贝尼特斯被解雇。西班牙人现在在英超联赛中 落后切尔西13分。(Rafa Benitez agreed to sign a threeyear contract with Bayern Munich. Chelsea drew 2-2 with Southampton and Benitez was dismissed. European champion Chelsea is now 13 points adrift of Premier League leader. The Spanish team is now 13 points adrift of Chelsea in Premier League.)
(En-to-ZhSum, EntE)
on Zh-to-En and En-to-Zh tasks except TNCLS. In contrast, the summary-level average scores in Zh-to-En task are generally higher than that in Ento-Zh task for each model as shown in Figure 3. One possible reason is that the summary-level average scores are influenced by the relationship between sentences. Note that En-to-Zh summaries are longer, with three sentences on average, while Zh-to-En summaries only contain one single sentence. Factual errors of conjunctions may exist in Eh-to-Zh summaries. Model Comparisons. For traditional pipelinebased methods, Pipe-ST* outperforms Pipe-ST in both En-to-Zh and Zh-to-En tasks, suggesting the impact of different translators on factuality. We also notice that the two pipeline-based methods account for nearly half of the translation-related errors, TenE, PluE, and TerE (50.0% in En-to-Zh and 53.7% in En-to-Zh), probably because machine translation is one step of the pipeline. Table
3 shows that Modern neural-based cross-lingual summarization models generate more factual summaries than pipeline methods, but the average is not great. Table 6 in Appendix B reports the inconsistencies in ROUGE and factuality scores of the models.

heading: 6 Automatic Evaluation of Factuality

heading: 6.1 Existing Evaluation Metrics
text: ROUGE (Lin, 2004) is the most commonly used reference-based evaluation metric in summarization. The ROUGE score is usually used as a measure of overall quality.
The following monolingual factuality metrics use the source document and the summary to be evaluated as the inputs. To apply them in the crosslingual scenarios, we use an automatic translator to translate the summaries in En-to-Zh tasks and the source documents in Zh-to-En tasks from Chinese into English. Please see Appendix C for more details.
BARTScore (Yuan et al., 2021) use the probability that BART (Lewis et al., 2020) generates the hypothesis given the source text to measure the faithfulness of the hypothesis.
FactCC (Kryscinski et al., 2020), a BERT-based model trained on a synthetic dataset to classify text pairs as factually consistent or inconsistent at sentence level.
DAE (Goyal and Durrett, 2021), an ELECTRAbased model that classifies each dependency arc in the model output as entailing the source text or not at dependency level.
QuestEval (Scialom et al., 2021), a QA-based metric that introduces question weighting and negative sampling into the training process to evaluate the factuality of text.

heading: 6.2 Exploration in Cross-lingual Settings
text: The existing monolingual metrics require the use of a translator, which is inconvenient. To make them directly usable in cross-lingual settings, we make the following attempts:
(1) For BARTScore, we replace the original checkpoint with mBART-50 (Tang et al., 2020) and use the multilingual version of tokenizer to allow multilingual inputs. We call it mBARTScore.
(2) For FactCC, we adapt its data augmentation methods of constructing synthetic data to generate cross-lingual data: In En-to-Zh task, sentences from the source document in English are
extracted. The sentences themselves are used as positive claims, and the pronouns, entities, dates, numbers, and negatives in them are replaced by other words of the same type in the source document as negative claims. The positive and negative claims are translated into Chinese by an automatic translator. Finally, the translated claims are combined with the source document in English. In Zh-to-En task, the source documents in Chinese are first translated into English, and then data augmentation is applied to the translated source documents to construct claims in the same way. Finally, the claims are combined with the source document in Chinese. The source document and sentences are concatenated and fed to mBERT (Devlin et al., 2019) to train binary classification. The models trained on the synthetic data of the two tasks separately are denoted as mFactCC-split. The models trained by mixing the synthetic data of the two tasks are denoted as mFactCC-mix. More details can be found in Appendix D.

heading: 6.3 Correlation with Human Evaluation
text: To measure the correlation between metrics and human judgments, we compute the Pearson correlation coefficients r and Spearman’s rank correlation coefficients ρ respectively at both system level and summary level. Furthermore, we also compute the binary classification accuracy of single sentences. we have the following findings from Table 5:
The performance of the metrics varies considerably across tasks and levels. All the existing metrics exhibit a higher correlation with human judgments in En-to-Zh task than that in Zh-to-En task. The correlations of the metrics are lower at summary level than at system level. The performance of the metrics differs relatively little at sentence level.
Compared to ROUGE, there is an advantage of the factuality metrics but it is not significant. Although ROUGE does not obtain the best performance at any level except system level of Zh-to-En task, its system-level correlation is close to the best results. Considering the existing factuality metrics need to be used with the help of translators and the translation process also introduces uncertainties and errors, it is challenging to introduce monolingual factuality metrics in cross-lingual summarization. However, this does not mean that the current evaluation mechanism does not need improvement, as we have illustrated the shortcomings of refer-
ence summaries in Section 5.1. The reason for ROUGE’s good performance may be related to how the dataset is constructed. Specifically, the training and test sets are not constructed in exactly the same way: the reference summaries of the training set are obtained through automatic translation, while the reference summaries of the test set are obtained through a combination of automatic translation and manual post-editing. The manual post-editing is likely to correct some obviously incorrect phrases. Summarization models fit the distribution of the training set, and errors in their outputs may be easier to capture when compared to the reference summaries.
mBARTScore and mFactCC adapted by us perform poorly. This suggests that it is challenging to evaluate the factuality of summaries in crosslingual settings. We observe that there is a big difference between the synthetic claims and modelgenerated summaries. For future work, it is possible to fine-tune mBARTScore or design other data
augmentation approaches.

heading: 6.4 Identification of Factual Error Types
text: To inspect capabilities of metrics identifying different types of factual errors, we use the result of the original correlation minus the correlation after ignoring the summaries with an error type as the measure. For each metric, we consider the three most frequent types HalE, ParE, ObjE, as well as MTE, and plot the contribution of error types to the overall correlation in Figure 4. A higher value indicates the better capabilities of the metric to capture the corresponding error types.
Similar to our discovery in Section 6.3, each metric exhibits a great difference between the two tasks. For example, almost all metrics correlate well with MTE in Zh-to-En task but have a negative correlation with MTE in En-to-Zh task. Figure 4 also reveals great limitations of factuality metrics in detecting different types of factual errors. Taking the entailment-based metrics as examples, DAE shows better ability at identifying HalE while FactCC has
a negative correlation. Nevertheless, FactCC has the highest correlation with EntE in both tasks, showing the effectiveness of entity swapping transformation of its data augmentation to capture entity errors.

heading: 7 Conclusion
text: In this work, we comprehensively evaluate and analyze the factuality of reference summaries and model-generated summaries in cross-lingual summarization, showing that there are special factual errors in them. Automatic evaluation of cross-lingual summarization is yet to be addressed due to the shortcomings of reference summaries and the limitations of monolingual factuality metrics. Moreover, our exploration of the automatic factuality evaluation in cross-lingual settings illustrates its challenging nature.
Limitations
The scenarios we studied are limited to Chinese to English and English to Chinese. For other languages, the factual characteristics may be different. The genre of the source documents we study is news or blog post. For other genres, such as dialogue, our conclusion may not apply.
The number of systems we selected is limited, so there is some chance of system-level evaluation of evaluation metrics.
Ethics Statement
We recruit annotators from a college campus. They are completely free to decide whether or not to participate in our annotation. The payment is 9 dollars per hour, higher than the local minimum wage. There is no personal information in our collected dataset. The information which may be used to identify the participants is deleted after the annotation.
The model-generated summaries may contain toxic language, which can make annotators uncomfortable. We reviewed the data before annotation and found no problematic samples.
We check the licenses of the artifacts used in this study and do not find conflicts. The license of the dataset we will release is CC BY-NC 4.0.

heading: Acknowledgements
text: This work was supported by National Key R&D Program of China (2021YFF0901502), National
Science Foundation of China (No. 62161160339), State Key Laboratory of Media Convergence Production Technology and Systems and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.

heading: A Union proportion of model generated error types
text: It can be seen from Figure 5 that the union annotation of error types have the same distribution with intersection result, i.e., HalE and ParE are the most frequent, followed by EntE, PreE. Particularly, the proportion of OthE in union(10.99%) is much higher than that in intersection(3.39%), suggesting the influence of subjective factors on the determination of error types.

heading: B ROUGE F1 and factuality scores of cross-lingual summarization models
text: Table 6 reports ROUGE F1 scores and the manually annotated factuality score of each model.

heading: C Details of existing metrics
text: For ROUGE, we use the Py-rouge package 5. All parameters are listed in Table 7. For Chinese text, we insert spaces between Chinese characters as pre-processing.
For monolingual factuality metrics, we use the same translator6 as used in Section 4.1 to translate the summaries or the source documents.
5https://github.com/Diego999/py-rouge 6https://huggingface.co/Helsinki-NLP/
opus-mt-zh-en/tree/main https://huggingface.co/Helsinki-NLP/ opus-mt-en-zh/tree/main
For FactCC7 and DAE 8, We use NLTK 9 to split a summary in English into sentences. For Chinese, we use regular expressions to slice sentences based on Chinese punctuation. Each sentence is classified as factually correct or incorrect. The factual score of a summary is measured as the ratio of sentences classified as correct.
For QuestEval 10, we use the reference-less mode. For BARTScore 11, we use the s → h mode and the checkpoint trained on Parabank2, which is available at the GitHub repository.

heading: D Details of the model implementation in cross-lingual settings
text: The checkpoint and tokenizer used for mBARTScore is available at https:// huggingface.co/facebook/mbart-large-50. We do not fine-tune it.
The En-to-Zh dataset contains 370K English documents, with training set of 364687 items, validation set of 3000 items and test set of 3000 items.
7https://github.com/salesforce/factCC 8https://github.com/tagoyal/
factuality-datasets 9v3.7, https://www.nltk.org/
10https://github.com/ThomasScialom/QuestEval 11https://github.com/neulab/BARTScore
The Zh-to-En dataset is split into training set, validation set, and test set with 1693713, 3000, and 3000 items.
For the data augmentation of mFactcc, we use the same translator12 as used in Section 4.1. For the En-to-Zh dataset, we randomly select 100000 samples from the training set and 2500 samples from the validation set and they are used to construct synthetic data. Finally, we construct a synthetic dataset with 200000 items as the training set and 5000 items as the validation set, where the ratio of the positive and negative items is 1:1. For the Zh-to-En dataset, the data is sampled in the same way resulting in the same size and ratio of the positive and negative items. When mixing the data, we randomly sample 100000 items and 2500 items from the training set and validation set of the above two synthetic datasets. The size of the mixing synthetic training set and validation set is 200000 and 5000. The positive items account for 50.41% in the training set and 50.64% in the validation set.
In all settings, the pre-trained checkpoint13 is used to initialize parameters and we train the model for 10 epochs with a learning rate of 2e-5 and a max sequence length of 512. We try two batch sizes, 6 and 12. The best checkpoint is chosen according to the classification accuracy on the validation set.
We use two GeForce GTX 1080 Ti with 12GB memory for training and inference. Each single training session costs 24-36 hours.

heading: E p-value of the correlation
text: In Table 8, we supplement the p-values corresponding to Pearson correlation and Spearman’s rank correlation coefficients in Section 6.3, showing the significance level of the correlation between two variables.

heading: F Annotation Details
text: In detail, the participants we recruited are from Asia. There are 5 females and 3 males, with an average age of around 24. We first conduct a qualification test before the formal annotation. 10 document-abstract pairs are randomly sampled with 5 in Zh-to-En and 5 in En-to-Zh datasets respectively. We annotated them first. Finally, we
12https://huggingface.co/Helsinki-NLP/ opus-mt-zh-en/tree/main https://huggingface.co/Helsinki-NLP/ opus-mt-en-zh/tree/main
13https://huggingface.co/ bert-base-multilingual-cased
calculated the accuracy of each participant based on our annotation. Higher accuracy means a more consistent understanding of our guidelines. Annotators who achieve at least 80% accuracy are considered qualified to continue the annotation.
We conducted the annotation procedure two times and measure the inter-annotator agreement through two metrics as reported in Section 4.3. For the first annotation, we obtain average moderate agreement of Cohen’s Kappa with κ=0.597 and substantial agreement of Krippendorff’s alpha with α=0.762. However, we notice the low interannotators agreement in En-to-Zh task with fair agreement (κ=0.338) of Cohen’s Kappa and moderate agreement (α=0.624) of Krippendorff’s alpha compared with good inter-agreement in Zh-to-En task. Moreover, we find that some of the annotators may not take the work seriously, and label most sentences as factual although the errors are obvious.
To achieve high-quality annotations, we replaced the annotators who have a low agreement with others. After retraining and re-evaluating, we ask the annotator to annotate 10 items a day. Inspired by Pagnoni et al. (2021), we continuously evaluate annotators during the task as described in Section 4.2 to alleviate human-made disagreement. Finally, we achieve almost perfect agreement on both the two metrics in the second annotation. Here we show a sample from the annotated document-summary pairs on two tasks in Table 9.

heading: G Additional Examples
text: Table 10 and Table 11 show two additional examples.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, in the "Limitations" Section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Yes, in the "Ethics Statement" Section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes. We summarize our main claims in the abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes. We use artifacts in Section 4 and Section 6. We create artifacts in Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes. We cite the authors in the corresponding sections.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Yes, in the "Ethics Statement" Section.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, in the "Ethics Statement" Section.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Yes, in the "Ethics Statement" Section.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Yes, in Section 4 and Appendix F.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in Section 3, Section 4, and Appendix D.
C 3 Did you run computational experiments? Yes, in Section 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes, in Appendix D.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes, in Appendix D.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, in Section 6 and Appendix D.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, in Appendix D.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, in Section 3.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Yes, in Section 3 and Appendix F.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Yes, in Section 3, Appendix F, and the "Ethics Statement" Section.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Yes, in the "Ethics Statement" Section.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No. There is no formal ethics committee in our institution, but our plan was discussed internally. Our data collection adheres to the relevant code of ethics.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Yes, in Appendix F.

heading: Future Work
text: For future work, it is possible to fine-tune mBARTScore or design other data
augmentation approaches