Detecting and Mitigating Hallucinations in Multilingual Summarisation
Authors: Yifu QIU
Yftah Ziser
Anna Korhonen
Edoardo Ponti
Shay B Cohen
Venue: EMNLP 2023 Main (2023)
Keywords: Summarisation
Multilingual NLP
Hallucination
Natural Language Generation
Faithfulness Evaluation
TL;DR: We develop mFACT to assess the faithfulness of non-English summaries. We show that cross-lingual transfer can boost summarisation performance but amplify hallucinations. We propose a method to reduce hallucinations for cross-lingual transfer.

Abstract: Hallucinations pose a significant challenge to the reliability of neural models for abstractive summarisation. While automatically generated summaries may be fluent, they often lack faithfulness to the original document. This issue becomes even more pronounced in low-resource languages, where summarisation requires cross-lingual transfer. With the existing faithful metrics focusing on English, even measuring the extent of this phenomenon in cross-lingual settings is hard. To address this, we first develop a novel metric, mFACT, evaluating the faithfulness of non-English summaries, leveraging translation-based transfer from multiple English faithfulness metrics. Through extensive experiments in multiple languages, we demonstrate that mFACT is best suited to detect hallucinations compared to alternative metrics. With mFACT, we assess a broad range of multilingual large language models, and find that they all tend to hallucinate often in languages different from English. We then propose a simple but effective method to reduce hallucinations in cross-lingual transfer, which weighs the loss of each training example by its faithfulness score. This method drastically increases both performance and faithfulness according to both automatic and human evaluation when compared to strong baselines for cross-lingual transfer such as MAD-X. Our code and dataset are available at https://github.com/yfqiu-nlp/mfact-summ.

Detecting and Mitigating Hallucinations in Multilingual Summarisation
Yifu Qiu
Yftth Ziser Anna Korhonen 'Edoardo M. Ponti 'Shay B. Cohen
Institute for Language. Cognition and Computation, University of Edinburgh Language Technology Lab, University of Cambridge {yifu. qyifu.qiu,ye yftah. .fiserrrreer,,............aa eponti, scohen) @ed. ac. uk
Abstract
edge (e.g., news, documents, meeting notes). Anec- dotally, was shown that the percentage of gen- crated summaries for Mali (Hermann et al., 2015; See et al., 2017) and XSum (Narayan et al., 2018) containing hallucinations amounts to up to 74.8% and 96.9% (Pagnoni et al., 2021), re- spectively. Hallucinations hinder the reliability of abstractive summarisation systems by potentially mislcading users with the misinformation they pro- duce.
Hallucinations pose significant challenge to the reliability of neural models for abstractive summarisation. While automatically gener- ated summaries may be fluent, they often lack faithfulness to the original document. This is- sue becomes even more pronounced in low- resource languages, where summarisation re- quires cross-lingual transfer. With the existing faithful metries focusing on English, even mea- suring the extent of this phenomenon in cross- lingual settings hard. To address this, we first develop a novel metric, mFACT, evaluat- ing the faithfulness of non-Enge non-English summaries, leveraging translation-based transfer from mul- tiple English faithfulness metrics. Through ex- tensive experiments in multiple languages, we demonstrate that mFACT best suited to de- tect hallucinations compared to alternative met- rics. With mFACT, we assess a broad range of multilingual large language models, and find that they all tend to hallucinate often in languages different from English. We then propose simple but effective method to re- duce hallucinations in cross-lingual transfer, which weighs the loss of each training example by its faithfulness score. This method dras- tically increases both performance and faith- fulness according both automatic and hu- man evaluation when compared to strong base- lines for cross- cross-lingen lingual transfer such as MAD-X. Our code and dataset are available at https: //gi wwtchu...om/y.... riiu-nlp/mfaaae. -umms
In addition, current summarisation models, open- source or proprietary, struggle low-resource set- tings (Parida and Motlicek, 2019; Hasan et al., 2021; Bai et al., 2021; ; Urlana et al., 2023), when the target language is under-representted (e.g.. Viet- namese and Urdu). Fortunately, cross-lingual trans- fer methods (Pfeiffer et al., 2020b; Xue et al., 2021; Hu et al., 2020) leverage task-specific knowledge learned from a resource-rich source language summarise documents in many resource aesource-poor tar- get languages, in a zero-shot fashion or only with few annotated examples. Nevertheless, remains unclear to what extent cross-lingual summarisation suffers from the problem of hallucination, com- pared to monolingual systems where English is the only language.
The main challenge in addressing this question is that most faithfulness evaluation metrics are available only for English and do not support low- resource languages. Hence, our first contribution (Section 2) is a model-based metric (mFACT) that measures the factual consistency of multilingual conditional generation, obtained from four diverse English faithfulness metrics (Goyal and Durrett, 2021; Fabbri et al., 2022: Cao et al., 2022a) via 'translate train' knowledge transfer (Artetxe et al., 2020). As illustrated in Figure 1, we use existing faithfulness metrics to label the English document- summary pairs as positive (i.e., faithful) or negative (i.c., hallucinated) and translate them into each tar- get language. We then train a classifier in each
1 Introduction
Recent neural abstractive summarisation models (Lewis et al., 2020; Liu and Liu, 2021; Liu et al., 2022; Fonseca et al., 2022; Ravaut et al., 2022) have shown promise in terms of ROUGE scores (Lin and Och, 2004). However, a well-known prob- lem with these models is hallucination (Maynez et al., 2020; Kryseinski et al., 2020; Laban al., 2022: Cao et al., 2022a) generating summaries that cannot be supported by ground-truth knowl-

---

Translation
DAE
Loss]
Top-K
QAFE
mFACT
Souree examples
ENFS
Ranking
Bottom-K
Documnett
Summary
a) Muttilinuual faithfulness
Loss weighting
Figure I: Pipeline of mFACT for transforring English faithfulness metrics to target languages via machine translation. We average the score four English metries to rank the training samples in XSum. We then translate the most faithful and hallucinated samples into each target language and train classifier to distinguish them.
target language to predict the faithfulness scores for the translated summary pairs. We verify the reliability of mFACT on the translated test set and, most importantly, with human evalua- tion. These confirm the effectiveness of mFACT in capturing hallucinations in target languages. Equipped with this metric, conduct tensive cross-lingen loogull transfer experiments on XL- Sum (Hasan et al., 2021) 1) for abstractive summari- sation in six typologically diverse languages: Chi- nese, Spanish, French, Hindi, Turkish and Viet- namese. We find that state-of -fee------h-rttttt cross-lingual transfer methods increase summarisation perfor- mance in the target languages, but also introduce more hallucinations compared to English monolin- gual models in comparable experimental settings, thus further exacerbating this tendency (Section 6).
human judgements. We observe significant gains both summarisation performance and faithfulness over a series of strong baselines (Section 8). In a nutshell, our main contributions are the following:
We propose mFACT, a multilingual faithful met- ric developed from four English faithfulness met- rics. This enables detecting hallucinated sum- maries in languages other than English.
To the best of our knowledge, we are the first to study hallucination in cross-lingual transfer setting. We show that state-of of-the-an methods like MAD-X (Pfeiffer et al., 2020b) can improve the performance for low-resource summarisa- tion, but also amplify hallucinations.
We apply mFACT to study the faithfulness summarisation of the recent multilingual Large Language Models. We observe that despite their scale, these models are still struggling to reduce hallucinations for languages other than English. We propose novel method to enhance faithful- ness and performance in cross-lingual transfer for summarisation, which consists weighting training samples' loss based on their faithfulness score. Both automatic and human evaluations validate the superiority of our method over exist- ing baselines.
We also employ the mFACT metric to assess the faithfulness some recently released multilingual large language models (LLMs), including Phoenix, BLOOMZ, and Vicuna (Chen et al., 2023; Muen- nighoff et al., 2022; Chiang et al., 2023; Le Scao al., 2022). We show that LLMs that multi- lingual data for pre-training conversational fine- tuning fail to ensure faithfulness in summarisation various languages, producing more hallucina- tions in low-resource ones.
To overcome this limitation and promote faithful summarisation in multiple languages, we adapt series of existing methods for reducing hallucina- tions originally devised for monolingual summari- sation (Section 3.2). In addition, we introduce novel, simple but effective method (Section 3.3): we weigh the loss for each training example ac- cording to their faithful scores. We evaluate our leightiig method with automated metrics and
mFACT: A Multilingual Metric for Faithfulness
The lack of faithful metrics in languages other than English greatly limits the evaluation (and hence, the prevention) of hallucinations in cross-lingual transfer for low-resource languages. In this section, we fill this gap by introducing mFACT. This is con- structed by transferring multiple English faithful

---

metrics into any target language, given the avail- ability of machine translation model.
fulness metrics', and use them score the training samples from the English XSum summarisation dataset (Narayan et al., 2018). Following Maynez et al. (2020), we select the metrics based on two cat- egories of hallucinations generated by the model: 1) intrinsic hallucinations where the summary dis- torts the information present in the document; 2) extrinsic hallucinations where the model adds in- formation that cannot be directly supported by the document. We select two model-based metrics cap- turing intrinsic hallucinations:
2.1 Translation-based Transfer for Faithfulness Metries
One straightforward way to implement a faithful metric in any target language by implementing it from scratch following the design of monolin- gual English metrics. However, these often rely on data annotated with auxiliary language-specific tools. For instance, Dependency Are Entailment (DAE; Goyal and Durrett 2021) requires an exter-
DAE (Goyal and Durrett, 2021) which consists in an entailment classifier trained with annota- tion at a fine-grained dependency level:
nal dependency parser to label fine-grained hal-
QA Eal (Fabbri et al., 2022) which focuses generating questions whose answer is span of the summary, and attempts to answer them based on the document alone;
lucinated segments. This impractical due to the lack of annotated data and auxiliary tools in most languages. Another strategy relies on "trans- late test" knowledge transfer (Artetxe et al., 2020), where test documents and their corresponding gen- erated summaries are translated from the target lan- guage to English. Then, English metrics can mea- sure faithfulness; however, this introduces noise from translation and is costly inference time. which makes this unsuitable for model develop- ment. For instance, model selection is commonly based on early stopping according to validation faithful scores (Choubey et al., 2021; Aharoni et al., 2022), which necessitates translating all generated summaries each validation step.
and two metries for extrinsic hallucinations:
ENFS % a simple rule-based measurement presented by Cao et al. (2022a), which counts the proportion of entities which appear in a summary but not in its corresponding document.
EntFA (Cao et al., 2022a) which estimates the posterior and prior probabilities of generated en- tities with language models conditioned (or not conditioned, respectively) on the source docu- ment. Using these probabilities as features, a KNN classificr detects token-level hallucination. We chose XSum as the source English dataset be- cause 1) all our selected faithfulness metrics are trained on XSum, which allows us to maximise the reliability of these metrics. 2) XSum has been shown include abundant and diverse hallucina- tions (Maynez al., 2020; Pagnoni al., 2021). 1), which allows our metries to capture as many types of hallucinations as possible.
Our solution instead is formulate faithfulness evaluation as binary classification problem, i.c.. predict whether a given summary pair is faithful or hallucinated. In other terms, our proposed approach aims to distil knowledge from multiple teacher models, i.e., existing English model-based metrics, into a target-language classi- fier as a student model. Specifically, we use mul- tiple English faithful metrics to assign the pscudo labels of "faithful" or "hallucinated" for English document- summmay pairs, then translate them to create faithfulness binary classification dataset in target languages. We then train the target-language classifier on the resulting silver dataset. Formally, we aim to obtain a faithfulness scoring model g(-) in target language tgt that predicts the faithfulness for given document-summan pair (x, y). Hence
We then normalise the scores from the above- mentioned four metrics between [0, 1] and aver- age them for each training sample. We rank the samples from the most faithful to the most halluci- nated according to the resulting faithfulness scores. The k top-ranked and k bottom-ranked document- summary pairs are then considered positive and negative examples, respectively. We translate these into series of target languages with the Google Translation API and create our silver faithfulness dataset splitting its examples with proportion of 05/2.5/2.0 2.. as the training/validation//en validattion/ttsinng sets.
g(tat) (x(tt)) ( (((r)) & p(z = 1 x (tal)) y(lgt) where z 1 and z = 0 represent whether the pair is faithful or hallucinated, respectively.
'Our simple sanity check in Appendix A.3 shows these model-basee hallucination metrics to be reliable.
The pipeline for creating mFACT is presented in Figure 1. We start with four diverse English faith-
Phttps: / //w/dddccddd////////////////o www.olooooolleeee.oooooooooooooe con/ transltee

---

Finally, multilingual BERT-based classifier is fine-tuned on our dataset. We follow the sentence- pair classificatiin setting from (Devlin et al., 2019) to concatenate the document- ducument-summary pairs as the input. A classifier receives the last-layer representa- tion for the [CLS] special token and returns a score between 0 (hallucinated) and I (faithful).
adapter.
Task Vector Negation (TVN; Ilharco et al. 2022). Task vector negation mitigates hallucinated generation by subtracting the task vector of the anti- expert model from the fine-tuned model. Formally, given a fine-tuned model with parameter do and an anti-expert model 0 the interpolated model parameters 0* are obtained as
3 Reducing Hallucination in Cross-lingual Transfer
0 0 - A(0 0o),
(1)
We first provide some background on cross-lingual transfer. Then, we show how to adapt several meth- ods promoting faithfulness in monolingual sum- marisation to cross-lingual transfer settings. Fi- nally, we describe new approach based loss weighting.
where À is the importance hyperparameter that con- trols the degree of fusion between the fine-tuned model and the anti-expert.
Contrastive Parameter Ensembling (CAPE; Choubey et al. 2021). To compensate for the poten- tial loss of summarisation ability by only subtract- ing the anti-expert task vector from the base model, CAPE proposes to also add the expert parameters. Formally, the interpolated model parameters 0* are obtained as:
3.1 Cross-lingual Transfer with MAD-X
We adopt the Multiple ADapters framework (MAD-
X; Pfeiffer et al. 2020b), which constitutes a state- of-the-art method for cross-lingual transfer. MAD- X learns independent language and task adapters (i.e., paramoter-efficient model fine-tunings), and then combines them. Specifically, to transfer the ability to summarise documents from source lan- guage target language, follow these steps: 1) We train two separate language adapters the Wikipedia corpora for both the source and get languages.
0- 00=00 + x - ),
(2)
where À again is the importance hyperparameter. DExpert Decoding (Liu et al., 2021). Contrary to Task Vector Negation and CAPE, which directly manipulate the model parameters, DExpert uses expert and anti-expert models to modify the pre- dicted logits at each decoding step. Given the base model fe and pair of expert fo+ and anti-expert fe models, the scores for the next token at each decoding step are:
We stack the (frozen) source language adapter with randomly initialised task adapter and train the latter with annotated data in the source language. 3) We stack the trained task adapter with the target language adapter and then perform zero-shot inference in the target language.
p(y, X, yea) = softmax + A(z, z, )), (3)
where 2i, 2, 2 are the outputs from Se. Set So- at time step t, respectively. Again, an importance hyper-parameter À controls the degree of fusion during decoding.
3.2 Expert and Anti-Expert Approaches
The majority of strategies to reduce hallucinations in monolingual settings rely creating experts or anti-experts that steer the model towards positive behaviour or away from negative behaviour. As a by-product of the pipeline to create our metric, mFACT (Section 2), we obtained two separate sub- sets of faithful and hallucinated samples in both source and target languages. These subsets can serve as training data for experts/anti-expees in multiple languages, thus making them suitable for cross-lingual transfer. We explore three methods in this family. In all in stances, we first train base adapter with the source summarisation dataset. Then, we further tune with the faithful (hallu- cinated) subset to obtain an expert (anti-expert)
3.3 Weighted Loss Approach
We also introduce a simple but effective approach to reduce hallucination during cross-lingual trans- fer. Previous works have shown that controlling the quality of the training samples can improve the model's faithfulness (Kang and Hashimoto, 2020; Aharoni et al., 2022). However, simply filtering out hallucinated training data may sacrifice the sum- marisation performance (Dziri et al., 2022).
We thus propose a "soft" data filtering approach where we weigh the training loss according to each sample's faithfulness score. More formally, we rely

---

Acc.
Prec.
Recall
F1
Models
R-11
DART
QAFE
ENFS.
EntFAt
Model
14
a
μ
μ
MAD-X
23.62
80.14
52.12
23.04
93,09
XNLI
52.9
0.7
71.6
3.3
8.4
1.5
15.0
2.5
XNLI
23.42
84.92
53.42
21.56
94.23
X-mmF
95.0
2.3
95.7
2,4
94.3
2.2
94.9
2.3
XNLI-mFACT
23.64
86.48
52.26
21.50
94.33
mF-T
64.2
6.2
98.9
0.9
28.3
12.9
42.5
16.9
mFACT-TF
23.97
84,25
53.14
21.16
94.51
mF
95.3
1.7
95.4
1.8
95.1
2.0
95.2
mFACT
23.24
85,29
54.30
20.68
94.63
Table I: Mean values (11) and standard deviations (a) of the test performance of four faithfulness classifiers over six target languages. X.-mF, mF-T, mF stand for XNLI-mFACT, TFFAC--rannsfer and mFACT, respec- tively. The detailed results for each language are given in Appendix A.5.
Table 2: Results for (inverse) cross-lingual transfer from other languages to English. We report the average values for performance (R-1) and faithfulness (DAE, QAFE, ENFS%, EntFA) metrics. The methods include the MAD-X baseline and our proposed loss weighting with four weighting metrics, including an XNIIItrained clas- sifier and mFACT. Numbers are averages 3 different random seeds for each of the 6 languages.
on a faithfulness metric for the source language, which outputs a score 20 for the ith document- summary pair's faithfulness. Then the update rule of training parameters for each batch becomes
for natural language inference (NLI), related task for which more annotated data is readily available, can be used also for assessing faithfulness for En- glish summarisation. We thus include a baseline, namely XNLI, which consists in fine-tuning mul- tilingual BERT with the corresponding language split in the XNLI dataset (Conneau et al., 2018) 8). As an alternative, we further fine-tune the XNLI base- line with our translated data (XNLI-mFACT), thus verifying whether combining the supervision signal from both sources boosts the performance. Finally, we incorporate an ablation study for using zero- shot multilingual transfer instead of "translate train" (Artetxe et al., 2020). In multilingual encoder on our English faithfulness classification dataset without translating it, then deploy it directly examples in other languages. Results and Discussion. We report the classifi- cation performance in Table 1. We find that NLI classifiers do not achieve level of performance on par with classifiers trained our faithfulness clas- sification dataset. This demonstrates that evaluat- ing faithfulness is indeed distinct from NLI, which is consistent with previous findings in assessing faithfulness in English (Kryscinski al., 2020; Maynez et al., 2020). Comparing mFACT and mFACT-Transfer, we also observe the positive ef- fects of translation- transfer, which achieves a much higher recall rate than zero-shot cross- -ingull transfer. Hence, mFACT is more likely to identify faithful document-summnn summary pairs such.
VoJ (x000, ,,(00))
- 0- 0- -
(4)
where O is the vector of trainable model parameters, a is the learning rate, m is the batch size, (:; is the loss function for single training example (xl9), y(),, and V.)(-) is the gradient of the loss function wrt. the model parameters.
4 Experimental Setup
Evaluation Metrics. We use ROUGE-1/2/L scores (Lin and Och, 2004) to evaluate the task of ab- stractive summarisation. We use the four metrics mentioned in Section 2 evaluate the faithfulness of English summaries and our mFACT metric for summaries in other languages. Dataset. We conduct our experiments on XL-Sum, which is a large-scale multilingual summarisation dataset (Hasan et al., 2021). XL-Sum provides a large collection of annotated document- summary pairs in 45 languages in addition to English. We test approach on six target languages: Chinese, Spanish, French, Hindi, Turkish and Vietnamese. Table 7 shows the dataset statistics.
we train
5 Faithfulness Classification Experiments
5.1 Classification Results
Firstly, verify the reliability of mFACT by us- ing our translated test sets in multiple languages benchmark mFACT and several baselines for faithfulness classification.
5.2 External Evaluation by Inverse Transfer
Finally, we conduct an evaluation based on inverse cross- linsulln transfer (i.c., from other languages to English) as downstream task with our newly intro-
Baselines. Previous works (Maynez et al., 2020; Kryscinski et al., 2020) showed that models train

---

duced approach (Section 3.3). This setting allows us compare the impact of using different multi- lingual faithfulness metrics, among those listed in Section 5.1, to weigh the training samples in tar-
MAD-X
Full Model
Metries
MFT
CLTF
MFT
CLTF
Faithful. Perform.
R-1
30.25
30.96
23.96
32.05
R-2
8.57
9.11
5.9
9.69
get languages. The logic behind this experiment is that if the scorer captures the model's faithfulness in target languages, the English summaries gener- ated by the corresponding model should be more faithful according the four English metrics from Section 2.1.
R-L
22.48
23.14
17.96
23.85
DAE (1)
68.17
66.69
84.33
53.24
QAFE (1)
34.44
33.87
63.52
30.98
ENFS% (L)
33.29
35.07
16.82
41.45
EntFA (1)
87.58
87.87
95.14
82.96
The results are shown in Table 2. Unsurprisingly, we observe that in general weighting the training samples in target languages with faithfulness met- rics can achieve considerable improvements over the MAD-X X baseline English faithfulness scores. This suggests that these metries are well aligned with the actual faithfulness of generated summaries. Specifically, comparing mFACT-Our and mFAACT Transfer methods with XNLI and XNCT+ we find that our constructed dataset is much more effective in improving faithfulness than NLI signal, which again verifies our previous assumption that faithfulness classification and NLI are only vaguely related. Finally mFACT-Transfer performs worse than mFACT in ROUGE, which can be caused by the much lower recall rate of mFACT-Transfer in Faithfulness classification (see Table 1).
'abble 3: Performance and faithfulness scores for few- shot pross-lnggual transfer (CLTF) and monolingual fine- tuning (MFT) on abstractive summarisation. CLTF gen- erally improves the model's performance but decreases its faithfulness. f and 4. indicate higher or lower values better, respectively.
ferent transfer methods on faithfulness. Results and Discussion In Figure 3, we observe that cross-lingual transfer improves ROUGE scores for both full-model fine-tumin tie-ttunig and MAD-X, out- performing monolingual fine-tuning. This under- scores its effectiveness in transferring task-specific knowledge from source to target languages in low- resource scenarios. However, it's important to note that leveraging source language data can also in- crease hallucination in both cases.
6 Cross-lingual Transfer Introduces Additional Hallucinations
7 Hallucinations in Multilingual Large Language Models
The second analysis of this paper aims to corrobo- rate our observation that cross-lingual transfer can introduce additional hallucinations over monolin- gual fine-tuning, though it improves the task per- formance for summarisation in the target language. Transfer Setup. We compare two data scenarios and two styles of fine-tuning, To begin, we inves- tigate the impact of initial training source data, followed by applying few-shot learning techniques on target data (cross-lingual transfer) instead of direct application. We attribute the difference in faithfulness scores to the additional hallucinations introduced by the training phase in the source lan- guage. Taking Chinese as an example of few-shot cross-lingual transfer, we train the summarisation model first XL-Sum (Chinese) and then with IK randomly sampled XSum (English) examples. Secondly, we compare fine-tuning the full model, where all parameters are updated, with parameter- efficient fine-tuning, where only the adapters are updated. This allows us to study the effect of dif-
We also assess the summarisation performance of recent multilingual large language models (LLMs) on XL-Sum in Table 5. We carefully select three representative multilingual LLMs for investigation, BLOOMZ-P3-7.ID (Muennighoff et al., 2022) represents the instruction-tunel model with En- glish P3 dataset, which derives from the multilin- gual BLOOM (Le Scao et al., 2022). We decide not test BLOOMZ-XP3 trained with machine- translated instructions from English P3 because we consider this experiment as an assessment to the cross-lingual transfer capabilities from the multilingual model.
Vicuna-7B (Chiang et al., 2023) harnesses 70K multilingual conversation -syyee interactions to fine-tune LLaMA. Vicuna originates from the monolingual LLaMA, and the inclusion of Vi- cuna aims to test the cross-lingual transfer ability arising from multilingual conversational tuning. Phoenix-7B (Chen et al., 2023) is the cur- rent state-of oate--o-thee the-art,,, which continues to train

---

Table 5: Assessing the multilingual summarisation per- formance for Vicuna-7B, Phoenix-7B, and BL.OM.7- 7.1B on four languages (Lang.) using ROUGE- 1/L I.. Method
R-1
R-2
R-I.
mFF
mF-T
bi%
tr%
Lang.
P. (%)
R-1
R-L
mF.
MAD-X
29.59
14.86
20.61
39.62
35.08
21.62
34.37
Chinese
CAPE
29.64
14,80
20.58
38.83
34,01
20.11
32.32
Phoenix
English
58.6
29.98
21.11
47.22
French
2.1
29.33
18.80
24.03
TVN
29.68
14.75
20.32
38.53
32.61
17.67
28.76
Dexpert
29.59
14.86
20.61
39.63
35.08
21.62
34.37
Spanish
3.0
15.92
10.65
18.69
Ours
31.24
16.13
22.06
43.16
37.85
30.16
47.02
Hindi
1.1
2.70
2.64
16.53
MAD-X
23.36
5.13
16.34
21.87
29.36
21.98
34.20
English
/
31.74
22.51
57.3
Spanish
CAPE
23.24
5.01
16.24
21.65
29.40
19.98
31.29
BLOOMZ Vicuna
French
/
23.82
15.99
23.02
TVN
23.53
5,06
16.48
23.82
30.54
18.02
28.60
Spanish
/
12.92
7.42
23.68
Dexpert
23.36
5.13
16.34
21.88
29.36
21.98
34.20
Hindi
/
1.30
1.29
13.53
Ours
24.30
6.10
17.41
23.83
33.31
34.54 51.69
MAD-X
25.51
7.78
19:07
28.41 41
19.32
25.57
39.02
English
30.0
17.07
10.89
53.70
Hindi
CAPE
25.80
7.85
19.20
29.11
19.53
23.77
36.58
French
12.9
23.17
13.97
26.34
TVN
25.28
7.73
19.15
32.76
24.61
21.57
33.28
Spanish 10.8
8.54
4.64
27.90
Dexpert
25.51
7.78
19.07
28.40
19.32
25.57
39.02
Hindi
1.3
8.22
7.80
12.52
Ours
24.47
7.46
48
28.44
48
19.52
34.86
50.99
MAD-X
17.22
6.33
14.59
33.24
25.53
38.72
54.12
Turkish
CAPE
17.12
6.23
14.55
35.04
26.69
36.47
51.47
TVN
16.95
6.28
14.49
34.56
25.97
34.05
48.91
Dexpert
17.22
6.33
14.59
33.22
25.53
38.72
54.12
(R- I/L) and mFACT (mF.) metries. We also report the percentage (P. (%) of samples in each language an LLM was exposed during their multilingual training.
Ours
17.16
6.28
14.46
34.91
25.83
45.34
61.50
Vietnamese
MAD-X
27.23
12.57
20.32
36.64
37.75
27.40
42.67
CAPE
27.01
12:45
20.15
36.71
37.79
25.89
40.64
TVN
26.73
12.36
20.07
38.41
39.34
25
40.48
Dexpert
27.23
12.57
20.32
36.61
37.75
27.40
42.67
multilingual training.
Ours
27.76
12.86
20.83
27
38.02
30.23
46.27
Table 5 demonstrates that current LLMs display notable faithfulness limitations in cross- cross-lingual transfer contexts for languages beyond English, in- cluding well-resourced languages like French and Spanish. Furthermore, a noticeable trend emerges: LLM faithfulness across languages tends to corre- late highly to the number of samples from target languages observed during their training. These observations align with recent findings (Lai et al., 2023; Laskar al., 2023) which highlight the chal- lenges in maintaining faithfulness while generating content low-resource languages.
MAD-X
26.02
7.97
19.02
38.71
42.66
18.88
30.29
French
CAPE
25.75
7.93
18.
80
37.54
40.91
18.00
28.88
TVN
25.54
7.86
18.71
38.18
41.74
17.37
27.68
Dexpert
26.02
7.97
19.02
38.74
42.66
18.88
30.29
Ours
27.70
9.09
20.27
36.83
39.75
33.81
50.57
MAD-X
24.82
9.11
18.33
33.08
31.62
25.69
39.11
Average
CAPE
24.76
9.05
18.25
33.15
31.39
24.04
36.86
TVN
24.62
9.01
18.20
34.38
32.47
22.42
34.62
Dexpert
24.82
9.11
18.33
33.08
31.62
25.69
39.11
Ours
25.44
9.65
18.92
34.25
32.38
34.82 51.34
Table 4: Automatic evaluation for zero-shot cross- lingual transfer performance from English to other lan- guages when selecting the checkpoint with the best vali- dation mFACT. Numbers represent the average three with different random seeds. mF stands for mFACT and mF-T stands for MIAAC--Transfe bi% and tr% stand for the percentages novel bigrams and trigrams.
8 Reducing Hallucinations
In this section, we test different methods for cross- lingual transfer of summarisation to multiple lan- guages and for promoting faithfulness. We com- pare our new method of loss weighting based on mFACT with MAD-X, as well as with a series of approaches for reducing hallucinations (Section 4). We evaluate these methods with automated metrics for performance, faithfulness. and abstractiveness (i.e., the ability to rephrase the document instead of copy-pasting spans of text). We also conduct human evaluations corroborate these results.
BLOOMZ with an additional 267K and 189K instances of multilingual instructions and con- versation rounds.
We select three languages, aside from English, which are present in the pre-training data for BLOOMZ and the conversational tuning data for Vicuna and Phoenix. We also report the percent- age of examples in each of these languages that these models have been exposed to during their
Automatic Evaluation. We report ROUGE scores for performance, faithfulness (mFACT), and ab- stractiveness (novel bigrams and trigrams in the

---

Chinese
stractiveness of different methods, which is known to be inversely correlated with faithfulness (Ladhak et al., 2022; Daheim et al., 2023). In fact, reducing hallucinations has the side effect of encouraging the model to copy-paste spans of the document (i.e., acquiring an extractive behaviour). Following Cao et al. (2022a) and See et al. (2017), we use the per- centage of novel n-grams in summaries compared with the document as measure of abstractiveness. Figure illustrates the distributions abstrac- tiveness and faithfulness for all models in six XL- Sum datasets. Both positive and negative predic- tions of mFACT scatter with different levels of abstractiveness. We also observe that summaries generated by the weighted loss method generally have higher level of abstractiveness when they are similarly faithful compared with other baselines. Table shows most expert/anti-expeer models sac- rifice abstractiveness to improve faithfulness score. In contrast, the weighted loss approach produces more novel n-grams. These findings show that our method does not improve faithfulness by simply favouring extractive summaries.
Spansih
French
Tindi
Turkish
Vietumese
Chinese
Spansih
French
Falthral
Hinii
Turkish
Vietuamese
0.2
MAD-X
StalionNoon dithoonttatte
Figure 2: Human preferences for summaries in terms Inform[ation [ and Faithful[ness). Annotators could choose MAD-X, weighted loss (WL, ours), non-differentiable (if summaries are too similar).
Pearson
Spearman
P
value
p
value
XNLI
0.22
0.10
0.25
0.07
XNLI-mF
0.25
0.07
0.28*
0.04
mF-T
0.44*
0.00
0.36'
0.01
0.45*
0.00
0.34*
0.01
Table 6: Correlation between several faithfulness met- rics and human preferences. mF and mF-T stand for mFACT and ALFACT-Transfer respectively. We calcu- late both Pearson and Spearman statistics on document- summary pairs from all six languages to ensure that the sample size is significant.
Human Evaluation. Finally, we recruited human annotators from the Prolific platform² for a blind comparison between MAD-X and our weighted-loss model. We randomly sampled nine documents for each language and paired them with the summaries generated by the two models. We asked the human participants the summaries via A/B testing in two aspects, Informativeness: An informative summary should cover as much information from the document as possible, while it should convey the main idea of the document.
evaluate
summary) for the test set each target language in Table 4. We first observe that the expert/anti-expert methods adapted from monolingual summarisation are partly effective for improving ROUGEs and mFACT score in cross-lingual transfer over MAD- X; however, no clear winner emerges among them, their gains are marginal or inconsistent. For ex- ample, TVN produces the most faithful summaries for Hindi and Vietnamese, CAPE for Turkish, and DExpert for French. All three models, however, dis- play similar trend of sacrificing ROUGE scores to improve faithfulness. Instead, as Table 4 demon- strates, our proposed weighted-loss approach (WL) improves the performance across the board while achieving comparable mFACT score with the most faithful expert models. In particular, WL achieves the best faithfulness in Chinese and Span- ish and the best ROUGE scores for all languages ex- cept Hindi. These results suggest that our weighted- loss method strikes the best balance between sum- marisation abilities and faithfulness.
Faithfulness: faithful summary should only con- tain information already present in the document" and should not contain information contradicting the document.
Participants will first read the document, then select the better summary (or both, if they are simi- lar) terms of informativeness and faithfulness (see Appendix A.5). We require participants to be native speakers of the language they evaluate and have obtained at least a bachelor's degree. Each document and its paired summaries are evaluated by 3 participants. These settings allow us achieve fair inter-rater agreement of 0.28 in terms of Fleiss' k (Landis and Koch, 1977).
The results in Figure indicate that human
A Abstractiveness. We also measure the levels of ab-

---

MAD-X
CAPE
TVN
DExpert
Weighted Loss
1.0
Novel 2-gramm%
0.8
0.6
0.4
0.0
0
1
0
1
0
0
0
1
mFACT
Figure 3: Distributions for Novel 2-gram % and mFACT scores for all five hallucination reduction methods in cross-lingual transfer for the datasets of languages in XL-Sum.
evaluators prefer the summaries generated by our weighted loss method rather than MAD-X, demon- strating that our weighted loss approach improves faithfulness and informativeness for all six lan- guages.
2022). Other methods include designing the spe- cific neural architecture (Huang et al., 2020; Qiu and Cohen, 2022: Cao et al., 2018), summary rank- ing (Falke al., 2019; Liu et al., 2022) and post- hoc correction (Zhu al., 2021: Dong et al., 2020; Cao et al., 2020; Zhao et al., 2020). However, there still a limited understanding of the effectiveness of these methods in cross-lingual transfer.
Finally, we study the correlation between the human preferences from Figure and various faith- fulness metrics presented in Section 5.1. From Ta- ble 6, it emerges that mFACT achieves the strongest correlation with human judgements (0.45 Pearson P and 0.34 Spearman p). which is statistically sig- nificant. In comparison with XNLI and XNL.-mF, we reconfirm that metrics designed for faithfulness classification, rather than natural language infer- ence, more effectively align with human prefer- ences.
10 Conclusion
We investigate how measure and mitigate hallu- cinations of summarisation models in cross-lingual transfer scenarios. We first propose a multilingual metric, mFACT, to facilitate the evaluation of faith- fulness in low-resource languages. By virtue of this new metric, we find empirical evidence that while common cross- wooss-lingual transfer methods ben- efit summarisation performance, they amplify hal- lucinations compared to monolingual counterparts. We also point out that faithfulness in summarisa- tion for languages other than English is still chal- lenging for multilingual large language models. Fi- nally, with the aim of reducing these hallucinations, we adapt several monolingual methods to cross- lingual transfer and propose new method based on weighting the loss according to the mFACT score of each training example. Based both automated metrics and human evaluation, we demonstrate that mFFAC is the most reliable metric in detecting hallucinations in multiple languages. Moreover, compared to a series of state-of-the-art baselines, we find that summaries produced by loss weighting achieve higher performance and abstractiveness, competitive faithfulness, and higher alignment with human preferences. We hope that this work will attract more attention from the community the phenomenon of hallucination languages dif- ferent from English and facilitate future research by establishing evaluation metrics and baselines.
9 Related Work
While faithfulness in summarisation is highly re- searched topic, previous works focused mostly on the English language (Pagnoni et al., 2021; Maynez et al., 2020; Fabbri et al., 2021). 1). To evaluate faith- fulness, state-of-the-art methods fall into three cat- egories. Firstly, validating faithfulness can be cast classification problem (Kryscinski et al., 2020; Goyal and Durrett, 2021; Laban al., 2022). Sec- ondly, faithfulness can be interpreted as answerabil- ity, and assessed with existing question answering models (Fabbri et al., 2022: Scialom et al., 2021). Finally, language models may be adopted to iden- tify extrinsic hallucinations (Filippova, 2020; Cao et al., 2022a). To improve faithfulness in summari- sation models, an approach related to ours changes the training dynamics, c.g. by filtering out halluci- nated data (Cao et al., 2022b; Kang and Hashimoto, 2020; Goyal et al., 2022). The expert/anti-expol approach aims to learn experts and anti-experts that alter the behaviour of base generative model (Liu et al., 2021; Choubey et al., 2021; Ilharco et al.,

---

Limitations
Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 6910-6924, Online. Association for Computational Linguistics.
We use machine translation to construct the faith- fulness classification dataset for training the faith- fulness metrics in target languages. The required resources may constrain the feasibility of extend- ing mFACT to other languages. The quality of the learned metries may also be limited by the propa- gation of errors during translation, especially for languages with poor translation performance. Ad- ditionally, although the weighted-loss approach is effective in a diverse sample of languages, we note that its gains in faithfulness are not consistent for all languages, as we discussed in Section 8. Find- ing a method that is equally effective in reducing hallucinations across all languages is still open research question for future work.
Meng Cao, Yue Dong, and Jackie Cheung. 2022a. Hal- lucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 3340-3354, Dublin, Ireland. Associa- tion for Computational Linguistics.
Meng Cao, Yue Dong, Jingyi Hc, and Jackie Chi Kit Cheung. 20226. Learning with rejection for abstrac- tive text summarization. In Proceedings of the 2022 Conference Empirical Methods in Natural Lan- guage Processing, pages 9768-9780, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguisties.
Ethical Consideration
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstrac- tive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251 -6258, Online. Association for Computational Linguistics.
All human workers participating in our evaluation are informed of the intended use of the provided assessments of summary quality and comply with the terms and conditions of the experiment, as spec- ified by Prolific. In regards to payment, workers from different regions are paid on the same high scale with a wage of E13.5 hourly. This work (and specifically, the human evaluation) has also passed an ethical review by the ethical panel in our insti- tute.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact- neural abstrac- tive summarization. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial In- telligence Conference and Eighth AAAI Symposium Educational Advances in Artificial Intelligence, 18/EAAI 18. AAAI Press.
Acknowledgements
Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang. Juhao Liang, Chen Zhang, Zhiyi Zhang, et al. 2023. Phoenix: Democratizing chatgpt across languages. arXiv preprint arXiv:2304.1044
We would like to thank Zheng Zhao and the anony- mous reviewers for their helpful feedback. We are grateful for Apple AI/ML scholarship awarded Yifu Qiu. We appreciate the use of comput- ing resources through the Baskerville cluster the University of Birmingham.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing g with 90%* chatgpt quality.
References
Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella La- pata. 2022. mface: Multilingual summarization with factual consistency evaluation.
Prafulla Kumar Choubey, Alexander R Fabbri, Jesse Vig, Chien-Sheng Wu, Wenhao Liu, and Nazneen Fatema Rajani. 2021. Cape: Contrastive parameter ensembling for reducing hallucination in abstractive summarization. arXiv e-prints, pages arXiv-2110.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross- cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7684, Online. Association for Computa- tional Linguistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- Yu Bai, Yang Gao, and Heyan Huang. 2021. Cross- lingual abstractive summarization with limited par- allel resources. In Proceedings of the 59th Annual
Proceedings of
guistics.

---

Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo M Ponti. 2023. Elastic weight removal for faithful and abstractive dialogue generation. XrXiv preprint XxXv:2303..7574
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine- teeeggaaied factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1449-1446 1462, Online. Association for Computa- tional Linguistics.
Jacob Devlin, Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tanya Goyal, Jiacheng Xu, Junyi Jessy Li, and Greg Durrett. 2022. Training dynamics for text summa- rization models. In Findings of the Association for Computational Linguisties: ACL 2022, pages 2061- 2073, Dublin, Ireland. Association for Computational Linguisties.
Yue Dong, Shuohang Wang. Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi- fact correction in abstractive text summarization, In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computa- tional Linguisties.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang. M. Sohel Rahman, and Rifat Shahriyar. 2021. XL- sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.
pages
pages 4693-4703, Online, Association for Computa- tional Linguistics.
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Za- iane, Mo Yu, Edoardo M Ponti, and Siva Reddy. 2022. FaithDial: A faithful benchmark for information- seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473 47344994
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFaciEvval: Improved QA- based factual consistency evaluation for summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2587-2601, Seattle, United States. Asso- ciation for Computational I Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross- cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 4411 14211. PMLR.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summariza- tion with semantic-driven cloze reward. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5094-5107, On- line. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Kryšcinski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summariza- tion Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajic Utama, Ido Dagan, and Iryna Gurevych. 2019, Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 22 2214-2220, Florence, Italy. Association for Computational Linguistics.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi. 2022. Edit- ing models with task arithmetic. arXiv preprint arXiv:2212.04089.
Artur Ilichev, Nikita Sorokin, Irina Piontkovskaya, and Valentin Malykh. 2021. Multiple teacher distillation for robust and greener models. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 601-610 Held Online. INCOMA Ltd.
Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 864-870, Online. As- sociation for Computational Linguistics.
Daniel Kang and Tatsunori B. Hashimoto. 2020. Im- proved natural language generation via loss trunca- tion, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718 718-731, Online, Association for Computational Lin- guistics.
Marcio Fonseca, Yftah Ziser, and Shay B Cohen. 2022. Factorizing content and budget decisions in abstrac- tive summarization of long documents. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6341 -6364.

---

Wojeiech Kryscinski. Bryan McCann, Caiming Xiong. and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computa- tional Linguisties.
Alisa Liu, Maarten Sap. Ximing Lu. Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-time con- trolled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6691-67000 Online. Association for Computational Linguistics.
pages
Philippe Laban. Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- national Linguistics, 10:163-177.
Yixin Liu and Pengfei Liu. 2021. SimCLS: sim- ple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 1065-1072, Online. Association for Computational Linguistics.
Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022. Faithful or extractive? on mitigating the dittfulnes--abaraacivennss trade- off in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Compustaiooal Linguisties (Volume I: Long Pupers), pages 1410-1421, Dublin, Ireland. Association for Computational Linguistics.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2890-2903 Dublin, Ireland. Association for Computational Lin- guisties.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man. Franck Demonccurr, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond en- glish: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv prepriat 2rXiv:2304.05613.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159-174.
Proceedings
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty. and Jimmy Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 431 439, Toronto, Canada. Association for Computational Lin- guistics.
Niklas Muennighoff. Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning, arXiv preprint arXiv:221 221007886...
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné. Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model.
Shashi Narayan, Shay Cohen, and Mirella Lapata. 2018. Don't ' give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequeeee to-sequencee pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 787 1871-
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metries. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4812 4829, Online. As- sociation for Computational Linguistics.
Chin Yew Lin and Franz Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and skip-bigram statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- 04), pages 605- Barcelona, Spain.
Shantipriya Parida and Petr Motlicek. 2019. Abstract text summarization: A low resource challenge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the

---

9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JCNE) pages 5994- 5998. Hong Kong, China. Association for Computa- tional Linguisties.
Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 48,4,, Online, Association for Computational Linguistics.
Jonas Pfeiffer, Andreas Rückléé Clifton Poth, Aishwarya Kamath, Ivan Vulić. Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. HdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021. One teacher is enough? pre-trained language model distillation from multiple teachers. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4408-4413 Online. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulić. Iryna Gurevych, and Se- bastian Ruder. 2020b. MAD-X: An Adapter-Based Pross-Lingure Transfer. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 7654-7673, Online. Association for Computa- tional Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguisties.
Framework for Multi-Tass
Yifu Qiu and Shay B. Cohen. 2022. Abstractive sum- marization guided by latent hierarchical document
structure. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5303-5317, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zheng Zhao, Shay Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive sum- marization. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2237- 2249, Online. Association for Computational Lin- guistics.
Mathicu Ravaut, Shafiq Joty, and Nancy Chen. 2022. SummaReranker: A multi-task ceaturr-of-experrc re-ranking framework for abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 718-733, Online. Association for Computational Linguistics.
1: Long Pupers), pages 4504-4524, Dublin, Ireland. Association for Computational L Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational I Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Proceedings of the 55th An-
Ashok Urlana, Pinzhen Chen, Zheng Zhao, Shay B Cohen, Manish Shrivastava, and Barry Haddow. 2023. Pmindiasum: Multilingual and cross- lingul headline summarization for languages in india. In Findings of EMNLP.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Pier-

---

A Appendix
pairs into the evaluation models and test their per- formance. We observe that all hallucination metries drop considerably, showing that these metries are indeed sensitive to random summaries and reliable to some extent.
A.1 Dataset Statistics
We show the dataset statistics for all six used sub-
sets of XL-Sum in table 7.
A.2 Implementation Details.
A.4 Translation Quality Check
mFACT Classifiers We implement mFACT with the transformers package (Wolf et al., 2020). We train the multilingual BERT model for two epochs, with batch size of 32 and a learning rate of 5e- 5. We set the max input length to 512 and apply truncation to the input article if necessary. The same hyper-parameter settings are applied all the languages we test.
Our first experiment is to confirm the effective- ness of mFACT in capturing hallucinations in target languages. To support our method, we conduct quality check for translation outputs, comparison of different metries on our translated faithfulness classification dataset, and an external evaluation of downstream tasks.
Machine translation (MT)-based transfer can ar- guably suffer from error propagation, where MT tools introduce hallucinations into their outputs. This issue is even more serious in our setting where translating faithful samples is necessary create the mFACC metric as training with false positives might significantly degrade its quality. To ensure the feasibility of our pipeline to develop mFACT, we first check the translation quality manually. We randomly pick 100 samples from the Chinese pos- itive set and label their faithfulness. Through this sanity check, we found 13 hallucinated samples; however, only 4 of them are caused by poor trans- lation, while the other 9 due to an incorrect ranking based on the four English metrics. This shows that MT-based transfer is mostly reliable: only a small amount of noise is introduced by MT.
Weighted Loss Summarisation Models We im- plement our weighted loss model for cross-lingual transfer with adapter-transformmrr package (Pfeiffer et al., 2020a). We use the officially re- leased mBART-50 checkpoint as the base model for equipping language and task adapters.
To train the language adapters, same adapter architecture and training settings in (Pfeiffer et al., 2020b). We use the batch size of 64, and learning rate of le-4. We train each adapter with 48K update steps.
follow the
Task Adapters To train the task adapters for sum- marisation, we set the batch size to 32, the learning rate to lc-4, label smoothing factor to 0.1. We use the polynomial scheduler for adjusting the learn- ing rate during training, with weighted decay at 0.01 and maximum gradient norm at 0.1. The model is trained for ten epochs, and we set the first 500 update steps as the warm-up stage. We select the best checkpoint following either the best validation ROUGE or the best mFACT score, re- spectively. During the decoding step for zero-shot cross- lingull transfer,
A.5 Extended Results for Faithfulness Classification
To gain deeper comprehension of the averaged faithfulness classification results presented in add reference to Table 1, we analyse the individual language-specific outcomes (Table 10). Across the six language experiments, we consistently observe significant performance gap between the models trained on the NLI task and those trained on the faithfulness classification task.
follow most settings of (Hasan et al., 2021). We apply the beam search with a size of 6, and the minimum/maximmm de- coding steps are set to 30/84, respectively. The length penalty applied at 0.6, and we block all repeated tri-grams.
The following is the guide for annotators to indi- cate whether summary is informative and faithful.
A.3 Sanity Check for English Faithfulness Metrics
A.6 Full-model transfer vs. MAD-X transfer
We perform sanity check experiment and report the results in Table 9 to verify the reliability of these mosel-based hallucination metrics. We randomly shuffle the alignments of document-summary pairs predicted by the mBART model and the reference. We then feed these misaligned cocument--uumaryy
We conduct comparative study the perfor- mance of summarisation and faithfulness in two cross-lingual transfer approaches: MAD-X style and full-model transfer.
For both MAD-X style and full-model cross- lingual transfer, we observe that cross-lingual trans-

---

Language
Family
Doc. Len.
Sum. Len.
Comp %
Prop.
Prop. C
Chinese
Sino-Tibetan
9.63
93.49
29.56
French
Indo- -udo-eu
743
43
12.07
99.20
26.72
Spanish
European
1242
11.12
84.71
42.93
Vietnamese
Austro-Asiatic
1647
50
6.09
/
Turkish
Turkic
9.33
/
Hindi
Indo-Europeat
1200
49
8.59
90.91
31.42
Table 7: Dataset statistics for six annotated datasets from XL-Sum (Hasan et al., 2021). We report the document length (Doc. Len.), summary length (Sum. Len.) and the corresponding compression percentage (Comp%). All lengths are measured in the unit of tokens. We also include quality measurements from the XL-Sum paper according to human annotators. Prop. A reports the percentage of the summaries that convey the main idea of the document. Prop. C reports the percentage of summaries that contain some additional information not presented in the source.
Document & Translation
Summaries & Translations
mFACT
MAD-X: B
A
N Fit 10 HXS360
0.12
& II
IS IA
Weighted
0.78
BE FE TAM
BI B
[...]
A 18 T2. AIHEL
poisoned with carbon monoxide, one in serious condition. About 60 workers were replacing rails in the undersea runnel berween Calais, France, [...] Sunday moming, welder fell and loter diagnosed with carbon monoxide poisoning, and other workers were also poisoned varying dogrees. They were sent to local French bospital for treatment, and the remmining 41 workers lave home rest. [...]
working
Tunnel
MAD-X: worker British Channel Tunnel was poisoned, one of them in serious condition. and the remaining 41 workers have been sent to French hospitals for treatment.
0.12
Weighted Loss: The accident occurred in the undersea tunnel near Calais, France. One worker was poisoned and 18 workers were injured, of them in serious condition.
0.78
Table 8: Examples of hallucinations (highlighted a orange colour) generated by MAD-X, a method for zero-shot cross-lingual transfer, on top of mBART-50 backbone. this work, we present 1) the mFACT metric to evaluate the faithfulness of summarisation models in target languages other than English and 2) a loss weighting method to reduce hallucinations cross-lingual transfer. We highlight the faithful pieces of information produced by our loss weighting but missed by MAD-X a green colour.
DAE (f)
QAFE (1)
ENFS% (1)
EntFA (1)
shot and few-shot transfer scenarios. While Full- model transfer demonstrates an advantage in the few-shot transfer scenario compare to MAD-X,
Reference
19.37
34.14
72.50
Shuffled
0.00
0.15
97.57
34.63
mBART
38.40
37.82
53.24
82.67
MAD-X performs better in the zero-shot scenario.
Shuffled
8e-5
0.13
97.42
36.12
Additionally, regardless of the transfer scenario.
MAD-X exhibits a lower occurrence of hallucina- tions.
Table 9: Hallucination scores of the reference and mBART's outputs and their corresponding shuffled document-summan pairs (Shuffled). All results evaluated on the XSum test set. 1 and L indicate higher and lower hallucination score is better, respectively. QAFE stands for QAFactEval.
A.7 Validation Faithfulness Curve for Weighted Loss Method
We provide the training curve of MAD-X and our weighted loss method in Figure 5.
fer leads to improved ROUGE scores, but it also results reduced faithfulness scores. Interest- ingly, when comparing the effects of Full-model and MAD-X transfer in Figure 3, we see that Full- model transfer exhibit a greater improvement in ROUGE scores. However, this improvement come at the expense of introducing more hallucinations.
Upon examining the curve, it becomes evident that the model trained with the weighted loss con- sistently exhibits higher faithfulness compared to the MAD-X baseline throughout the entire train- ing process. This observation serves evidence for the effectiveness of our approach in guiding the model's optimisation towards increased faith- fulness by diverting its training away from halluci- nated samples.
In Figure 4, we further compare the performance of Full-model and MAD-X transfer in both zero-

---

Models
Ace.
Prec.
Recall
Fl
22-
Model
32.
Full
160
Spunish
XNLI
54.00
76.47
10,48
18.44
21.5-
MAD-N
XNLI-mFACT
96.40
97.13
95.56
96.34
15.415551
25.85
mFAcceTransfer
70.60
98.10
41,53
58.36
5,25
mFACT
97.00
97.55
96.37
96.96
6:55
5.0
4:35
9:30
XNLI
53.00
74.07
8.06
14.55
Hindi
XNLI-mFACT
94.20
94.33
93.95
94.14
4,8
625
6:0
Reage-1
Resgo=2
Reaze-1
Rosge-2
RosgooL
mFAATT-TrnsSee
54.00
100
7.26
13.53
mFACT
94.00
95.04
92.74
93.88
541
Full Madel
02.11
Pull Midel
MAD X
MAD X
XNLI
52.40
69.23
7.26
13.14
50
Turkish
XNLI-MFACT
96.60
97.53
95.56
96.54
70
70
$4.55
mFACT- Transfer
59.60
100
18.55
31.29
60
mFACT
97.00
97.17
96.77
96.97
50
Vietnamese
XNLI
53.40
72.73
9.68
17.08
60
XNLI-mFACT
96.00
96.72
95.16
95.93
QAIe:
mFAACT-Transter
68.00
97.83
36.29
52.94
Zemodux Transfer
Fewedhot
mFACT
95.40
94.47
96.37
95.41
Figure 4: Comparison of Full-model and MAD-X X cross- lingual transfer in ROUGE and faithfulness. The left column the zero-shot performance, and the right col- umn is the few-shot performance. We provide the aver- age scores over all six languages.
XNLI
52.60
67.74
8.47
15.05
French
XNLI-mFACT
96.20
96.73
mFACT-Transfer
65.80
98.73
31,45
47.71
mFACT
95.60
95.20
95.97
95.58
XNLI
52.20
69.57
6.45
11.80
Chinese
XNLI-mFACT
90.80
91.39
89.91
90.65
Model
mFACceTransfee
67.40
98.85
34.67
51.34
mFACT
92.60
92.71
92.34
92.53
Val. MFAACT-inglis
0.48
Weighted Loss
Table 10: Classification performance translated faithfulness dataset for all target languages.
0.46
0.44
A.8 Prompts Used for Multilingual LLM'' Summarisation
0.42
2
10
We show the prompt templates used for all lan- guages our LLM's summarisation experiments in Figure 6.
Epoch
Figure 5; Validation mFACT scores curve for each model's training dynamics. Weighted loss consistently outperforms MAD-X in terms of faithfulness during the whole training period.
A.9 Assembling Metrics for mFACT does better than Single Metric
We conducted an additional experiment to support our assembling design of mFACT. Rather than av- eraging four metrics, we individually apply single English metric DAE, QAFE, ENFS, and EntFA to rank the XSum dataset and train multilin- gual classifier similar mFACT-TTrnsfer without translation, denoted as DAE-T, QAFE-T, ENFS-T, and EntFA-T.
emerges with the highest human correlation when compared to other four metrics. This observation underscores mFACT's better correlation with hu- man evaluations. The reason could relying on a single metric can introduce biased preference models and a lack of diversity for captured halluci- nations. In general, multiple teacher models lead to robust, unbiased process (Wu et al., 2021; Ilichev et al., 2021). Using diverse metrics in mFACT's training helps the classifier detect various hallu- cination types our inverse transfer experiments (Table 2) also show mFACT's promising correla- tions with both intrinsic and extrinsic hallucination metrics.
To examine mFACT with other metrics originat- ing from each single metric, we extend the human evaluation results in Table 6. We compare these four metrics with FFACT-Transfer, and again measure the Pearson and Spearman correlations human annotations.
In Table 11, we find mFACT consistently

---

Language
Prompt
Summarize the given article: $(en_articles
English
-
Chinese
$(zh_article)
Résumer l'article donné:
French
$(fr (ffr_article)
Resume el artículo dado: $(es_article)
Spanish
(िय अप रेड संदय wega कर: $(hi_artiee article]
Hindi
Figure 6: Prompt templates used for conducting summarisation experiments with multilingual large lan- guage models.
Pearson p value
Spearman
p
p value
mF-T
0.44
0.00
0.36
0.01
DAE-T
0.34*
0.01
0.33*
0.01
QAFE-T
0.25
0.07
0,29*
0.04
ENFS-T
0.24
0.07
0.29*
0.04
EntFA-T
0.35*
0.01
0.36"
0.01
Table II:
Correlation with human preferences for mFACT and four transferred metrics developing from single metric. We again calculate both Pearson and Spearman statistics on document-sumnan pairs from all languages ensure that the sample size signif- icant.
A.10 Strategy for Selecting Best Model Checkpoint
Table 12 compares the summarisation model per- formance when we select the model checkpoint with the best ROUGE-1 or the best mFACT score. We find that under both strategies, the weighted loss model can achieve better ROUGE and faithful- ness scores in most languages. However, similar to other works (Choubey et al., 2021; Aharoni al., 2022), selecting the model checkpoint with the best validation faithfulness score has a higher positive contribution to model's faithfulness.
A.11 Distributions of Faithfulness and Abstractiveness for All Languages
We show the distributions for the percentage of novel 2-grams and mFACT scores for all six lan- guages in Figure 7.

---

Best Validation ROUGE-1
Best Validation mFACT-
Method
R-2
mF
mF.T
b1%
tr%
R-2
R-I
mF
mF.T
bi%
MAD-X
27.97
14,10
19.95
38. 29
33.99
28.23
44.15
29.59
14.86
20.61
39.62
35.08
21.62
34.37
Chinese
CAPE
28.15
14.57
20.74
42.56
39.59
27.12
42.11
29.64
14,80
20.58
38.83
34,01
20.11 11
32.32
TVN
19.64
9.64
14.27
36.05
31.28
9.98
16.68
29.68
14.75
20.32
38.53
32.61
17.67
28.76
Dexpert
27.98
14.10
19.96
38.17
33.97
28.21
44.12
29.59
14.86
20.61
39.63
35.08
21 62
34.37
Ours
30.81
16.14
22.04
42.50
35.88
36.38
54.85
31.24
16.13
22.06
43.16
37.85
30.16
47.02
MAD-X
20.77
4.89
15.06
20.57
29.25
33.91
50.47
23.36
5.13
16.34
21.87
29.36
21.98
34.20
Spanish
CAPE
21.16
4.91
15,28
22.03
31.02
29.99
45.22
23.24
5.01
16.24
21.65
29.40
19.98
31.29
TVN
20.27
4.94
14.98
28.61
36.05
18.92
30.66
23.53
5.06
16.48
23.82
30.54
18.02
28.60
Dexpert
20.75
4.88
15.04
20.46
29.25
33.85
50.42
23.36
5.13
16.34
21.88
29.36
21.98
34.20
Ours
22.62
5.54
16.31
22.09
32.50
41.90
60.49
24.30
6.10
17.41
23.83
33.31
34.54
51.69
MAD-X
20.08
5,44
15.10
22.75
16.42
34.96
51.35
25.51
7.78
19.07
28.41
19.32
25.57
39.02
Hindi
CAPR
20.97
6.11
16.07
23.50
16.45
30.95
45.81
25.80
7.85
19.20
29.11
19.53
23.77
36.58
TVN
19.44
5.78
15.48
34.36
27.22
24.12
35.71
25.28
7.73
19.15
32.76
24.61
21.57
33.28
Dexpert
19.96
5.74
15.32
22.79
16.43
34.97
51.37
25.51
7.78
28.40
19.32
25.57
39.02
Ours
20.99
5.73
15.70
20.32
13.75
47.44
65.9
24.47
7.46
18.48
28.48
19.52
34.86
50.99
MAD-X
17.16
5.56
14.00
30.16
21.08
43.59
60.56
17:22
6,33
14.59
33.24
25.53
38.72
54.12
Turkish
CAPE
17.66
5.72
14.33
29.88
22.50
39.32
56.45
17.12
6.23
14.55
35.04
26.69
36.47
51.47
TVN
17.49
5.82
14.63
34.05
22.91
33.68
48.82
16.95
6.28
14.49
34.56
25.97
34.05
48.91
Dexpert
17.17
5.56
14.01
43.64
60.61
17.22
6.33
33.22
25.53
38.72
54.12
Ours
17.66
5,95
14.44
29.80
22.21
54.91
72.2
17.16
6.28
14.46
34.91
25.83
45.34
61.50
Vietnamese
MAD-X
25.85
11.85
19.45
35.09
34.10
33.99
50.73
27.23
12.57
20.32
36.64
37.75
27.40
42.67
CAPE
26.20
11.96
19.65
36.98
36.79
30.68
46.77
27.01
12.45
20.15
36.71
37.79
25.89
40.64
TVN
24.40
11.32
18.83
36.94
34.85
25.69
39.38
26.73
12.36
20.07
38.41
39.34
25.86
40.48
Dexpert
25.86
11.85
19.46
35.15
34.19
33.99
50.74
27.23
12.57
20.32
36.61
37.75
27.40
42.67
Ours
27.01
12.35
20.17
37.56
38.32
35.32
52.02
27.76
12.86
20.83
38.27
38.02
30.23
46.27
MAD-X
25.82
8:31
19.10
38.21
40.79
27.38
42.21
26.02
7.97
19.02
38.71
42.66
18.88
30.29
French
CAPE
25.83
8,40
19.08
26.11
40.44
25.75
7.93
18.
37.54
40.91
18.00
28.88
TVN
24.42
7.90
18.34
42.27
44.60
16.97
28.35
25.54
7.86
18.71
38.18
41.74
17.37
27.68
Dexpert
25.82
8.31
19.10
38.21
40.82
27.37
42.19
26.02
7.97
19.02
38.74
42.66
18.88
30.29
Ours
27.63
9.21
20.28
35.57
39.47
41.52
59.22
27.70
9.09
20.27
36.83
39.75
33.81
50.57
Table 12: Automatic evaluation for zero-shot performance in English-to-others cross-lingual transfer direction while selecting the checkpoint with the best validation ROUGE-I the best validation mFACT score. run model results with three different random seeds. mF stands for mFACT and mF-T stands for mFACT-Transfer tr% and bi% are the percentage of novel tri-gram and bi-gram, respectively.

---

zh. CN-MAD-X
CN-CAPE
CN-TVN
CN-DExpert
CN-Weighted Loss
1.00
Novel 2-gram%
0.75
0.50
0.25
0.00
XX-MAD-X
XX-CAPE
XX-TVN
XX-DExpert
-s_XX- Weighted Loss
Novel 2-gram%
0.75
0.50
0.25
0.00
ft XX-MAD-X
ft XX-CAPE
fr XX-TVN
fr XX-DExpert
fr XX-Weighted Loss
1.00
Novel Novel 2-gram7%
0.75
0.50
0.00
hi IN-MAD-X
hi IN-CAPE
hi IN-TVN
hi IN-DExpert
hi IN- IN-Weighted Loss
1.00
Novel 2 1-grmm/
0.50
0.25
vi. VN-MAD-X
vi VN-CAPE
vi VN-TVN
vi_ VN-DExpert
vi. VN-Weighted Loss
1.00
Novel 2-el2--g 3-gram%
0.75
0.50
0.25
IN-MAD-X
hi IN-CAPE
hi IN-TVN
IN-DExpert
hi IN-Weighted Loss
1.00
Novel Novel 2-gram%6
0.75
0.25
0.00
mFACT
Figure 7: Distributions for Novel 2-gram& and mFACT scores for all hallucination reduction methods in cross- lingual transfer for six languages.


Reviews:
[{"review_id": "3dj1JghIX6", "reviewer": "Reviewer_1ZfH", "summary": "", "questions": "(1) One thing wish to know is that if \"averaging\" is the best way to assemble the 4 model-based metrics. What if each individual metric is used to train mFact alone, will mFact beat all the metrics when they are trained alone?\n\n(2) in Table 5, is there result on the human evaluated correlation to compare mFact with the other 4 metrics?\n\n(3) any chance to give more discussion on the result in Table 4? Appreciate if more discussion can be provided on Hindi and Turkish language. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that mFACT can contribute to the NLP research community and their comments. We hope the additional experiments we conducted will ease the main soundness concerns. \n\n**Q**: For mFact, wish to see more comparison. For example, in addition to XNLI, any chance to include other baselines for comparison?\n\n**A**: There are currently only a handful of established baselines dedicated specifically to multilingual faithfulness evaluation, as evidenced by the scarcity of reported alternatives in the existing literature. We are aware of a recent contribution at ACL 2023 [1], where XNLI is repurposed as an auxiliary task for faithfulness classification due to its linguistic similarity to our task. This underlines the inherent connection between Natural Language Inference (NLI) and faithfulness assessment [2]. \n\nHowever, we firmly believe in delineating the nuances that set NLI apart from faithfulness classification, particularly in the context of hallucination detection. NLI, while pertinent, might not holistically capture the intricate aspects of faithfulness evaluation as we show in this paper. A specialised metric tailored to the challenges of hallucination detection is crucial for robust future research in this domain.\n\nReferences:\n\n[1] Multilingual Summarization with Factual Consistency Evaluation. Aharoni et al., ACL 2023 Findings.\n\n[2] Evaluating the Factual Consistency of Abstractive Text Summarization. EMNLP 2020.\n\n**Q**: One thing wish to know is that if \"averaging\" is the best way to assemble the 4 model-based metrics. What if each individual metric is used to train mFact alone, will mFact beat all the metrics when they are trained alone?\n\n**A**: We believe that relying on a single metric can introduce biased preference in models and a lack of diversity for captured hallucinations. In general, multiple teacher models lead to a robust, unbiased process [1, 2]. Using diverse metrics in mFACT's training helps the classifier detect various hallucination types - our inverse transfer experiments (Table 2) show mFACT's promising correlations with *both* intrinsic and extrinsic hallucination metrics.\n\nWe conducted an additional experiment to support our argument. Rather than averaging four metrics, we individually apply single English metrics - DAE, QAFE, ENFS, and EntFA — to rank the XSum dataset and train a multilingual classifier similar to mFACT, denoted as DAE-M, QAFE-M, ENFS-M, and EntFA-M. While the faithfulness classification is not important by itself, we provide the results here to first show that a reliable classifier can be trained for all of the metrics as a sanity check. \n\n| Model     | Acc.  | Prec.  | Recall | F1    |\n|-------------|-------|--------|--------|-------|\n| mFACT-Transfer | 96.0 | 96.34  | 95.56  | 95.95 |\n| DAE-M       | 95.2  | 95.16  | 95.16  | 95.16 |\n| QAFE-M      | 87.2  | 83.09  | 93.15  | 87.83 |\n| ENFS-M      | 96.8  | 98.33  | 95.16  | 96.72 |\n| EntFA-M     | 96.8  | 94.27  | 99.6   | 96.86 |\n\nTo further verify our intuition that mFACT's aggregation of multiple metrics is primed to capture a wider array of hallucinations, we execute an additional inverse transfer experiment (XL-Sum Chinese -> XSum English) using classifiers trained with signals from single metrics. We chose to focus on the EntFA metric, as its classifier is the most reliable of all the examined metrics (96.86 F1). \nThe results are presented below:\n\n|      Model    | DAE↑   | QAFE↑  | ENFS↓  | EntFA↑ |\n|----------|--------|--------|--------|--------|\n| MAD-X    | 83.38  | 69.69  | 16.38  | 95.6   |\n| mFACT-T  | **88.01** | **78.78** | **12.11** | **96.96** |\n| EntFA-M  | 83.47  | 68.57  | 15.98  | 95.52  |\n\n\nFrom these outcomes, we observe that training the classifier with a single metric leads to inferior results across all the metrics, including the EntFA in which the EntFA-M model is specialised. We also note that even though EntFA-M can capture extrinsic hallucination well, it still has a limited ability to capture intrinsic hallucination (see weak DAE and QAFE improvements), showing the importance of assembling multiple metrics to capture varied types of hallucinations. We finally incorporate these 4 metrics into human evaluation as discussed in the next response. \n\nReferences:\n\n[1] One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers. Wu et al., ACL Findings 2021.\n\n[2] Multiple Teacher Distillation for Robust and Greener Models. Ilichev et al., RANLP 2021.\n\n\n**Q**: in Table 5, is there result on the human evaluated correlation to compare mFact with the other 4 metrics?\n\n**A**: To examine mFACT with four other metrics in the actual model's outputs rather than a silver-reference test set, we extend the existing results in Table 5. Using the specialised models we built (see details in the previous answer), we measure the Pearson and Spearman correlations to human annotation:\n\n| Metric    | Pearson (rau) | P-val | Spearman (rau) | P-value |\n|-----------|---------------|-------|----------------|---------|\n| mFACT-T   | **0.44**      | 0     | **0.36**       | 0.01    |\n| DAE-M     | 0.34          | 0.01  | 0.33           | 0.01    |\n| QAFE-M    | 0.25          | 0.07  | 0.29           | 0.04    |\n| ENFS-M    | 0.24          | 0.07  | 0.29           | 0.04    |\n| EntFA-M   | 0.35          | 0.01  | **0.36**       | 0.01    |\n\n\nmFACT consistently emerges with the highest human correlation when compared to these four metrics. This observation underscores mFACT's robustness and its capacity to better correlate with human evaluations.\n\n**Q**: \"mFact is adopted in the weighted loss, which reports better Rouge in Chinese, Spanish, Vietnamese, French, but a little mixed in Hindi and Turkish.\" - any chance to give more discussion on the result in Table 4? Appreciate if more discussion can be provided on Hindi and Turkish language.\n\n**A**: Your question is intriguing. In short, we currently lack a definitive explanation for these performance variations in specific languages. However, we can suggest potential reasons for further investigation. \n- Firstly, discrepancies could arise due to varying translation quality from the Google API across languages, impacting performance negatively for languages with lower translation quality. \n- Secondly, the pre-training of our used multilingual models (e.g., mBERT and mBART) might underrepresent certain languages, leading to poorer results in our downstream task. \n- Thirdly, this mixed performance in ROUGE and faithfulness (we see weighting-loss training improves faithfulness) could also be generally attributed to the disparity between the objectives of enhancing ROUGE scores and faithfulness [1,2,3]. And this is because the heuristic-based dataest construction naturally includes hallucinations in even the testing set [4,5]. We'll include these discussions in Section 7 for context.\n  \nReferences:\n\n[1] CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization\n\n[2] Reducing Quantity Hallucinations in Abstractive Summarization\n\n[3] Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality\n\n[4] Benchmarking Large Language Models for News Summarization\n\n[5] Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) propose mFact, a model-based metric for faithfulness/hallucination measurement\n\n(2) extensive experiments are conducted to support the paper (in the main content and also in appendix)", "reasons_to_reject": "Not necessarily strong negative points:\n\n(1) for mFact, wish to see more comparison. For example, in addition to XNLI, any chance to include other baselines for comparison? \n", "questions_to_authors": "(1) One thing wish to know is that if \"averaging\" is the best way to assemble the 4 model-based metrics. What if each individual metric is used to train mFact alone, will mFact beat all the metrics when they are trained alone?\n\n(2) in Table 5, is there result on the human evaluated correlation to compare mFact with the other 4 metrics?\n\n(3) any chance to give more discussion on the result in Table 4? Appreciate if more discussion can be provided on Hindi and Turkish language. ", "paper_topic_and_main_contributions": "In multilingual summarization task, the hallucination becomes more severe in cross lingual setting, where the supervised data is in low resource settings. This paper studies this problem, with the goal of reducing the hallucination in cross lingual transfer learning. (from English source language to different target languages).\n\nTo address the research problem, authors first propose a model-based metric (mFact) to measure the faithfulness of the document-summary pair. To train the model predicting mFact measure, first, pseudo faithful and pseudo hallucinated document-summary pairs are selected by an averaged empirical metric. Then, the document-summary pairs are translated into target language, and finally a multilingual BERT is fine-tuned based on the data generated. The empirical metric is obtained by averaging 4 model-based metric to rank the document-summary pair from faithful to hallucinated. \n\nWith the support of mFact, the training instance in cross lingual summarization is weighted according to the mFact metric, which helps to improve the summarization quality and reduce the hallucination. Experiments are conducted on XLSum. The experiments first compare mFact with XNLI metric (for NLI task) to verify the effectiveness of mFact in summarization task. Then mFact is adopted in the weighted loss, which reports better Rouge in Chinese, Spanish, Vietnamese, French, but a little mixed in Hindi and Turkish. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CBMD0eWKxN", "reviewer": "Reviewer_HygN", "summary": "", "questions": "Would you train different classifiers to obtain mFACT for each language, or would you directly train a multilingual classifier with mixed datasets?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n**Q**: The authors should compare their results with ChatGPT, as it can achieve better performance than cross-lingual transfer baselines, which will make your contribution less meaningful.\n\n**A**: We firmly assert that employing ChatGPT or any analogous black-box model as a foundational reference in a scientific inquiry comes with substantial drawbacks. For example, recent research has exposed the significant variability in ChatGPT's performance over time, undermining the reproducibility of experiments [1]. Moreover, assessing a black-box model poses the challenge of data contamination, an issue of great magnitude with such data-intensive models [2, 3]. For more reasons to avoid using black-box models as baselines, we kindly refer the reviewer to a more comprehensive blog post written by one of ACL 2023 organizers, Anna Rogers [4].\n\nWith that being acknowledged, even a potent black-box LLM like ChatGPT exhibits inferiority when pitted against more compact, task-specific models. A case in point is [5]'s demonstration that ChatGPT lags behind state-of-the-art summarization models for monolingual English summarization. This disparity becomes more pronounced when assessing less commonly spoken languages [6]. As for the faithfulness of such models, recent studies demonstrated that ChatGPT is far from being hallucination-free when using English or low-resource languages as input [7], where it struggles mostly with extrinsic hallucinations [8]. The inverse transfer results we provide in Table 2 show that mFACT correlates well with both intrinsic metrics (DAE and QAFE) and extrinsic metrics (ENFS and EntFA), which are still a significant issue even for strong black-box models such as ChatGPT. \n\nReferences:\n\n[1] Can we trust the evaluation on ChatGPT? https://arxiv.org/pdf/2307.09009.pdf\n\n[2] Can we trust the evaluation on ChatGPT? https://arxiv.org/pdf/2303.12767.pdf\n\n[3] Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4. https://arxiv.org/pdf/2305.00118.pdf\n\n[4] Closed AI Models Make Bad Baselines. https://hackingsemantics.xyz/2023/closed-baselines/\n\n[5] A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets. https://aclanthology.org/2023.findings-acl.29.pdf\n\n[6] ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning. https://arxiv.org/pdf/2304.05613.pdf\n\n[7] Hallucinations in Large Multilingual Translation Models. https://arxiv.org/pdf/2303.16104.pdf\n\n[8] A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. https://arxiv.org/pdf/2302.04023.pdf\n\n**Q**: Your pipeline for creating mFACT is too complex. It is model-based, and it need to be trained on the XSum dataset. The Google API is also needed. The authors should provide more details about the cost to dispel users' concerns.\n\n**A**: We want to emphasise that we do *not* use the Google API at inference time but rather use it offline once to obtain the translated training example. We will clarify it better in our paper. As for using the XSum dataset for obtaining our metric, we believe the efforts we invest are on par with other popular metrics, e.g., the NLI-based methods we use as baselines [1, 2], which require training specialized models to evaluate faithfulness. Leveraging translation for developing datasets/resources is a common practice in cross-lingual and multilingual summarization [3], and future research with limited resources can be carried out using an open-source translator if additional adaptation is needed or use our mFACT-transfer for easily extending other languages using zero-shot cross-lingual transfer. For each language, our method has 92M characters to translate (~1000 GBP) on average. We will add these details to our paper.  \n\nReferences:\n\n[1] On Faithfulness and Factuality in Abstractive Summarization. https://aclanthology.org/2020.acl-main.173.pdf\n\n[2] Evaluating the Factual Consistency of Abstractive Text Summarization. https://arxiv.org/pdf/1910.12840.pdf\n\n[3] NCLS: Neural Cross-Lingual Summarization. https://aclanthology.org/D19-1302/\n\n**Q**: The weighted loss approach is not easily used to train large models. So, I don't think the mitigating hallucination method will contribute much.\n\n**A**: As we established in our first answer, we believe that compact, task-specific models are also of great importance, as they achieve state-of-the-art across many tasks. In addition, there is a vast body of research aiming to fine-tune LLMs in a more cost-efficient manner. Two widely used methods for that are LoRA (Low-Rank Adaptation) [1] and its quantized version QLoRA [2]. We believe that such advances will allow researchers to use our method for LLMs as well. \n\nFurthermore, the weighted loss approach shows the potential to reduce hallucinations by acting as a \"soft\" sample selector during training. This adaptability can be extended to address other text attributes, such as bias and human preference, using an appropriate rating model. The weighted loss technique, characterised by its simplicity and effectiveness, outperforms existing benchmarks. Its ease of use, reproducibility, and lightweight nature position it as a valuable baseline for advancing research in precise multilingual summarization.\n\nReferences:\n\n[1] LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS. https://arxiv.org/pdf/2106.09685.pdf\n\n[2] QLORA: Efficient Finetuning of Quantized LLMs. https://arxiv.org/pdf/2305.14314.pdf\n\n**Q**: Would you train different classifiers to obtain mFACT for each language, or would you directly train a multilingual classifier with mixed datasets?\n\n**A**: We trained different classifiers, as we believe this strategic choice safeguards against the risk of classifier degradation and aligns with our commitment to ensuring the robustness and adaptability of mFACT to varied languages and contexts. The second option shares semantic content across translated instances requires a careful balance to avoid the classifier overfitting to recurring hallucinated errors in multiple languages, which can compromise its ability to generalise effectively across diverse linguistic contexts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a new multilingual faithful metric mFACT, which can be used in low-resource languages. They have conducted experiments to show the effectiveness of mFACT.\n2. They employ the mFACT metric to design a method for mitigating hallucination.\n3. They analyze the experiment results in detail.\n4. The paper is mostly well-written and easy to follow.", "reasons_to_reject": "1. I don't think the experiments support the second contribution of the paper. They ignore the development of the LLMs. Previous work shows that the LLMs learn from English resources and improve performance in low-resource languages. The authors haven't conduct experiments to prove that hallucinations emerge more on low-resource languages than English. And they haven't deployed mFact on such experiments.\nI highly recommend the authors report the multilingual performance of LLMs like ChatGPT. If LLMs can achieve better performance than cross-lingual transfer baselines and your methods, proposing a method to mitigate hallucination in a cross-lingual setting will be less meaningful. And previous studies have demonstrated that abstracts generated by ChatGPT rarely contain intrinsic hallucinations.\n2. According to the description in Section 2.1, your pipeline for creating mFACT is too complex. It is model-based, and it need to be trained on the XSum dataset. The Google API is also needed. So, I think mFACT is hard to be applied. The authors should provide more details about the cost to dispel users' concerns.\n3. The weighted loss approach is not easily used to train large models. So, I don't think the mitigating hallucination method will contribute much.", "questions_to_authors": "Would you train different classifiers to obtain mFACT for each language, or would you directly train a multilingual classifier with mixed datasets?", "paper_topic_and_main_contributions": "This paper primarily addresses the issue of hallucination in multilingual summarization. The authors introduce a model-based metric to facilitate the evaluation and detection of hallucinations in low-resource languages. Additionally, they believe that crosslingual transfer methods tend to introduce more hallucinations while improving summarization performance. To address this, they propose a weighted loss method that leverages the mFACT metric.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. If you weight the loss according to the faithfulness score of each sample, it seems that the placement of z(i) in formula (4) is incorrect.\n2. Line 417, ROUGE seems to be faithfulness."}, "extra_text": {}}, {"review_id": "5XXWNXW7ym", "reviewer": "Reviewer_Pf1A", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that mFACT can contribute to the NLP research community and their comments.\n\n**Q**: The proposed mFACT metric need lots of data samples in English for training the model, which could be lack of generalisation ability.\n\n**A**: While we use 40K document-summary pairs, they can be automatically labelled by using English metrics, which achieves promising results on both our classification task, inverse-transfer verification and human evaluation. In terms of data-related costs, we are on par with other popular metrics, e.g., the NLI-based methods we use as baselines [1, 2], which require training specialised models to evaluate faithfulness in the monolingual case. Having said that, we agree that mFACT data requirements might be a limitation in some cases, and we will mention it in the \"Limitations\" section. \n\nReferences: \n\n[1] On Faithfulness and Factuality in Abstractive Summarization. https://aclanthology.org/2020.acl-main.173.pdf\n\n[2] Evaluating the Factual Consistency of Abstractive Text Summarization. https://arxiv.org/pdf/1910.12840.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is the first work that study hallucination in a cross-lingual transfer setting, which can potentially facilitate related research.\n2. The proposed mFACT model-based metric can accurately detect hallucination (95.3%)\n3. The proposed mitigation approach generally reduce the hallucination in the generated summary.", "reasons_to_reject": "1. The proposed mFACT metric need lots of data samples in English for training the model, which could be lack of generalization ability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investages how to measure and mitigate hallucinations of summarisation models in cross-lingual transfer scenarios. The main contributions are:  \n1. In order to detect the hallucination in other languages, the peper proposed  mFACT, a multilingual faithful metric developed based on four English faithfulness metrics by firstly rank top-k and bottom-k hallucinated samples and train a classifier based on the selected samples.\n2. Before mitigate hallucination, the paper investaged the hallucination in a cross-lingual transfer setting, especially in abstractive summarization. The authors found cross-lingual methods can improve the performance for low-resource summarisation but they also amplify hallucinations.\n3. The paper proposed a novel method to migitage hallucination in cross-lingual setting by weighting the loss of training samples based on their faithfulness score. And the results show that the propsoed method can effecitvely reduce hallucination.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed approach focuses on addressing hallucination issues in multilingual summarization, particularly in cross-lingual transfer learning with low-resource languages. It introduces a model-based metric called mFact to measure the faithfulness of document-summary pairs, helping to reduce hallucinations. This metric is created through a meticulous process involving selecting pseudo-faithful and hallucinated pairs, translating them into the target language, and fine-tuning a multilingual BERT model. By leveraging mFact, the approach introduces a weighted loss method to improve summarization quality and reduce hallucinations. 

Pro:
- Introduction of the novel mFact metric, which is quite novel for studying hallucination in cross-lingual settings
- Extensive experiments validate its effectiveness in enhancing ROUGE scores across multiple languages

Cons:
- There are some concerns regarding the complexity of the mFact metric creation, its generalizability, and the robustness of the hallucination mitigation approach
- Need more insights into the practical application of the mFact metric, including discussions on associated costs and potential challenges, would be valuable. 
- Mixed results observed in Hindi and Turkish with no further clarification

Given these points, I would recommend for acceptance of the findings, but not against it being accepted to the main conference if SAC finds it suitable.
Decision: Accept-Main