Evaluating and Modeling Attribution for Cross-Lingual Question Answering
Authors: Benjamin Muller
John Frederick Wieting
Jonathan H. Clark
Tom Kwiatkowski
Sebastian Ruder
Livio Baldini Soares
Roee Aharoni
Jonathan Herzig
Xinyi Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Attribution
Cross-Lingual Question Answering
Multilingual Modeling
Open-Retrieval Question Answering
Attribution Detection
TL;DR: We show that open-retrieval question-answering systems lack attribution in the cross-lingual setting. We collect data in 5 languages and present automatic attribution detection methods to improve the attribution level of these systems.

Abstract: Trustworthy answer content is abundant in many high-resource languages and is instantly accessible through question answering systems — yet this content can be hard to access for those that do not speak these languages. The leap forward in cross-lingual modeling quality offered by generative language models offers much promise, yet their raw generations often fall short in factuality. To improve trustworthiness in these systems, a promising direction is to attribute the answer to a retrieved source, possibly in a content-rich language different from the query. Our work is the first to study attribution for cross-lingual question answering. First, we collect data in 5 languages to assess the attribution level of a state-of-the-art cross-lingual QA system. To our surprise, we find that a substantial portion of the answers is not attributable to any retrieved passages (up to 50% of answers exactly matching a gold reference)  despite the system being able to attend directly to the retrieved text. Second, to address this poor attribution level, we experiment with a wide range of attribution detection techniques. We find that Natural Language Inference models and PaLM 2 fine-tuned on a very small amount of attribution data can accurately detect attribution. With these models, we improve the attribution level of a cross-lingual QA system. Overall, we show that current academic generative cross-lingual QA systems have substantial shortcomings in attribution and we build tooling to mitigate these issues.

Evaluating and Modeling Attribution for Cross-Lingual Question Answering
Benjamin M John Wieting, Jonathan H. Clark?, Tom Kwiatkowski Sebastian Ruder", Livio Baldini Soares²
Roee Aharoni Jonathan Herzig,, Xinyi Wang" 1 INRIA Paris, Gooole DeepMind, "Google Research
Abstract
Introduction
Open-Retrieval Question Answering (ORQA) de- livers promising performance for information- seeking question unswering in about 20 languages (Asai et al., 2021b, 2022; Muller et al., 2022). ORQA models typically consist of a retriever that retrieves documents in a large corpus, followed by generator that generates a short answer based on the top-ranked documents.
Trustworthy answer content abundant in many high-resource languages and is in- stantly accessible through question answering systems-yet this content can be hard to access for those that do not speak these languages. The leap forward in cross-lingual modeling quality offered by generative language models offers much promise, yet their raw generations often fall short in factuality. To improve trustwor- thiness these systems, a promising direction is to aftribute the answer to a retrieved source, possibly in a content-rich language different from the query. Our work is the first study at- tribution for cross- langull question answering. First, we introduce the XOR-AuriQA dataset assess the attribution level of state-of-the the- art cross-lingual question answering (QA) sys- tem in 5 languages. To our surprise, we find that substantial portion of the answers is not attributable to any retrieved passages (up to 47% of answers exactly matching gold refer- ence) despite the system being able to attend directly to the retrieved text. Second, to address this poor attribution level, we experiment with a wide range of attribution detection techniques. We find that Natural Language Inference mod- els and PaLM 2 fine-tuned a very small amount of attribution data can accurately detect attribution. With these models, we improve the attribution level of cross-lingual QA system. Overall, we show that current academic genera- tive cross-lingual QA systems have substantial shortcomings in attribution and we build tool- ing to mitigate these issues.
Recent work in ORQA reached new state of the art by not only retrieving documents in the same language as the query but by also retrieving pas- sages cross-lingualyy, in additional languages (Asai et al., 2021b). This approach is particularly ben- eficial for languages with limited online written content (Kornai. 2013; Valentim et al., 2021), po- tentially allowing users to access information that may not be available in their language.
ORQA models are typically evaluated with maatgigg metrics (e.g., Exact-Match) based on extensive collections of question and answer pairs in multiple languages (Clark et al., 2020; Longpre et al., 2021). However, these metrics are limited in three ways. First, they are inherently hard to scale to real-world applications, they require the collection of gold answers for all the queries. Second, in some cases, the answer can be correct without any overlap with a gold refer- ence (Bulian et al., 2022). Third, short answers are usually not enough to provide trustworthy answer, and users may prefer to access the under- lying source document. To address this last chal- lenge, Bohnet al. (2022) framed new task called Attributed Question Answering (AQA). Given a query, AQA consists of predicting a short answer along with a supporting document retrieved from a large corpus (e.g. Wikipedia).
'The XOR-AttrIAA datasct is available https: //github cww/ reooogleerreeac/ggoogggo tree/master/ master/xattaaaaaa / _attriqa. XOR-AuriQA includes approximately 10,000 annotated examples foster research in modeling and evaluation attribution cross lingual settings.
This work is the first study on Attributed Ques-
tion Answering in the cross-lingual setting. Mca-
Correspondence to (jwieting, poniettingjjhclare Maelaakk]@goolee moggoglle.com.. Work done as an intern at Google Research.
We refer to a system that generates an answer using text

---

XORQA Pipeline
Attribution User Language
($1) In-Language Attribution
In-language annotators needed Translation needed for portion of the passages
Query?
mDPR
Translate
Translate
In-Language
Passages
Translate
Cross-Language
(S2) In-English Attribution
Passages
mGen
Translate
Annotators needed only in English Translation needed for queries, answers and in-language passages
Answer
Figure I: Attribution scenarios for cross-lingual Open-Retrieval Question Answering (XORQA), Given query in a source language, retrieval model (MDPR) retrieves source language cross- lingual documents, which are then used by a generation model, MGEN, produce source language answer. For S1, in-language annotators assess attribution directly in the user's language while for S2, annotators validate attribution in English. We collect data for both scenarios in Bengali, Finnish, Japanese, Russian and Telugu.
suring attribution (Rashkin et al., 2021) in the cross- lingual setting is more complex than in the mono- lingual case. Indeed, in this case, the document supporting the generated answer may be in a lan- guage different from the query and answer. Hence, attribution can be defined in various ways depend- ing on the query, document, and answer languages. this work, we introduce two attribution scenar- ios, namely (S1) in-language attribution for which the attribution is measured in the language of the query, and (S2) in- In-English attribution for which at- tribution is measured in English. We note that both scenarios may require translating portion of the documents, the query, the answer. We illustrate these scenarios in Figure 1.
(Anil et al., 2023) outperforms all the other models despite being fine-tuned on a very small sample of our collected data (250 examples). This result shows the potential of using large language models to create state-of-the-at cross-lingual attribution models using very little annotated data, allowing them to be inexpensively created for use many the languages of the world. Additionally, we find that for Bengali, Finnish, Japanese and Russian, a T5 model fine-tuned on large natural language inference corpora reaches very high accuracy com- pared to the baselines.
Our analysis shows that PaLM 2 can detect more than 86% of attributed answers that are not exactly matching the gold reference, showing that it is a useful alternative exact match evaluation. These answers may be answers that are not captured by the gold references but that are alternative correct answers or they can be answers that are semanti- cally equivalent to the gold reference but that do not overlap with the gold reference (e.g. different units). We discuss these cases in Section 4.2.
Based this framework, we first measure the at- tribution level of CORA (Asai al., 20216), state- of-the-art cross-lingual ORQA system. We collect data in languages: Bengali, Finnish, Japanese, Russian, and Telugu. To our surprise, large portion of generated answers was found not at- tributable to any retrieved passage. For instance, in Japanese, up to 47% of answers exactly matching the gold reference are not attributable. This poor attribution may hurt the trust into QA systems and limit their deployment.
In summary, we make the following four con- tributions: (i) Our work is the first to study attri- bution for question answering in cross-lingual framework. We define two attribution scenarios, in- language attribution and in-English Attribution and annotate approximately 10.000 examples in five languages; (ii) Using this data, we evaluate the tribution of CORA, a state-of -f-the-rttttt cross lingualingual QA system. We show that large portion (7%-47%
To improve the attribution level of cross- lingull QA systems, experiment with wide range of attribution detection models. We show that PaLM
in any language, possihly in a language different from the query, as a cross-lingual QA system.

---

depending on the language) of the answers are not altributable-neithe to in-language passages nor to cross-language ones; (iii) We show that PaLM and NLI models can accurately detect attribution in the cross-lingual setting, significantly outperform- ing all other baselines and reaching above 90% accuracy for all 5 languages. Moreover, our work is the first to approach it with large language mod- els, and we show that with scarce amounts of data (250 examples), they can outperform NLI models trained on millions of examples; and (iv) Using our attribution detection model as a reranker, we show that we reach an average of +55% in attribution compared to a model with no reranking.
We build upon previous work that showed that for many languages (c.g.. Japanese), cross- lingual QA systems outperform monolingual sys- tems (Asai et al., 2021b; Muller et al., 2022). In this setting, given a query in a language L, the goal is to generate an answer using evidence passages from multilingual corpus C.
2.3 Modeling Attributed Cross-Lingual QA
Asai et al. (2022) showed that the best systems- according to the Exact-Match metric- for ques- tion answering in languages other than English are based on a cross-lingual open-retrieval (XORQA) pipeline. We thus model attributed cross-lingual QA with CORA (Asai et al., 2021b), a of-te-ofthhe- art XORQA model. Figure (left-panel) illustrates typical XORQA pipeline.
2
Attribution for Cross-Lingual Question Answering
2.1 Attribution of Generative Language Models
CORA consists of two components: a multilin- gual dense retriever (MDPR) which is based on mBERT (Devlin al., 2019) fine-tuned for dense passage retrieval (Karpukhin et al., 2020), and a multilingual generator (MGEN) based on mT5- Base (Xue et al., 2021) fine-tuned for question answering. Given a query, MDPR ranks all the pas- sages from Wikipedia regardless of their language. In practice, most of the top-ranked passages are ei- ther in the same language the query or in English (we report the language distribution in Table 8 in the Appendíx). Then, the top passages are fed MGEN, which generates the answer. Depending on the languages (and their associated subword to- kenization), the average number of passages varies between 5 (for Bengali) and 10 (for Japanese). CORA is designed to generate short answer using multiple passages. To use CORA for AQA, we must select a single passage supporting the an- swer. In this work, we consider the passages that have been ranked highest by MDPR and fed to the generator as our pool of potential attribution passages. We measure and report the attribution level of answers and passages (a. p) by taking the TOP- --rerrieeed passage by MDPR well as ALL the passages retrieved and fed the generator, Fi- nally, we report in Section 5.3 the attribution level of answers and passages (a, p) after reranking the top passages with our NLI-based attribution detec- tion model.
Generative language models have made impressive progress in the past few years (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020; Chowdhery et al., 2022). They can now perform most NLP tasks with relatively high accuracy in zero-shot and few-shot settings. However, generating text without reference to Human-writtedd trusted sources can be harmful in the real world. Indeed, even the largest models may assign a high probability to false and potentially harmful utterances (Bender et al., 2021; Bommasani et al., 2021; Weidinger et al., 2022). To overcome this challenge, Rashkin al. (2021) introduced the Auributable to Identi- fied Sources (AIS), human evaluation framework for identifying whether source document supports the generated text, or in other words, whether the generations can be attributed to given document.
2.2 Attributed Question Answering
The need for attribution is particularly vivid for information-seeking use cases. To address this need, Bohnet et al. (2022) defined the Attributed Question Answering (AQA) task. Given query q and large corpus of text C (c.g. Wikipedia), AQA consists of predicting an answer a and a passage p E C to attribute the predicted answer.
(AQA) (g.C) + (a,p)
(1)
Bohnet et al. (2022) experimented with the AQA task where the questions, answers, and passages were English. Our work is the first to study this task in a cross-lingual framework.
Recall that the selected passage can be in any language but is typically in English or in the query language (cf. Table 8). This leads us to define two attribution evaluation scenarios.

---

2.4 Cross-Lingual QA Attribution Evaluation
BN
JA
RU
TE
of examples collected ( unique querles exampler)
We introduce two attribution evaluation scenarios illustrated in Figure 1.
(S1)
396 1560
202 806
146
1263
186 7789
323 1212
(82)
305 /570
202 812
1461 1262
186 790
228/ 444
Inter-Annotaior
Agreement:
Agreement
(SI)
91.9
91.0
86.0
90.1
90.3
(S1) In-Language Attribution Evaluation In this scenario, attribution is assessed in the language of the query, while the query, answer, and passage (q, a. p) are in the same language. From appli- cation perspective, this scenario evaluates directly what potential user of an attributed QA system would experience by receiving the answer to their question and the attributed source document their language. As illustrated in Figure I, this scenario involves automatically translating the portion of the passages retrieved in languages different from the query into the query language.
(S2)
94.9
Disagreement between annotation scenarior
S1,452
6.7
10.5
5.2
7.5
S1
11.7
Table 1: Each example (g, a, p) is annotated by 3 in- dependent raters. We report the agreement with con- sensus which measures the proportion of examples that agrees with the majority vote. We report statistics on the in-language autribution scenario (S1) and the in- English attribution scenario (S2). We also report in the ratings collected in (S1) and (S2) ((S1) (S22) and the disagreement on the portion of examples that have been translated from English ((S) ((S1))A(S2) TR.).
(S2) In-English Attribution Evaluation In this scenario, the query, answer, and passage (g, a, p) are all in English during human annotation; we automatically translate the query and answer into English along with the passages retrieved in lan- guages other than English (cf. Figure 1). We im- plement this scenario as it favors scalability, since collecting data in English is usually easier than in other languages due to the availability of raters. Moreover, significant portion of the passages re- trieved by tross-lingus QA systems are in English, assessing attribution directly in English is most straightforward for these passages. For polyglot users. this scenario is also appealing as they may understand English and be interested in accessing the attributed document in English along with the answer in their language.
and passage triplet (g, a, p), attribution evaluation measures the portion of answers a attributed to p.
In contrast, Exact-Match requires gold answer ä to compare to the predicted answer a. In Section 4.2, we show how Exact-Match dif- fers from attribution. We show that some correct answers according to exact-match are not attributed to any retrieved passage, while some non-exactly- matching answers are legitimate answers attributed to reference passages.
The XOR-AttriQA Dataset
To the best of our knowledge, our work is the first to study the problem of attribution in the cross- lingual setting. To make this study feasible, we collect the first multilingual attribution dataset. We use the attribution evaluation framework defined by Rashkin al. (2021). We hire Bengali. Finnish, Japanese, Russian, and Telugu-speaking raters for the in-language scenario (S1) and Einllih-speaking raters for the in-English scenario (S2). Our analysis is based on the XOR-TyDiQN dataset (Asai et al., 2021 a) in Bengali, Finnish, Japanese, Russian and Telugu. To limit cost, we randomly sample about 50% of the validation set except for Bengali and Telugu (S1) annotations for which we take the en- tire set. We retrieve the passages and predict the answers using the CORA system. We only evalu- ate the passages that are fed to the generator. For each (query, answer, passage) triplet, we ask three raters to answer "Is the answer attributed to the pas- sage?".5 To ensure the quality of the data collected
For both scenarios, translation is performed au- tomatically using the Google Translate A
Evaluation Metric For both scenarios, we col- lect evaluation data to assess if a predicted answer can be attributed to a retrieved passage. Following Bohnet al. (2022), we measure the accuracy of system by counting the proportion of answers with an attributed passage. We refer this score as AIS.
We note that this evaluation method fundamen- tally differs from traditional QA system metrics, which are usually based on string-matching meth- ods, e.g., Exact-Match (EM; Rajpurkar et al., 2016; Petroni et al., 2021). Indeed, given a query, answer
Abbout 1.S billion people speak En- glish, as reported by economist.cc woonomist.comgrree wwt.//rraaaiiccttaallle0000.111 where-are-en bhbe-woord---eet-eeggll.. chttps://cloon google. wttee.coo/ttaaeeeeeeeeeeee com/translate
https://www.
Before this step, we ensure that the (query, answer) pair is fully interpretable by the rater. We point to (Rashkin et al.,

---

Raters' Demographic and Cultural Background Even though translating passages does not lead higher disagreement compared to the original pas- sages, we do observe disagreement between (S1) and (S2) (between 4. 4.1% and and 10.5% as reported of answers sxaetly-matching the gold answer attributed to at least one passage fed to the MGEN.
Attributable predictions (AIS) of EM
BN
Fl
JA
TE
67.3
80.4
53.1
67.5
93.1
Table 2:
in Table 1). We partially explain the disagreement by the demographic and cultural context of the raters. For instance, English speakers rated the example <Query: How many countries are there in the United States of America? Answer: 50> as attributed to the passage "The United States of America (USA), commonly known as the United States (U.S. or US) or America, is a country com- posed of 50 states, federal district, five major self-governing territories, and various possessions." while Telugu speakers did not do the same for the example translated into Telugu. We hypothesize that familiarity with the USA and the concept of states made the raters understand the question more loosely and accept the "50 states" mention as supportive of the answer. We leave for future work the careful quantification of this phenomenon.
we report in Table the inter-annotator agreement (IAA). The agreement is above 90% for both the in-language scenario and the In-English scenario for all languages except Japanese. Appendix B provides more detail on the annotation process as well as the agreement with expert annotations on a small sample of the data (cf. Table 9). For each example, we assign the attribution label based on the majority vote of the three raters.
In-English vs. In-Language Attribution As re- ported in Table the inter-annotator agreement observed is similar whether we collect the data in the in-English scenario (S1) compared to the in-language scenario (S2). The only large differ- ences are observed for Telugu, for which the IAA is 8 points above when we collect the data in Telugu. In consequence, we will use the annotation from the in-language scenario (S1) as the gold labels to evaluate all our models (cf. 4 and 5). Indeed, (S1) evaluates what a potential user may experience. So given the fact that (S1) is as good (or better) as (S2) concerning data quality, we decide to select the data from (S1).
Attribution Evaluation of CORA
Based on our newly collected XOR-AttriQA dataset, we now evaluate the attribution level of CORA.
4.1 Lack of Attribution of XORQA Predictions
We start by focusing on the subset of answers that match the gold reference based Exact Match (EM). We hypothesize that these answers at- tributable in most cases and that non-attributable answers should be the exception, not the rule. In- deed, by design, CORA uses retrieved passages to generate answers. Intuitively, it is hard to conceive how the model could generate a correct answer without supporting passages. However, it is known that language models "memorize" knowledge in their parameters (Petroni et al., 2019; Roberts al., 2020), which could enable this ability.
Impact of Translation on Attribution Assess- ment Both scenarios require translating a portion of the data automatically (cf. Fig. 1). We hypothe- size that translating passages from English to the user language may lead, in some cases, to losing attribution. Indeed, assuming that a passage in En- glish supports the answer, we easily imagine cases in which translation errors could cause the translated passage not to carry the information that supports the answer. However, the disagreement be- tween the annotation in (S1) and (S2) is not higher when we look at passages that have been translated compared to passages that have not for 4/5 lan- guages (cf. comparison between row (S1) and row (S2) TR.) as reported in Table 1). In addition, after manually reviewing the data, do not find any cases where translation errors cause disagreement.
We report in Table the proportion of answers that match the gold reference and that are at- tributable to a retrieved passage. To our surprise, we find a very large number of non-attributable answers. For Japanese, only 53.1% of the answers attributed to at least one passage.
We provide examples of non-attributed exactly- matching answers in Figure 2. We find that these non non-attributed answers exactly-machhing the refer-
2021) for an exhaustive definition of interpretabiliit.

---

ABS
KCEM
BMD-EM]
AB8
(FEM
B00-EM
DMN-EM
25,04022
41.86555
22.536.1
36.5446.0
64.7750
25.23477
11,3345
22.4510
K2206
26,29..
45.097755
2344666
22,931,4
690993.1
13,2/1.9
2,38.3
0,000
2.63.
18440
3,653
0.3355
0U00.0
0.3.1
1.0.1
0.00.0
0,3003
1.70,0
0.003
Table 3: % of attributed answers to the TOP-1/ALL passages fed to MGEN. We report AIS (ef. sec. 2.4) on XOR-TyDiQA validation split. of EM corresponds to the % of attributable answers among the Exact-Matched answers. non-EM corresponds to the % of attributable answers among the non Bon-Exact-Mand Matched answers (i.e. that differ from the gold answer). We report the attribution-level considering passages any languages (row ANY), only the English passages as candidates (row EN), only the in-language passages (row LANG.)
candidates.
Query: 21-1 2. 7122 212 FFRR ? Was Karl Marx a
usually the case for Yes/No answers. Some answers are correct and seem to be using the information in the passage provided to the generator. However, in most cases, the information provided in the passage is incomplete to support the answer. This is the case for the second example in Figure 2: "What is the name of the mother tongue of the Marma people?" was answered with "Burmese". While the passage contains relevant information about the Burmese language, it does not draw a connection with the "Marma people" mentioned in the question.
historian? Answer: Yes Gold Answer: Yes
Passage:
7 7 x 72 EIt.
74 1 1
2 ターク 6. LI
( & 727 7
L
$ + 11
&
2
9 b
+ 12 E 1
&
Marsism is one of the socialist thought systems established based on the ideas developed by Karl Marx and Friedrich En- gels. is often called scientific socialism. By turning capital into the common property of society, Marsism abolishes the disastrous nature of wage labour, in which workers live only to multiply capital.
These results show-without ambiguity- ORQA systems, even when they generate correct answers, do not always provide a relevant source passage to support the generated answer. In other words, this means that for a significant portion of answers, ORQA systems are right but without any evidence-they are right for the wrong reasons.
Query: भोशाी अनगनोली Rią Gidld नोम क ? What the name of the mother tongue of the Marma people?
Answer:
बपी Burmese Gold Answer: अीी Burmese
Passage:
atf डरर at uitratatta सभt fog) eea अीडिाा Feg भागीa बााा--प लभाायtt aafe eetn afferरी अनयप fee बाब पायननााार aाारtत,, on दहश uाा सो परर oी अोशाश eeश सराड аτडा करी लसाा रिकर sog महक whs şाtाti аtयतht कया सा war अााा Rie Brorengfite snsero aele Brenet cur aाee alf डागह Benfé पाड रहाोलर मााागाााकर कारिकरय अनाय ot aी wोय (कस नी कमन uाterfrr लभकायीय सपt पहनाा सी नोयोर аअअा शाडि a के (ut डोीरर (sa9 -oort stee) शिोर केा 4w99 boet ett sart महक ths डोयाeि fafox अacttitt etat cve sigfte, setयta, सााी a walht ereta spe exter aė The Burmese language or the language of Myannar is a guage of the noloBBemesee sub- brunch of the Tibeto-Burmess branch of the Tibetai language family. Exactly when the Burmese people came to Myanmar cannot be said. However, the oldest reltgious texts written in Burmese date back to the 10th century AD.. Standard Hurmese thougld to have origi- nated from dialect the lower valleys of central Myanmar. Most people in present-day Myanmur speak alect of this Burmese language, Burmese was influenced first by Pali and then by Mon (12th-13 13th centuries), Then, from the 16th to the 19th century. the language came into contact with warious Europeun languages, such as Portuguese, Dutch, English and French. this
4.2 Analysis of CORA's Attribution Level
We now analyze the attribution level of all answers predicted by CORA, not only the correct ones. We report in Table 3 the attribution level of CORA. Depending on the language, between 1 11.8% (for JA) and 38.7% (for FI) of answers are attributed to the ToP-1 passage retrieved by MDPR. In addition, for all languages, we find that between 31.7-50.95 9% of answers are attributed to at least one passage in the ones provided to the generator (ALL).
regional di-
Impact of Cross-Language Attribution One of the key ingredients of the performance of CORA is its ability to use passages cross-lingually (mainly in English) (Asai et al., 2021b). We now look at how often the generated answers are attributable to these cross-lingually retrieved passages. We find that between 0.3% and 4.0% of answers in Telugu and Finnish respectively can be attributed to an English passage (while not being attributed to any passage in the same language as the query; cf. EN row in Table 3).
Figure 2: Examples of CORA correct answers not at- tributed to any passage. We illustrate how the model can be guided to generate correct answers which are not fully supported by the passage.
ence are of various types. Some of these answers seem to be random guesses from the generator that happen to be matching the gold reference regard- less of the quality of the retrieved passages. This is

---

Attribution vs. Exact-Match In Section 4.1, we found that a large portion of answers exactly matching the gold reference are not attributable. We now look at the answers that are not exactly matching the reference (cf. column non-EM in Table 3). We hypothesize that attribution can po- tentially complement string- mrinng-macchig metrics and find answers that otherwise would be considered incorrect. In Telugu, find that 13.2% of such answers are attributed to the ToP-1 passage. We provide such examples in the Appendix in Figure 3. Some answers are semantically equivalent to the gold reference but are spelled differently or em- ploy different measuring units (e.g.. "crore" used in Telugu vs. "ten million"). Some answers are semantically different for the gold reference but are attributable to passage (e.g., the liver as the largest organ).
models as follows: premise: "Sp" hypothesis: the answer to the question "Sq" is "Sa" where p. q. and a are inserted appropriately.
MT5-QA We use the training splits of the TyDi QA dataset (Clark et al., 2020) to train the attri- bution detection model. We employ the query, passage, answer triplets from TyDi QA as our at- tributed examples (our positive class). For non- attributed examples, we mine negative passages as follows: given a query, we start with the entire Wikipedia document from TyDi QA that answers the query. We sample from this document 10 pas- sages that are different from the positive passage (i.e. the passage that answers the query). This tech- nique provides strong negative passages by pro- viding passages that are topically closely related to the positive passage but that do not answer the question. was used successfully by Garg et al. (2020). We fine-tune mT5-XXL (Xue et al., 2021) on the concatenation of the training data in English, Bengali, Finnish, Japanese, Russian and Telugu.
5 Attribution Detection for XORQA
So far, we found that tlate-of-the-ar cross-lingual question answering systems lack attribution. We showed that a large portion of answers are not at- tributed any passages, by collecting large col- lection of attribution data in five languages. However, collecting attribution data costly and time consuming. In a deployment setting, would simply be infeasible to annotate every (query, an- swer, passage) triplet. In order to address this issue, we explore automatic attribution detection tech- niques. We build upon previous work on ground- ing and factual consistency in English (Honovich et al., 2022). We also experiment with PaL.M 2 (Anil et al., 2023) a new ofate-of-the-art multilin- gual large language model (LLM) in few-shot and scarce data (250 examples) settings.
(M)T5-NLI Following Honovich al. (2022) who found that NLI-fine-tuned T5 is accurate for factual consistency detection, we experiment with several English and multilingual NLI models. Sim- ilar to Bohnet et al. (2022), we make use of the best English NLI model from Honovich et al. (2022), T5- 1 IB model fine-tuned ral language inference datasets, fact verification, and paraphrase detection datasets. We experi- ment with it in the translate-test setting (noted T5- NLI TRANSLATE-TEST) for which we translate the queries, passages, and answers to English.? To model attribution detection in multiple languages, we fine-tuned the mT5-XXL model (Xue et al., 2021) on translations of the mixture of the NLI datasets to the non oon-Eihlish languages (noted MT5- NLI TRANSLATE-TRAIE
mixture of natu-
5.1 Attribution Detection Models
Given a query q, a short answer a and a passage candidate we frame attribution detection as a binary classification task:
To better model the portion of passages a language different from the query, we also fine-tune the model by adding exam- ples for which only the hypothesis has been trans- lated while the premise is kept in English (noted MT5-NLI X- -ETAASLLAT--TRR.. TRAINN.
(g, a. -> ais
with aïs E 1). 1 corresponds to the attributed class (i.e., the answer is attributed to the passage) and 0 corresponds to the non-attributed class. We note that query and answers are always in the same language (in Bengali, Finnish, Japanese, Russian or Telugu), while the passage may be in a different language (mainly English). Following Honovich et al. (2022), we model this task by prompting the
PALM 2 FEW SHOT To avoid costly fine-tuning, we experiment with in-context learning using the
'TS is fine-tuned on the concatenation of the MNLI (Nan- gia et al., 2017), SNLI (Bowman et al., 2015). FEVER (Thome et al., 2018), PAW'S (Zhang et al., 2019). SciTaiL (Khot et al., 2018) and VitaminC (Schuster et al., 2021) datascts.
'Exeept for the portion of the passages retrieved in English that do not need translation.

---

Model
Tuning Data (f)
Inference
BN
FT
RU
TF
MATCH
Ø
IN-EN
83.9/72.0
85.4/75.8
91.97717
86.877.55
84.5
STRING- MATCH
IN-LANG
77.1
78.3
904/77.3
8755/00
88.6/8..
MT3 QA-TRANSLANEE-EST
TyDiOA (-100A
IN-EN
88.07917
8555/91
90.27902
86.67924
878799.2
MTSQQA
TyDiQA 100k)
IN-LANG
89.4/92.0
88,3/922
91.5/92.9
91.0/94.7
92.4/96.8
15-NLL-TRANSLATEE
(-M(-
IN-EN
91.87952
9T.2/96.
95.1795.7
92.4796.0
94.2795.T
MT5-NLI TRANSLATETT
(~
IN-LANG
91.1/93.8
90,4/94.6
93.0/93.8
92.9/96.1
93.8/96.0
M FINESTUNED
Atrribation (-3000)
IN-LANG
81.9771.
8
94.57643
88,777877
PAI.M RINETTUNDD
Atribution (~100)
IN-LANG
/2.3/95.0
92.6/97.2
96.4795.9
94.5/
94.8/96.8
PALM 2, LORA-TUNED
Attribation (~1100)
IN-LANG
91.5//6.0
88.3/96.4
94.7/97.0
93.7/97.8
93.7 /9.8 96.8
PALM 2 24-SHOT
Attribation (>-4)
IN-LANG
91.5/87.3
87.4/8.11
92.0/85.6
70.5
90.6/77.3
PALM 2 2--SHOT W/ CoT PROMPT
Attribution (+-4)
IN-LANG
83.7/86.8
85.
71.7800 80.4
88.
84.7 88.6
Table 4: Performance of Attribution detection models. We report the Accuracy ROC AUC scores the XOR-AttriQA dataset. The Accuracy is computed with the probability threshold that maximizes it on an inde- pendent set. For each language, the best Accuracy/ ROC AUC scores are bolded and the second best scores are underlined.
PaLM 2 large language model (Anil et al., 2023). We use the Small version and evaluate the model af- ter prompting the model with 4-shots with and with- out chain-of-thought prompting (Wei et al., 2022). Each language is evaluated with its own prompts, and two negative examples and two positive exam- ples sampled for each language. For each pair, one passage is chosen be in-language while the other is chosen to be in- in-English. th is done by manually writing rationale that ex- plains the attribution (or lack attribution) a given answer in English.
answer and passage to English.
5.2 Results
We report the accuracy and ROC-AUC (Flach al., 201 I) scores in Table 4. We compute the predic- tion with a decision threshold tuned on an inde- pendent validation dataset on which we measure the accuracy of the model. PaLM outperforms all the other models despite being fine-tuned very small sample of data, which is encourag- ing as it shows we can leverage LLMs for cross- lingual attribution with very little annotated data that is expensive to produce. We also found few- shot performance to also have strong results, that could probably be improved with more shots and leveraging larger LLMs. NLI fine-tuned models outperform MT5-QA and STRING-MATCH for all the languages in the translate-test setting. Finally, we report in Table the portion of attributed an- swers not matching the gold references that the best AIS detection model accurately predicts. We find that PaLM 2 accurately predicts more than 86% of these answers (between 86.3 and 96.4% depending on the language). This shows the po- tential of using attribution detection to expand the space of legitimate answers beyond relying only on mrrinngmatching metrics.
MT5 / PALM 2 ATTRIBUTION Finally, we experiment with fine-tuning directly on a small sample of the attribution data collected. We sample 250 examples in the 5 languages and fine- tume mT5-XXL (Xue et al., 2021) and PaLM Small (Anil et al., 2023). For mT5 we fine-tune the entire model, while for PaLM 2, we both fine-tune on the whole dataset and also fine-tune with Low- Rank Adaptation (LoRA) (Hu et al., 2021) to avoid overfitting and reduce fine-tuning cost. For these experiments, use the same constant learning rate of 0.0001, dropout rate (Srivastava et al. 2014) of 0.1, and batch size of 128 for tuning both mT55 and PaLM 2. For fine-tuning with LoRA, we used a learning rate of 0.00005 and tuned the model over ranks in {4, 16, 64, 256). For all models, we used the validation set for checkpoint selection.
STRING-MATCH We define simple baseline. For answers that are not "Yo"","No", if the string a is included in the passage p. we predict 1, oth- erwise O. This means that we consider the answer be attributed to the passage if it is included in it. For Yes/No answers, we predict 0 (the majority class). We also use it after translating the query,
BN
FT
JA
RU
TE
Acc.
88.2
94.5
86.3
96.4
91.6
Table 5: % of attributed non-EM examples that are accurately detected by PALM 2.

---

Lang.
TOP-1
ALL
T5-NLI reranked
rerank passages improve the attribution-level of state-of-the-at XORQA system.
EN
27.9
45.6
39.2 (+40.4%)
38.7
50.9
46.0 (+19.0%
Our result points to two critical research di- rections to further make progress in information- seeking QA. First, we observed that in some lan- guages (e.g.. Telugu), cross-lingual passages con- tribute very moderately to the attribution level. This shows that more progress is needed in cross-linguat retriever systems. Second, we showed that string- matching metries based on gold references are in- herently imperfect evaluation methods for QA and showed that PaLM can be used to detect rele- vant attributed passages accurately with only small amounts of training data. This means that large language moodels-based attribution detection can potentially be used as evaluation metrics for QA in multiple languages. Further work is needed design robust LLM--bsed metrics for cross-lingual infornntion-ssekking QA.
11.8
37.3
29.1 (+146.2%)
RU
27.5
40.9
39.6 (+43.99)
23.3
31.7
30.3 (+30.0%)
Avg. 25.8
41.3
36.8 (+55.9)
Table 6: % of attributed answers based the top-1 MDPR-retrieved passage, AILL the passages retrieved fed to the generator, and the TOP-1 reranked passage (TS-NLI reranked) with TS-NLI-TRANNLLATT-TI our best NLI fine-tuned model.
5.3 NLI Model for Reranking
Using our best T5-based attribution detection model (T5-NLI TRANSLATE--EEST) we now come back to our original goal of improving the attribution level of our cross- mooss-lingual question an- swering system. We leave for future work the use of PaLM for reranking.
Given our pool of candidate passages, we use our attribution detection model as a reranker and se- lect the passage which is the most likely attribute the answer according to the model. We report the reranking attribution score in Table 6. We find that our NLI model can accurately rerank the passages. For instance for Telugu, we are able to increase the top-1 performance from 23.3 to 31.7, an im- provement of +30.0%. Across all the languages, reranking with our NLI model leads am average relative increase of 55.9% across the 5 languages.
7 Conclusion
By ensuring that the model predictions are sup- ported by human-written text, attribution one the most promising ways to deploy NLP systems safely. In this work, we introduced and released the XOR-AttriQ dataset that includes approximately 10,000 examples in Bengali, Finnish, Japanese, Russian and Telugu. Thanks XOR-AttrQA. we observe that state-of-the-arr QA systems lack attri- bution in the cross-lingual setting. We showed that PaLM 2 and NLI models are promising methods to detect attributed passages typologically di- QA. Hav-
6 Discussion and Future Directions
verse languages for information ing provided evidence for the lack of attribution in academic generative QA system, built tooling to detect and mitigate these issues, and releasing our collected attribution data in 5 languages, we hope to enable trustworthy cross-lingust QA systems meet the information needs of people around the world.
Language model-based NLP systems are making fast and continuous progress in generating fluent and helpful text. Despite this progress, these mod- els still make a lot of factual errors, specifically in languages different from English. Attribution the most promising approach addressing this issue (Rashkin et al., 2021). Our work finds that even the best XORQA system predictions lack attribution. These results can be explained by the tendency of these models memorize facts (Roberts et al.. 2020) and to hallucinate answers (Ji et al., 2023), which are in some cases correct. This shows that we need make progress in detecting and selecting attributed sources that support the generated answer cross-lingure QA systems. In this work, we pro- posed to use a large language model (PaLM 2) and natural language inference models to detect and
Limitations
Our work focused on evaluating and improving the attribution level of a state-of-the-an XORQA pipeline. Given recent progress, LLMs are now increasingly used in a closed-bou bookk setting (Roberts et al., 2020) for question answering, i.e., without relying on any retrieved passages to answer a ques- tion. Attributing the generations of these models is therefore becoming critical. addition to improv- ing the attribution level of open-retrieval question
"Using translation ensures that everything is in the same language potentially improving the string-matching accuracy.

---

answering pipeline, we hope XOR-AttriQA and the attribution detection experiments we presented will also be used to design attribution detection models for closed-book QA systems.
References
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint ar 2r05v 10403.
Contributions
Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021a. XOR QA: Cross-lingual open-retrieval question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 547-564, Online. Association for Computa- tional Linguistics.
In this section, we provide more detail about the contributions of each author.
General Overview
Primary Contributors Benjamin Muller, John Wieting
Akari Asai, Shayne Longpre, Jungo Kasai, Chia- Hsuan Lee, Rui Zhang, Junjic Hu, Ikuya Yamada, Jonathan H. Clark, and Eunsol Choi. 2022. MIA 2022 shared task: Evaluating cross-lingual open- retrieval question answering for diverse languages. In Proceedings of the Workshop on Multilingual In- sormation Access (MIA), pages 108-120, Seattle, USA. Association for Computational Linguisties.
Major Contributors Jonathan H. Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Baldini Soares
Supporting Contributors Roee Aharoni, Jonathan Herzig, Xinyi Wang
Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Ha- jishirzi. 202 One question answering model for many languages with cross-lingual dense passage trieval. Advances Neural Information Process- ing Systems, volume 34, pages 7547-7560. Curran Associates, Inc.
By Contribution
Advising John Wieting, Jonathan Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Baldini Soares, Roee Aharoni, Jonathan Herzig
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, FAceT '21, page 610-623. New York, NY, USA. Association for Computing Machinery.
Code Benjamin Muller, John Wieting, Livio Bal- dini Soares, Roee Aharoni, Jonathan Herzig
Data Collection Benjamin Muller, John Wicting
Experiments Benjamin Muller, John Wieting
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni. Daniel Andor, Livio Baldini Soares, Jacob Eisenstein. Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint 2rXiv:2212.0800)
Modeling Benjamin Muller, John Wieting, Xinyi Wang, Roee Aharoni, Jonathan Herzig
Project Design Jonathan H. Clark, John Wieting, Benjamin Muller, Tom Kwiatkowski, Sebastian Ruder
Rishi Bommasani. Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bemstein, Jeannette Bohg. Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint Nxiv:2108.072555
Project Initiation Jonathan H. Clark, Tom Kwiatkowski
Acknowledgements
Samuel R. Bowman, Gabor Angeli, Christopher Potts. and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empiri- cal Methods Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
We thank the raters involved in the data collec- tion process for their work. In addition, we want thank Michael Collins, Dipanjan Das, Vitaly Nikolaev, Jason Riesa, and Pat Verga for the valu- able discussion and feedback they provided on this project.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda

---

Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nlya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Ziwei Ji. Nayeon Lee, Rita Frieske. Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv, 55(12).
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Jannis Bulian, Christian Buck, Wojciech Gajewski, Ben- jamin Börschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 291-305, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science q answering. In AAAI Conference on Artificial Intelligence.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
András Kornai. 2013. Digital language death. PLOS ONE, 8(1)):-11.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for mul- tilingual open domain question answering. Transac- fions of the Association for Computational Linguis- tics, 9: 1389-1406.
Jonathan H. Clark, Funsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolacv, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typo- logically diverse languages. Transactions the As- sociation for Computational Linguistics, 8:454 470.
Benjamin Muller, Luca Soldaini, Rik Koncel- Kedziorski, Eric Lind, and Alessandro Moschitti. 2022. Pooss-lingul open-domain question answer- ing with answer sentence generation. In Proceed- ings of the 2nd Conference of the Asia-Pccific Chap- ter of the Association for Computational Linguisties and the 12th International Joint Conference on Natu- rul Language Processing (Volume I: Long Papers), pages 337-353, Online only. Association for Compu- tational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel Bowman. 2017. The RepEval 2017 shared task: Multi genre natural language inference with sentence representations. In Proceedings of the 2nd Workshop on Evaluating Vector Space Represen- tations for NLP, pages 1-10, Copenhagen, Denmark. Association for Computational Linguistics.
Peter A. Flach, José Hermández- Orallo, and C. Ferri. 2011. A coherent interpretation of auc as a measure of aggregated classification performance. In Interna- tional Conference on Machine Learning.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020. Tanda: Transfer and adapt pre-trained trans- former models for answer sentence selection. In Proceedings of the AAAI conference on artificial in- telligenee, 05, pages 7780-7788.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Or Honovich, Roee Aharoni, Jonathan Herzig. Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document- coamment-grounaeee grounded Dialogue and Conversational Question Answering. pages 161- 175, Dublin, Ireland. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and

---

Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational L Linguistics.
knowledge propagation across wikipedia languages. Proceedings of the International AAAI Conference Web and Social Media, 15(1): 1046-1022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Xia, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng. Borja Balle. Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeha Birhame, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick. Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of risks posed by language models. In 2022 ACM Conference Fairness, Accountabil- ity, and Transparency, FAccT '22, page 214-229, New York, NY, USA. Association for Computing Machinery.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m5: A massively multilingual pre- pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Hannah Rashkin, Vitaly Nikolaev, Mathew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gau- rav Singh Tomar, Iulia Turc, and David Reitter. 2021. Measuring attribution in natural language generation models. CoRR, abs/21 12. 12870.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418 5426, Online. Association for Computational Linguisties.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1298 1308. Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. pages 624 643, Online. As- sociation for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):129-1195)
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Rodolfo Vieira Valentim, Giovanni Comarela, Souneil Park, and Diego Trumper. 2021. Tracking

---

Appendices accompanying "Evaluating and Modeling Attribution for Cross-Lingual Question Answering"
Table 8 shows the language distribution of pas- sages retrieved by MDPR. Most passages are either in the same language as the query or in English.
A System and Data
B Data Collection
A.1 Codebase
B.1 Inter-Annotator Agreement Details
To rum inference with CORA (Asai et al., 2021b), we used the original codebase released by the authors available at https: //gi thub. com/ AkariAsai/CORA, To build attribution detection models with T5, we used the original check- points from (Xue et al., 2021) fine-tuned using the 15x library available at https: //g thub.ccm/ google-researchhtt5.
Lang.
Consenaus w. Expert Agreement
Individual Rater
with Expert Agreement
BN
IN-EN (S2)
86.49
84.68
IN-LANG (ST)
96.67
93.33
IN-EN
JA
IN-LANDE (81)
93.62
91.19
IN-EI
83.33
72.22
93.75
RU
IN-EN
100
88.89
TE
(ST
6..75
86.35
IN-EN (S2)
91.23
85.10
A.2 XOR-TyDiQA
Table 9: Attribution annotations quality: report the agreement between hired raters and expert judgment on small sample of data.
Language
yes/no
short spans
all
BN
35.3
45.8
45.6
NB: Expert IN-LANG evaluation is done based trans- lated queries and answers.
45.2
52.1
50.9
37.0
37.6
37.3
RU
36.0
41.2
40,9
TE
31.7
31.7
We report in Table the agreement between expert raters and hired raters on small number of examples. We find that this agreement is above 90% for all languages in the in-language scenario (S1).
Table 7: % of CORA's answers that are attributed to at least one passage per answer type.
All our experiments are based on the XOR- TyDiQA dataset (Asai et al., 2021a) available at https://nlp: camnpp.cs..... washington. adduxooqq// We focused on Bengali, Finnish, Japanese, Russian and Telugu data. We only used the query and gold answers from XOR-TyDoO Tyy-AA (and ignored the gold passages for which we use retriever). XOR- TyDiQA answers are of two types: Yes/No answers short spans extracted from Wikipedia passages (Clark et al., 2020). We report in Table 7 the dif- ference in attribution between Yes/No and short span answers. We find that for most languages, Yes/No answers are less attributable compared to short answer spans.
B.2 AIS Score
The AIS data collection framework (Rashkin et al., 2021) consists of two annotation steps. First, the raters are shown a question and answer and asked "Is the answer interpretable to you". If the response is positive, the rater shown the source passage and asked "Is the answer attributed to the passage". At each step, the rater is asked to answer Yes, or No or to flag the example if it is corrupted. For each question, answer, and passage triplet (g. P), we collect the rating of three raters (for each annotation scenarios). These three ratings aggregated to get single label 0 1 for each (q, a, triplet with the following criterion:
A.3
Languages Distribution of MDPR
Language Distribution of Retrieved Passages
We only keep the examples that received at least two ratings. This means that we exclude examples flagged by two raters more.
Query lang.
BN
ki
JA
RU
TE
Passage lang.
IN-LANG
44.9
69.2
82.8
87.3
49.1
22.9
15.7
10.6
40.0
OTHERS LANG.
5.9
7.9
1.5
2.1
4.2
We assign the label "attributable" (1) the triplet if the example received at least two votes to the question "Is the answer attributed to the passage"; otherwise, we set the label to non-attributable (0).
Table 8: Language distribution of passages retrieved by multilingual dense retriever (MDPR from (Asai al.. 202 1b)). IN-LANG means the passage is in the same language as the query.

---

The number of examples collected is available in Table I.
Query: Stny 36 tracgn 280667 What capital of
Kenya?
Answer 2000 Naimbi Gold answer: 2ld Nairobi
Examples of Attribution without Exact-Match
Passage:
Soog (eorñ Republic of Kenya) Dags esip tory, dordy eag e'd 2.5 cdo. di &eoon shardar dowigg strrrr dipe éroarodár deew ree)). Go orsigno 2G0. Kenya (English Republic of Kenya) The Republic of Kenya is a country in Fast Africa. is bordered by Eshiopia to the north, Somalla capital Nairobi.
northeust and Tangania to the south.
Query: dods gaodcingo Seler 2odiinnf dašrärö7 How many people died on average World War II?
Answer: edth Eey Six crores
Gold answer: 70-85 andci 70-85 millions
Passage:
Egy. wodse" angfety cheee as disinss seing ctu@ Sches adsByaron (esārr, ba5--ad sicren élsocc wodartse! End artodió 5ø 85om diugie" drensod Biseoo. evtnom eair dre eggs, arogaus rrosss adchesyoel dug SarerreSiab ardded sajja, Nosrdd ests Eqriood dag@8 sdmbonsd diugoo gooos d0get3 vdjos dg Sgabouddrr 38000d. Boda d0os auggoo aoddond wros aandow dads sodve ragedes arosek as eccaar see' Noärth as salabod Sosr drrdy wtae stoabd dinaabs d's and adye wods aisees ara atrjess seggasoe" geeree aeRepsardd (áxš we's end ăds). good angoor & diogo Soñows egg isgo Kodid s8 uge 56 w3.85 wew (1944 wel wodd devs (1760) Gotwod The countries involved faced kind of perfect simation (ie, all available, regardless militar--cvvilac dis- tinctions, were involved in the war in some way). As result, all the economic, industrial and technological resources of the respective countries had to be used for war purposes. This war known as bloodiest the history the world, which caused the death about six
people.
Query: armss 46doō'a te ag esáchão S6 Which the largest organ in the human body?
Answer: dddd the skin Gold answer: Passage:
the liver
Seaa (Skin) ass sódoce eaaag essebaam deee aares Sasggass erdenoeran. addia aodasow sāy etae grrpey dggoo aasdogressg aido e'arpoá wa aaa dotieeee doewoB. áamaa nassauaas ages winay 'Bdjerea soarts. ädjjee siaayor arssaadioo eooáio ee Bocs shdewoeras. ancyssijo asrjfão sood Sdyaaboa.. Bissen, âgd fodues anöydonjs Boāso. fay ina Aosio Sõjáeran. eâs ddatves sher #oord. agø möd adaiies āyr Goends. wdai asssy sroore Sases 3866558 ásdges doerth. & aggerees sidrao Stipe's 'Beess wi donsscrgo.oow melanin is called albuns.
Skin is the largest organ in our body. It has three important layers. The skin covers the entire body and protects the inter- parts. Skin is lacking at the pores. comes different colors. The science of skin called 'Dermatoloyy' The skin mainty has two layers namely epidermis and dermis. The epidermis is forned from the epidermis. Hairy and sweat glands belong to the epidermis, Nails are also formed from it. Africans are black. Northern Europeans are white. The people of some other parts of Asia of these color differences the pigment called 'melanine the skin. low melanin is called
between the The
Figure 3: Examples of attributed answers that are not matching the gold reference (noted non-EM in Table 3). We display single passage fed to the generator, which the answer is attributed.


Reviews:
[{"review_id": "jUo18HJPaB", "reviewer": "Reviewer_PwG7", "summary": "", "questions": "a. Though palm 2 is only finetuned on a few samples, it is still very costly for a binary classification model. Is there any intuition on why in–context learning does not work well for the attribution task?\n\nb. Why does T5-NLI perform better than mT5-NLI in most cases? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. \n\nWe want to reaffirm some distinctions between our attribution detection models and standard retrievers. First, the retriever is very inaccurate to provide attributed passages. As we reported in Table 6, only 25.8% on average of the top-1 passage provided by the retriever is an attributable source for the predicted answer. This means that a passage selection step is needed to improve this number. Second, our attribution detection system takes as input the (question, passage, answer) triplet, making the attribution detection tailored toward the predicted answer. Using this approach, we find that we are able to improve the attribution level with our attribution detection model by +55.9% on average.\n\nWe showcased our system using Wikipedia since that is the domain of the XORQA data and the state-of-the-art CORA model we study. Indeed, further work is needed to showcase our approach on a larger number of domains (e.g., medical, legal text). Our approach is, however, directly applicable to other domains (given an existing XORQA pipeline).  Additionally, attribution detection for general knowledge information-seeking question answering requires a trustworthy corpus of text (as described in \"Measuring attribution in natural language generation\" by Rashkin et al. (2021)). Due to its scale, multilingual coverage and overall good quality, we believe Wikipedia is the most relevant corpus to attribute answers for information-seeking QA.\n\nThe amount of data used for QA is an order of magnitude smaller compared to data to train the NLI models. The number of available multilingual QA training resources is limited. The XOR-TyDiQA was the only resource known to us that includes questions, answers, and full documents allowing us to build an attribution detection training dataset (with both positive and negative passages). We agree that a combination of QA and NLI fine-tuning data will likely improve the performance even more. \n\nThank you for the feedback about the training details. We will add the computing cost and training hyperparameters to the paper. In summary, we used a batch size of 32 and a constant learning rate of 0.0001. All the training runs were done for up to 50k steps, and the best checkpoint was selected on a validation set that follows the training distribution. In total, the training runs for the NLI attribution classifiers took about 100 hours on a Google Cloud TPUv3 Pod slice with 32 chips. LoRA tuning for PaLM 2 also used a Google Cloud TPUv3 Pod slice with 32 chips and took about 6 hours.\n\nWe will expand our Limitations section as per your suggestion (including limitations such as a single domain, using CORA only, training costs, etc.). We agree and thank you for pointing this out.\n\nUsing PALM 2 with in-context learning for attribution detection did not compete with the other approaches. Attribution detection requires discriminating between attributed passages and passages that seem supportive of the answer but that are not (see Table 3 for examples). Simple in-context few-shot learning did not deliver good results, significantly below the NLI models, but we believe further prompting exploration could improve the performance. These results, with in-context learning lagging behind fine-tuning, are consistent with the literature as well. For instance, this topic is discussed in Section 4.3 of \"XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages\" by Ruder et al. (2023) in a similar setting to ours where in-context learning was not as effective as fine-tuning. We include additional results below for in-context learning using 4-shot Chain-of-Thought prompting on PaLM 2 M (using reasoning steps we provide):\n\n| Language | Accuracy | ROC AUC |\n|-----------------|-----------------|-----------------|\n| BN | 79.8 | 86.4 |\n| FI | 77.1 |84.9 |\n| JA | 79.2 | 88.0 |\n| RU | 74.8 | 85.5 |\n| TE | 90.0 | 50.0 |\n\nThese in-context learning results are well below that of the fine-tuning experiments in Table 5. We will include these results and discussion in the next version of the paper. Note that for in-context learning for TE, the model became stuck repeating a portion of the prompt.\n\nThe difference between T5-NLI and mT5-NLI is a combination of two factors. T5 is, generally speaking, a better language model for English, than mT5 is for other languages. This is mainly due to English having significantly more pretraining data than other languages. Second, mT5-NLI relies on translated fine-tuning data. This may hurt the generalization of the model to attribution detection. \n\nAs for reproducibility, our results use publicly available pretained models (with the exception of PaLM-2 which requires an API) and we will be releasing our ~10,000 example dataset to the community as promised in the footnote of the abstract. A preliminary version of the data is [here](https://www.dropbox.com/scl/fi/vyn5jve3mrxgctadays8p/xor-attriqa-v1.0.zip?rlkey=se3xxdi3cr5668zdd1cnlr6ci&dl=0).\n\nThank you again for the valuable comments; we hope our answers will bring more clarity about our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and easy to read. It introduces attribution in cross-lingual QA.  In addition to presenting existing attribution problems in the current system, the paper also collects new annotations, proposes two methods for attribution classification and could further improve QA performance. ", "reasons_to_reject": "- It’s great to see the paper tries to improve explainability in QA systems by attributing answers to retrieved documents. However, framing attribution detection as a binary classification problem does not provide further clarity compared to the ranking provided by the retriever.\n- Domain is limited to wikipedia articles. There’s a lack of discussion about how this work could be extended to other domains.\n- Comparison between mT5–QA and mT5-NLI. The amount of data used to finetune these two variants are significantly different. I’m wondering if adding more QA data to the former, would that bring the gap closer? Or training on a combination of QA and NLI data would further boost the performance?\n- There’s no mention of the compute cost and the training details. \n- The authors appear to have not invested much thought into the limitation section, as it does not discuss the paper's own limitations.l\n", "questions_to_authors": "a. Though palm 2 is only finetuned on a few samples, it is still very costly for a binary classification model. Is there any intuition on why in–context learning does not work well for the attribution task?\n\nb. Why does T5-NLI perform better than mT5-NLI in most cases? \n", "paper_topic_and_main_contributions": "This paper studies attribution of QA in a cross-lingual setting. It introduces two attribution scenarios, in-language and in-English attribution. To assess  existing cross-lingual QA models, they collect 10000 human annotations in 5 languages. They show that in the existing model, a large portion of the answers generated are not attributable. The paper also introduces attribution classifiers by finetuning a small sample on PALM 2 or finetuned T5 variants. Using such classifiers to help rerank the passages, they are above to further improve original system performance. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Shen et al., 2023: xPQA: Cross-Lingual Product Question Answering in 12 Languages", "Typos_Grammar_Style_and_Presentation_Improvements": "Missing % in the last row of Table 6"}, "extra_text": {}}, {"review_id": "WqcbVJpVA6", "reviewer": "Reviewer_TzMb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your overall positive review. \n\nIndeed, our paper only displays the evaluation of a single cross-lingual open-retrieval pipeline (CORA). We managed this project on a budget constrained setting for data collection, and even then we were able to gather about 10,000 data points annotated 3 ways. We, therefore, prioritized both the language-scaling aspect (so we could cover five typologically diverse languages) and evaluating two attribution settings (in-language and in-English attribution) instead of evaluating more models. However, we note that CORA is a standard open-retrieval pipeline, and our approach handles any (query, passage, answer) and is not restricted to CORA outputs. Therefore, we believe our findings on improving attribution will generalize to other pipelines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A novel dataset in a novel setup of cross-lingual question answering.", "reasons_to_reject": "The evaluation of attribution has been done for only one model, although being popular but not the best in current time. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is devoted to a novel setup in cross-lingual question-answering, namely attribution evaluation. The authors present also a dataset for evidence evaluation and the evaluation results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hkBy9tf5QB", "reviewer": "Reviewer_zM5q", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We share your excitement. We agree that attribution is a very promising approach to addressing factual errors, and we hope our work makes a meaningful contribution to building better attributed systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Attribution is the most promising approach in addressing factual errors. This paper finds that even the best XORQA system predictions lack attribution. This paper proposed to use a large language model and natural language inference models to detect and rerank passages to improve the attribution-level of a state-of-the-art XORQA system.", "reasons_to_reject": "I don't see a big reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors make the following four contributions:\n1. Their work is the first to study attribution for question answering in a cross-lingual framework. They define two attribution scenarios, in-language attribution and in-English attribution and annotate about 10,000 examples in five languages;\n1. using this data they evaluate the attribution of CORA, a state-of-the-art cross-lingual QA system. They show that a large portion (7%-47%, depending on the language) of the answers are not attributable neither to in-language passages nor to cross-language ones;\n1. they show that PaLM 2 and NLI models can accurately detect attribution in the cross-lingual setting, significantly outperforming all other baselines and reaching above 90% accuracy for all 5 languages; and\n1. using their attribution detection model as a reranker, they show that the model reach an average of +55% in attribution compared to a model with no reranking.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work is the first to study attribution in cross-lingual QA with two scenarios: in-language and in-English attribution. The authors collect multilingual attribution datasets for languages with diverse typologies and analyze the potential confounding factors in the data collection process such as translation quality, interannotators’ agreement, and the annotators’ background. Then, they present the existing attribution problem in the current ORQA systems and perform error analysis. Finally, the authors presented solutions that help cross-lingual attribution for LLMs and improve QA performance substantially. The authors sufficiently addressed reviewer PwG7’s confusion between the retriever and their proposed attribution detection models and justified the study on the Wikipedia domain in their work. The authors also draw upon the results from the paper to answer reviewers’ questions, which suggests that the experimental setup is careful and sound. Note that in the next iteration of the paper, the training costs should be included as the paper works with extremely large LMs such as PaLM 2. 

I think the work is timely as the community is working on the trustworthiness and explainability of large language models that are currently widely used for user-facing QA tasks. Agreeing with reviewer PwG7, I think the paper is well-written and easy to read.
Decision: Accept-Main