Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
Authors: Kosuke Akimoto
Kunihiro Takeoka
Masafumi Oyamada
Venue: EMNLP 2023 Findings (2023)
Keywords: retrieval-augmented generation models
fusion-in-decoder
question answering

Abstract: Retrieval-augmented generation models augment knowledge encoded in a language model by providing additional relevant external knowledge (context) during generation. Although it has been shown that the quantity and quality of context impact the performance of retrieval-augmented generation models during inference, limited research explores how these characteristics affect model training. This paper explores how context quantity and quality during model training affect the performance of Fusion-in-Decoder (FiD), the state-of-the-art retrieval-augmented generation model, in extractive open-domain question answering tasks. Experimental results suggest that FiD models overfit to context quality during training and show suboptimal performance when evaluated on different context quality. Through the experimental results, we also reveal FiD models trained with different context quality have different cross-attention distribution patterns. Specifically, as context quality during training increases, FiD models tend to attend more uniformly to each passage in context. Finally, based on these observations, we propose a method to mitigate overfitting to specific context quality by introducing bias to the cross-attention distribution, which we demonstrate to be effective in improving the performance of FiD models on different context quality.

Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
Kosuke Akimoto and Kunihiro Takeoka and Masafumi Oyamada
NEC Corporation (kosuke_a, k_takeoka, oyamadaJenec.coo
Abstract
information that arises after their training phase (Kasai et al., 2022).
Retrieval- -augeeeell generation models aug- ment knowledge encoded in language model by providing additional relevant external knowl- edge (context) during generation. Although has been shown that the quantity and quality of context impact the performance of retrieval- augmented generation models during inference, limited research explores how these character- isties affect model training. This paper ex- plores how context quantity and quality dur- ing model training affect the performance of iuscodr (FiD), the state-of-the-art retrieval -uggmeted generation model, in tractive open-domain question answering tasks. Experimental results suggest that FiD mod- els overfit to context quality during training and show suboptimal performance when eval- uated different context quality. Through the experimental results, we also reveal FiD models trained with different context quality have different cross-attention distribution pat- terns. Specifically, as context quality during training increases, FiD models tend attend more uniformly to each passage in context. Fi- nally, based on these observations, we propose a method to mitigate overfitting specific con- text quality by introducing bias to the cross- attention distribution, which we demonstrate to be effective in improving the performance of FiD models on different context quality.
In order to address these challenges, retrieval- augmented generation models have been recently proposed (Izacard and Grave, 2021b; Lewis et al., 2020). These models draw inspiration from retrieval-based extractive open-domain question answering methods (Chen et al., 2017) and uti- lize additional relevant external knowledge (e.g., a Wikipedia article about an entity in a given ques- tion) during generation augment knowledge en- coded in language model. detrieval-auggentee generation models have demonstrated effectiveness in knowledge-intenn intensive tasks (Petroni al., 2021) such as question answering and fact checking (Hof- stätter et al., 2022), and have been reported to re- duce hallucinations in dialogue tasks (Shuster et al.. 2021).
The external knowledge given to the models is called a context, and is usually obtained through information retrieval systems (Lin et al., 2022). Multiple passages, typically up to 100, are of- ten used collectively a single context to sure the high recall of relevant information. This strategy addresses the limitations of retrieval sys- tems, which may return irrelevant passages and fail to capture relevant information in the top results. When dealing with contexts composed of multiple passages, we can define their quantity (the number of passages in the context) and quality (the pro- portion of relevant passages in the context). Since the context quantity and quality vary depending model configuration application, e.g., the performance of the retrieval system and the com- putational resources available, understanding how these characteristics impact the model performance becomes an important research question.
1 Introduction
Recently, large-scale pre-trained language mod- els have achieved impressive performance in the field of Natural Language Generation, which in- cludes tasks that require real-world knowledge, c.g., closed-book question answering and common sense reasoning (Brown al., 2020). However, these models are still prone to generate factually incorrect outputs known as hallucinations (Ji et al., 2023), particularly when dealing with rare entities (Mallen et al., 2023). Also, they cannot handle new
Indeed, during the inference phase, it has been shown that the quantity and quality of contexts im- pact the performance of retrieval -ageevaa-e-ggeeeedddd gen- eration models. For example, Izacard and Grave

---

(2021b) showed that increasing the number of top- ranked retrieved passages used as a context during inference improves the performance of their model in the question answering task, and Weller et al. (2022) found that the model prediction is distracted more strongly as the proportion of conflicting mis- information in the context increases.
training by controlling the selectivity of its cross- attention distribution. We present am empirical anal- ysis demonstrating the proposed method's s effective- ness in improving the performance of a trained FiD model when deployed in environments with differ- ent context quality than those used during training.
2 Experimental Setup
However, regarding the training phase, it is not yet fully understood how these context characteris- ties impact the performance of the trained models. Limited research suggests that increasing the num- ber of retrieved passages used as a context during training improves question answering performance (Izacard and Grave, 2021b) and reduces memo- rization (Chen et al., 2022). Still, the impact of quantity and quality of contexts is mixed in these studies, relevant passages are typically biased towards higher rank in the retrieval result, and sim- ply increasing the number of top-ranked passages changes both the quantity and quality of the con- text.
In this section, we describe the task (82.1) and model architecture ($2.2) used in our experiments, and we define quality and quantity that we used in this paper ($2.3, 2.4).
2.1 Task and Dataset
This study focuses on the extractive open-domain question answering task, where models have extract answers from retrieved documents. We con- ducted experiments on two standard benchmark datasets of the task:
Natural Questions (Kwiatkowski et al., 2019) contains questions submitted to Google Search engine. We use the open-domain ver- sion of this dataset presented by Lee et al. (2019).
In this paper, we focused on extractive open- domain question answering tasks and investigated the impact of context quantity and quality on the training of Ausionnin-Decoded (FiD) (Izac- ard and Grave, 2021b), state-of-the-ar retrieval- augmented generation model. We demonstrate that context quality during training affects the perfor- mance of the trained model. As far as our knowl- edge, this work is the first attempt to explicitly control context quality and investigate its effect on training of sttrieval-augmenne generation models. Key insights obtained through our experiments are follows:
TriviaQA (Joshi et al., 2017) contains ques- tions authored by trivia enthusiasts. Following Lee et al. (2019), we use the unfiltered set the dataset.
For each dataset, following (Izacard and Grave, 202 we used top-100 passages retrieved by DPR (Karpukhin et al., 2020)'. As an evalua- tion metric, we computed the exact match (EM) between ground-truth answer and predicted an- swer generated by greedy decoding?. We evaluated the performance the development set of each dataset.
FiD models overfit context quality during training, resulting in deteriorated performance when evaluated on a different quality of con- text.
2.2 Model
FiD models overfit less to context quantity compared to context quality.
In our experiments, we focused on Fusion-in- Decoder (FiD) (Izacard and Grave, 2021b), a state- of-the-art architecture for retrieval-augmmenter gen- eration model. FiD is extended from sequence-to to- sequence models, such T5 (Raffel et al., 2020), and consists of Transformer encoder E and de- coder D (Vaswani et al., 2017).
FiD models trained with different context qualities show different patterns of cross- attention probability. As context quality dur- ing training increases, the trained models tend to attend more uniformly to each passage in context and vice versa.
'The retrieved passages were obtained from the published dataset at https:/ thub. com/ faaebookreeearccchh... PI a question has multiple ground-truth answers, EM is computed as one if the predicted answer matches any one of the ground-truth answers.
Based on these observations, we propose method to mitigate the overfitting of a trained FiD model to specific context quality without additional

---

Given a question q and its context h= where p: is the i-th passage of the context, FiD model converts each passage Pj to pi by template
we considered passage that did not contain any ground-truth answers to the question as irrelevant. We respectively denote the set of relevant and irrelevant passages of question by R(g) and R(q), and we omit (q) when it is not necessary.
"question: {q} title: [t} context: {c}". Here, (q), {t), and {c} are respectively replaced by 9. the title of Pt. and the main text of p;. Then, FiD model independently encodes each converted passage pi by the encoder E and feeds their con- catenation to the decoder D to get predicted answer a as follows:
2.4 Context Quality and Quantity
For a question q and a context {pr) of N passages, we define context quality and quantity as follows:
Context quality is defined as the proportion of passages in e that is relevant to q. i.e. [R(gl
(1)
N
We followed standard practice and trained FiD models by minimizing a cross-entropy loss of a ground-truth answer. As the position of passage is not considered while encoding, the prediction of a FiD model is insensitive to the order of the passages. Thus did not perform any special shuffling ordering of the passages during training and evaluation. We used t5-base² (Raffel al., 2020) to initialize the model. See Appendix A for other implementation details of training and inference of FiD models.
Context quantity the number of passages c, i.e. N.
3 Case Studies
In this section, we describe our experiments to investigate how context quantity and quality dur- ing training affect the performance of FiD mod- els. Throughout the experiments in this section, created various training and evaluation environ- ments with controlled context quantity or quality by sampling n* relevant passages from R(g) and n irrelevant passages from R(4) for each ques- tion q, without replacement'. In the rest this paper, we define n n+ + n as the number of total passages and k = # as the ratio of irrelevant passages to relevant ones. We will use subscripts "train and "eval respectively to denote the values of training and evaluation environments if required.
2.3 Relevant and Irrelevant Passage
In this paper, we adopt the same definition of rel- evant and irrelevant passage as in Li et al. (2022). More specifically, a passage is relevant to a ques- tion if it logically entails an answer to the question, and it is irrelevant if it does not.
However, in our open-domain setting, ground- truth annotation of passage relevance exists for Nat- ural Questions and TriviaQA, As discussed by Li et al. (2022), a simple rule that determines pas- sage that contains a ground-truit truth answer as a relc- vant passage is insufficient to filter out irrelevant passages that contain the answer but do not entail the answer. Since accurately determining whether a passage is relevant or not is crucial for estimat- ing context quality, we applied an additional rule extract relevant passages. We fed pair of question and a passage to pre-trained question an- swering model and deemed the passage relevant if the predicted answer matched a ground-truth an- swer to the question Following Li et al. (2022),
3.1 Effeet of Context Quality during Training
To investigate the effect of context quality during model training, we created training and eval- uation environments with the same context quan- tity but different context qualities. More specifi- cally, for each total number of passages (i.e., con- text quantity) n in {10, 25, 60). we varied the value of n among {1, 21,2,33 3} (Natural Question) or [1, 2, 3, 5, 10} (TriviaQA) to obtain environments with different context qualities.
Setting:
Result: Figure 1 shows the performance of mod- els with different training context qualities?. We can obtain the following observations from the fig- ure: (i) For given evaluation context quality, mod- els trained with similar context quality showed the
Pht http:///hgggggggge ace. co/t5- base
'To annotate passage a question q in dataset D, we used FiD models trained a subset of that did not contain q. See Appendix C for more details.
"See Appendix for more details of the experimental design.
*We discarded those passages that contained a ground-truth answer but did not pass the additional filter, and we did not use those passage in our experiments.
"See Figure 6,7 and Table 7,8 in Appendix D for full results.

---

raa i, Roxal
1.0
Exact match (EM)
0.8
Ptznin
Aezoin 25
0.6
60
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Context quality,
TTaaa during model training
Merain
Figure I: Performance FiD models on Natural Questions with varying training context quality. Panels represent different evaluation environments with different (M (evar) pairs, and a red dashed line shows the context quality of the corresponding evaluation environment. Red stars represent the best-performed models in the corresponding evaluation environments. Dotted lines show models trained on the same context quantity Tigain-
highest performance. (ii) There was trend of monotonically decreasing performance as training context quality deviated further from evaluation context quality. (iii) Difference of context quantity had negligible impact in the above trends.
fied in such cases due to limited context quantities. Nevertheless, experiments did not reveal a con- sistent trend in performance changes due to varying context quantity. Hence, the results indicate that context quantity's s influence is relatively insignifi- cant compared to context quality's.
Insight: FiD models overfit to context quality during training, and suboptimal performance can be obtained when evaluated on different context qualities.
Insight: Training context quantity has less influ- ence model performance compared to context quality.
3.2 Effect of Context Quantity during Training
3.3 Effect of Mixed Context Quality during Training
Setting: To investigate the effect of context quan- tity during model training, we created training and evaluation environments with the same text quality but different context quantities. More specifically, for each ratio k among {1,5, 20}, we varied the value n among [1,2,3} (Nat- ural Questions) or 2, 3,5, 5, 8, 10} (TriviaQA) to change context q
Generally, in practical uncontrolled settings, train- ing dataset for FiD models may consist of ques- tions with different context qualities. Thus, we conducted experiments with mixed context quali- ties to investigate whether a similar overfitting phe- nomenon occurs for a training dataset with multiple context qualities.
Setting: We created three environments for each dataset. Specifically, context quantity was set 10, and n was varied among {1, 2, for Natural Questions, and context quantity was set to n = 25, and n was varied among {2, 5, 10} for TriviaQA. Then, we uniformly mixed each subset of the three environments and trained FiD models in each of them 10, Performance in mixed environ- ment computed by averaging the performance in each constituting environment.
Result: Figure 2 shows the performance of mod- cls with different training context quantities", As can be seen in the figure, the influence of context quantity on model training was generally less sig- nificant than that context quality 153. How- ever, we observed a more significant influence for smaller Kirain (higher context quality) in compar- ison to larger Ktaii (lower context quality), espe- cially in the cases where the training context quan- tity was small. One possible explanation for this behavior is that the impact of noise in the anno- tation of relevant (or irrelevant) passages disturbs the actual context quality, and this effect magni-
Result: Table shows model performance for each pair training and evaluation environment Natural Questions High scores at diagonal ele- ments of the table show that the models performed best or as well the best model when they were
"Since larget questions were only guaranteed have least 64 irrelevant passages, we did not conduct experiments in environments with kns 64.
l'We mixed environments by randomly selectig value of nt and sampling passages accordingly for each question training step. "See Table for the result in TriviaQA.
"See Figure 8, 9 and Table 9, 10 in Appendix D for full results.

---

rea
1.0
Excct match (EM)
0.8
0.6
"K" Risoin
Ryoln
XX
0.4
0.2
20
40
60
20
40 60
10
60
Context quantity, n² (Rarsin + 1), during model training
Figure 2: Performance of FiD models on TriviaQA with varying training context quantity. Panels represent different evaluation environments with different (nean Reval) pairs, and red dashed line shows the context quantity of the corresponding evaluation environment. Dotted lines show models trained on the same context quality T=
Table 1:
Performance of FiD models trained in each mixture of environments on Natural Questions. The color intensity indicates the relative performance within the same evaluation mixture of environments (column). Checkmarks indicate which environment included each mixture.
of cross-attention varied among FiD models trained on different context qualities 1$3.4.1). Then, we conducted intervention experiments to validate that different patterns of cross-attention explain part of the overfitting context quality ($3.4.2).
3.4.1
Investigation on Patterns of Cross-attention Probability
nt oval +) smm
2
3
2
3
Setting: We denote cross- atsssttttntion probability from the 1-th decoder token to the j-th token of the i-th imput passage ツ. at the k-th decoder layer (M)
69.8
82.9
88.6
85.7
79.2
76.4
80.4
65.0
92.2
78.6
75.3
93.0
62.3
85.2
92.5
88.8
77.4
73.7
80.0
by Gil Following (Izacard and Grave, 2021a), we
69.1
84.0
90.0
87.0
79.5
76.5
81.0
69.7
83.9
89.6
86.8
79.6
76.8
81.1
computed cross-attention probability from the first decoder token, Ciyi and computed aggregated (k) cross-attention probability e(h)) ΣS Cyy for each passage ps.
68.3
84.6
90.7
87.6
79.5
76.4
81.2
(k)
evaluated in the same mixture of environments as one in training. For example, the models trained in the uniform mixture all environments performed best only when they were evaluated on the same mixture. It suggests that covering all context qual- ities during evaluation is insufficient for optimal performance, and the distribution of the context qualities also matters to the performance.
We conducted the following two analyses:
(i) We analyzed how much cross-attention prob- ability allocated relevant passages at each layer, i.e., EiEiippeR)
(ii) We analyzed the difference between the dis- tribution of cross-attention probability to rel- evant passages, i.e., (a(²) li E R), and that irrelevant passages, i.e., (o (1) R).
Insight: FiD models overfit the distribution of context qualities during training.
3.4 Effeet of Context Quality during Training Model's Cross-attention
We focused our analyses on FiD models trained for Natural Questions in $3.1 with the follow- ing settings: (n, n++++= E {10, 1), (10, 2), (10, 3)}. Note that these models were trained with the same context quantity but different qualities. We ana- lyzed these models in two evaluation environments of Natural Questions with the following settings: (n+,, n ) {(3, 7), (3, 57))-
As we discussed in $3.1, FiDs trained on differ- ent context qualities may overfit to each quality, and they perform differently in the same evaluation environment. We hypothesize that overfitting to different context quality occurs due to changes in how model selects relevant passages since lower context quality may force the model concen- trate more on selecting passages and vice versa. Thus, as first step to investigate which aspect of model inference affected by different context qualities during training, we analyzed how patterns
Result: Table 2 shows the cross-attention prob- ability that was allocated relevant passages each layer (Analysis (i)). As shown in the table, both evaluation environments, FiD models trained

---

yelevanit passuges
irrelevant poussagts
relevaat paesages
irrolevani passiges
20
100
20
cab (Low)
75
duaiity
10
o o'lcain (High
0
0
0.1
cooss atteuioo probalilit
0.0
passagge
0.00
0.02
envitomment (#T..n [)) (3.7) (Digh)
Drlustion conponmers (6T.8) - (3,57) (Low)
Figure 3: Distribution of cross-attention probability to each relevant or irrelevant passage at Layer 9. A similar trend can be seen in other higher layers. Red vertical dashed lines represent uniform cross-attention probability, i.e., y if context quantity is N.
Table 2: Cross- Cross-attention probability allocated to relevant passages at each layer. High and Low respectively represent high and low context quality.
explain their relatively lower performance a high- quality environment ($3.2), because such excessive selectivity makes the model overlook necessary in- formation in ignored relevant passages and, as a result, fail to correctly answer the questions. It may be the case that, when evaluated in a high-quality environment (i.e., where the majority of passages relevant to the question), it is more optimal for the model to examine all passages more uniformly without being overly selective. This claim is empir- ically supported by the result our experiments in $4.2.
Evaluation environment .nn
7)
(3,57)
High
Low
Training
(10.
2
2
x2*
High
Low
High
Low
Layer
31.5
31.9
31.0
5.4
5.5
5.3
Layer 2
31.8
32.3
32.1
3.4
5.6
5.6
Layer 3
33.7
34.5
33.6
6.1
6.3
6.0
Layer 4
32.5
33.4
32.3
5.7
6.0
5.7
Layer
32.4
33.7
33.3
6.0
6.0
Tayer
35.4
36.2
6.6
Layer 7
45.6
47.4
8.5
10.7
13.7
Insight: FiD models trained with different con- text quality show different levels of selectivity w.r.t. allocation of cross-attention probability. Models trained with lower context quality attend more se- lectively to relevant passages.
Layer 8
40.8
44.1
45.8
8.6
10.3
13.2
L 9
47.4
51.9
56.3
11.3
15.1
21.8
Layer 10
48.2
52.9
55.5
10.8
13.9
20.4
Layer II
45.6
49.8
53.4
10.1
12.6
19.0
Layer 12
39.1
41.2
42.9
9.5
11.4
15.7
with lower context quality attended more strongly to relevant passages, especially at higher layers that are closer to the output layer!?
3.4.2 Intervention Experiment
Results in $3.4. suggests that overfitting of FiD models to different context quality due to dif- ferent level selectivity of their cross- attention. We validate this claim by intervening on the cross- attention probability of these models during infer- ence.
A similar trend is also observed in Figure 3 that shows the distribution of cross-attention probability to a relevant or irrelevant passage for each model (Analysis (ii)). Models trained with lower context quality showed more long-tailed distribution for relevant passages, and there was more significant difference between the distribution for relevant and irrelevant passages, which suggests they trained attend more selectively to relevant passages. On the contrary, the distribution is relatively closer to uniform distribution for models trained with higher context quality.
Setting: We intervened on the cross-attention probability of FiD models so that the ratio of cross- attention probability to a relevant passage pi E R and an irrelevant passage E R to be r for all layers. Intuitively, the model completely ignores irrelevant passages when O, whereas the model attends uniformly to all passages when = 1. More specifically, for each decoder layer k, we
We conjecture that this excessive selectivity of models trained in a low-quality environment may
converted original cross-attention probability Gjt
(k)
into intervened version di (k) as follows:
'he result may also suggest that a function to select rele- vant passages is leamed at higher layers, which are consistent with the claims made hy Tenney at al. (2019) that higher layers capture task-specific information.
(k) y
((k) 1/1
w
(k)
(2)

---

Evaluation enviromment (n*,,) .n)
levels of selectivity to relevant passages, we propose to change sharpness of distribution of cross-attention probability during inference. More specifically, we introduce temperature parameter T (T > 0) and compute total cross-attention probabil- ity from the 1-th decoder token to the i-th passage at the k-th layer as follows:
(3. 7)
(3, 57)
1.0
0.95
*
0.8
Excett match (EM)
x
0.90
0.6
X
NI train
0.85
train
(k) "
softmax Too ond
1oR(2,
0.4
0.80
0.2
Then, use Equation (2) to convert cross- attention probability as $3.4.213 Intuitively, the model attends more uniformly as T becomes larger, which simulates the overfitting effect of a FiD model trained with higher context quality and vice versa.
0.75
No
1:2
1:0.1
1.0
NV
1:2
1:0.1
1.0
Ratio jutervened cros-atteattoon probability to relevant and irrelevent passage, 1 :
Note that proposed temperature parameter does not change the set of input passages and can be tuned complementary with other existing hyper- parameter that changes the set of input passages, e.g., the number of input passages.
Figure 4: Model performance under intervention on cross-attention probability. "No" represents setting without intervention.
(N)
where Wi
Irin pf E R and wi
nt Irn if pj E R. We selected from {1, 0.1,0) and conducted experiments the same models and evaluation environments as in $3.4.1.
4.2 Experiment
To validate the effectiveness of our proposed method, we adapted models trained in $3.1 by the proposed method and evaluated their perfor- mance on evaluation environments with different context qualities where neval = 3 and neval E [10, 25, 60). Since the temperature parameter T has to be tuned, we conducted 2-fold cross- validation. Specifically, we split the evaluation data into two folds and scarched optimal temperature parameter T E (0.125, 0.25, 0.5, 1, 2, 1, 1,8} based on the EM score on one fold, and then, we used a model adapted with 7 to evaluate performance on the other f4
Result: Figure 4 shows model performance with atteettottnno proba- bility. In both evaluation environments with lower and higher context quality, the difference in the performance of the models decreased, and the inter- vention mitigated the effect overfitting to context quality.
and without intervention
Insight: The result suggests that the difference of cross cross-attention probability as described in $3.4.1 is element that explains the overfitting of FiD models context quality.
Figure shows the performance of FiD mod- els with and without adaptation by the proposed me As shown in the figure, the proposed method improved the performance of the models in environments with different context qualities com- pared to those during training, and it reduced the effect of overfitting to context quality. Also, T
4 Adapting Models to Different Context Quality
While FiD models overfit to context quality during training shown in $3, it is not desirable to train dedicated model for each target environment that has potentially different context qualities from each other. Thus, in this section, we propose a method to mitigate the effect of overfitting and adapt an already trained FiD model to an environment with different context quality.
We use w(th) instead of with in Equation (2) when decod- ing the 1-th token. Note that the cross-attention prohability is sequentially computed for each decoder token during infer- ence.
We used single temperature parameter 7 for ques- tions a instead of using a different temperature param- eter each question. Predicting optimal temperature parameter for each input question an interesting direction future works.
4.1 Proposed Method
Based on the insights in $3.4 that shows the over- fitting to context quality occurs due to the differ-
see Figure 10 in Appendix D for the result in TriviaQA.

---

3, Revol
10
Reval
3, Reval
25
Neval
3, Pevval
60
0.9
(RM)
0188
0.7
0.6
Parzin
Purio
0.5
Puráa
60
Optimal temperator 7
8
+**
2
0.5
X
0.125
0.0
0.1
0.3
0.1
0.2
0.0
0.2 0.3
quality, (Cnaaee during model training
Figure 5: Top panels: Performance of FiD models on Natural Questions with adaptation by the proposed method (solid lines) and without adaptation (dotted lines). Bottom panels: Optimal temperature parameter selected for each model. Multiple T were selected for context qualities, i..., training environments, because selected " for each of the three models trained with different random seeds for each training environment. Panels represent different evaluation environments with different (n Reval) pairs, and a red dashed line shows the context quality of the corresponding evaluation environment.
increased the case of lower context qualities dur- ing training, and vice versa, which corroborates our finding that more uniform cross-attention, cor- responding to higher I* is effective when context quality in evaluation is higher than one in training.
2021). The approach has also been shown effec- tive in improving inference of pre-trained language models (e.g., GPT-3 (Brown et al., 2020)) without additional training (Lazaridou et al., 2022; Mallen et al., 2023). For a comprehensive survey on this evolving field, refer to (Yu et al., 2022).
5 Related Works
Effect of Context Characteristics Retrieval- augmented Models: Several studies have investi- gated how context characteristics affect inference of retrieval- augmentd generation models. For ex- ample, increasing the number of top-ranking pas- sages used as the context has been found to im- prove performance in question answering (Izacard and Grave, 2021b) and response/prose generation (Zhang al., 2022b), while higher proportion of false information in the context degrades perfor- mance in question answering (Weller et al., 1022)16 Liu et al. (2023) found that the performance language models on multi-document question an- swering is influenced by the position of a relevant document.
Retrieval-augmented Generation Models: Lewis al. (2020) introduced retrieval augmentation ap- proach, firstly originating in the field of extrac- tive open-domain question answering (Chen et al., 2017), to sequence-to-sequeee dated its effectiveness in knowledge-intensive tasks. Contemporary work by Min et al. (2020) applied the retrieval augmentation approach to the task of ambiguous question answering. Izacard and Grave (2021b) proposed Fusion-in-Decuee --Decoder (FiD), which each retrieved passage is independently encoded and then jointly input to the decoder. FiD achieves high scalability for the number of pas- sages and effectively aggregates information by jointly using all passages in the decoder.
models and vali-
However, limited knowledge is available regard- ing the impact of context characteristics the training of retrieval-augmennte geneartion mod- els. Notably, few existing research suggest that the model performance in question answering im-
Recently, the retrieval-augee augmented language gen- eration approach has received attention, including its incorporation into the language model pretrain- ing (Borgeaud et al., 2022; Izacard et al., 2022; Zhang et al., 2022b), dialogue generation (Komeili et al., 2022), and code generation (Parvez et al.,
Du et al. (2022) reported similar results in fact verification, while their focus is not on language generation models.

---

proves by providing more top-ranking passages during training (Izacard and Grave, 2021b) or by randomly masking top-ranking passages during training (Zhang et al., 2022a), and that the model's memorization behavior is reduced by increasing recall of relevant information in the context during training (Longpre et al., 2021; Chen et al., 2022).
Model Architecture: Secondly, our analysis only largeted FiD models, and is unclear whether different architectures such as RAG (Lewis et al., 2020) and Internet-augmenned language models (Lazaridou et al., 2022) produce simi- lar results. Also, it is an interesting direction of future work to conduct similar investiga- tions non-generative setrieval-augmente models such as FiE. (Kedia et al., 2022).
6 Conclusion
this paper, we investigate how context quality and quantity affect the training of FiD models in ex- tractive open-domain question answering tasks. We show that FiD models tend to overfit to the context quality during training, resulting in degraded per- formance when evaluated in environments with dif- ferent context qualities. Additionally, our research reveals that the overfitting to context quality is par- tially explained by different patterns in the model's cross-attention probability. Based on these obser- vations, we propose changing the selectivity of the cross-attention probability to mitigate the effect of overfitting to context quality. The results of this paper suggest broad spectrum of future work, in- cluding more sophisticated adaptation methods and investigations of the effect of other context char- acteristics on the training of Attreval-augmeen generation models.
Model Size:
Thirdly, our experiments are focused on only t5-base, and it is unclear how scaling model size changes the behavior of overfitting context quality.
Characteristic of Context: Lastly, coverage of our analysis is limited to quality and quan- tity, and further research is required to investi- gate the effect of other context characteristics. For example, in the field of extractive ques- tion answering. it has been shown that models may overfit to answer positions the con- texts (Ko et al., 2020), be misled by adversar- ially inserted sentence (Jia and Liang, 2017: Jiang and Bansal, 2019), and be susceptible to whether answer is in the most similar sen- tence in the context (Sugawara al., 2018). These findings suggest that those context char- acteristics may also affect estrieval-auggeent generation models.
7 Limitations
Other than that, our experiments involved the automatic annotation of relevant and irrelevant pas- sages, which may limit the accuracy of our analysis. Future studies should incorporate human annota- tion to ensure the high quality of the annotation.
In this study, we investigated how the quality and quantity of context affect the training of FiD mod- els in extractive ppen-nnmminn question answering tasks. Our experiments revealed for the first time that context quality significantly impacts FiD mod- els' training and that FiD models tend to overfit to context quality of the training data. The impli- cations of our findings suggest that various con- text characteristics similarly affect the training of retrieval-augnee -uggeentd generation models, potentially leading to issues such as overfitting.
Also, passages with similar relevant information can impact models differently due to qualitative factors such as readability and writing style. Nev- ertheless, as quantitatively evaluating these factors poses challenges, study did not conduct fine- grained analysis regarding these aspects.
However, our experiments have several limita- tions that reduce the generalizability of our find- ings:
References
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann. Trevor Cai, Eliza Rutherford, Katie Milli- can. George Bm Van Den Driessche, Sean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference ma- chine learning, pages 2206-2240. PMLR.
Task:
Firstly, in this paper, only focused on the extractive open-domain question answering task, and is unclear whether similar results can be obtained in other tasks such as dialogue generation, fact verification, code generation, and summarization.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Yichen Jiang and Mohit Bansal. 2019. Avoiding reason- ing shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceed- ings the 57th Anmual Meeting of the Association for Computational Linguistics, pages 2726-2736, Flo- rence, Italy. Association for Computational Linguis- tics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 1870- 1879. Vancouver, Canada. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver. Canada. Association for Computational Linguistics.
Hung-Ting Miig Chen, Michael Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowl- edge conflicts: Recalibrating models to reflect con- flicting evidence. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 2292-2307, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Procexsing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Yibing Du. Antoine Bosselut, and Christopher Man- ning. 2022. Synthetic disinformation attacks on au- tomated fact verification systems. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pages 10581-10589.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What's the answer right now? arXiv preprint 2337.3333.
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2022. Fid-light: Efficient and ef- fective
Akhil Kedia, Mohd Abbas Zaidi, and Haejun Lee. 2022. FiE: Building a global probability space by leverag- ing early fusion encoder for open-domain question answering. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Process- ing, pages 4246-4260. Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
text generation. arXiv preprint 2rXiv:2209. 14290.
Gautier Izacard and Edouard Grave. 2021a. Distilling knowledge from reader to retriever for question an- swering. In ICLR 202 202h-9th International Conference on Learning Representations.
Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. 2020. Look at the first sentence: Position bias in question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 1109-11211 Online. Association for Computa- tional Linguisties.
Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmente dialogue generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pages 8460-8478, Dublin, Ireland. Association for Computational Linguistics.
Gautier Izacard, Patrick Lewis. Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Scbastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arX1v:200..33299.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wai Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Cumpu- tational Linguistics, 7:452-466.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishi,, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation, ACM Comput- ing Surveys, 55(12):1-38 38..
Robin Jia and Perey Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot

---

prompting for open-domain question answering, arXiv preprint arX3::2003.05/15
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedinggs of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguisties.
Kenton Lee, Ming-Wei Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguisties. pages 6086-666 6096, Florence. Italy. Association for Computational Linguistics.
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensiin ulp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9495
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21( (1())5485-5551
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik. Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory, arXiv preprint arXiv:2211.05110
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2022. Pretrained transformers for text ranking: Bert and beyond. Springer Nature.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading compre- hension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2208-4219, Brussels, Belgium. Association for Computational Linguistics.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How lan- guage models use long contexts. arXiv preprint arXiv:230.00312..
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois. and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7052-7063, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das. and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non- -onpaaameitiic mem- ories. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages 9802-9822 Toronto, Canada. Association for Computational Linguistics.
Orion Weller, Aleem Khan, Nathaniel Weir, Dawn Lawrie, and Benjamin Van Durme. 2022. Defending against poisoning attacks in open-domain question answering. arXiv preprint 20200:2212.10002.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: Statc-of-the-at natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2719-2734, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A

---

survey of nrowledge-enhaneee text generation. ACM Computing Surveys, 54(11s):1-38
Table 3: Size of datasets used for training and evalua- tion.
Shujian Zhang, Chengyue Gong, and Xingchao Liu. 2022a. Passage-mass mask: A learnable regularization strategy for retriever-reader models. Proceedings of the 2022 Conference on Empirical Methods in Nat-
Natural Questions
TriviaQA
Dwain 20728
Daev
Deval
Duain
Daev
Deval 1434
3048
2589
11414
1695
ural Language Processing, pages 3931-3943, Abu Dhahi, United Arab Emirates. Association for Com- putational Linguistics.
Table Hyperparameters for Seq2SeqTrainer
parameter learning rale
value
Yizhe Zhang, Siqi Sun, Xiang Gao, Yuwei Fang, Chris Brockett. Michel Galley, Jianfeng Gao, and Bill Dolan. 2022b. Retgen: joint framework for re- trieval and grounded text generation modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11739-11747.
0.00005
Ir se_sccceduer__yy type
constamL_ wwith, _ warmup
warmup_stee steps weight_decay max max__rad norm max_stepa
1000
0.01
15000
per_ deviceeeeeee_. _rriinnrinnnnna batch size
A Implementation Details
gradient accumulation
eval_steps save_steps
500
A.1 Training and Evaluation Data
500
save_strategy
steps
We trained FiD models with the original train set of each dataset, and further split the original train set into Drrai for training and dev for evalu- ating performance during training. To train models in a strictly extractive task environment, we ex- cluded questions for which no retrieved passage contained any of their ground-truth answers. For evaluation, we used the original development set of each dataset as evaluation data Deval- For a fair comparison, we used the same set of questions with at least 3 (Natural Questions) or 10 (Trivi- aQA) relevant passages and at least 64 irrelevant passages during training or evaluation. Statistics of the datasets are shown in Table 3.
hand, since questions in TriviaQA are more exhaus- tively annotated with paraphrases of ground-truth answers, randomly sampled one ground-truth answer that appeared in any of the input passages as a target output at every training step.
We tokenized each input passage p. described $2.2 and target outputs by the tokenizer of t5- tase. For both training and inference, to fix the sequence length of each tokenized passage to 256, we con- ducted truncation for longer passages padding for shorter passages. We did not truncate target outputs during training and set a maximum length of a predicted answer to 50 during inference.
A.2 Details of FiD Training and Inference
In experiments where we subsampled passages to control context quality and quantity, to reduce the effect of bias in sampled passages, we sampled different passages at every training step instead of repeatedly using the fixed set of passages sampled before training.
Our model implementation of FiD, including the loss function for training, is based on the official implementation by Izacard and Grave (2021b)"7 For both training and inference, we used transformers (Ver. 4.23.1) (Wolf et al., 2020). We trained the models with Seq2SeqTrainer pro- vided transformers. Hyperparameters for Soq2SeqTrainent used our experiments are listed in Table 4 and other hyperparameters were set to default values. Since we trained models with 32 A 100 GPUs, the effective batch size is 64, We used model checkpoint with highest EM on Daey for downstream evaluations.
B Details of Experimental Design
We trained three FiD models with different random seeds for each training environment and conducted evaluations for these models in each evaluation environment. We sampled five different sets of passages for each evaluation environment and com- puted average model performance these sets of passages. We independently sampled relevant and irrelevant passages, and thus, the same of relevant (or irrelevant) passages was sampled re- gardless of the number of irrelevant (or relevant) passages as long as the number of relevant (or irrel-
Since most questions in Natural Questions are annotated with only one ground-truth answer, we used the first ground-trutt truth answer for each question as a target output for model training. On the other
hhtttps://ithhu.... http://ttwt.ccc/aaaceoooreeeeeee

---

Table 5: Performance FiD models trained each mix- ture of environments TriviaQA. The color intensity indicates the relative performance within the same eval- uation mixture of environments (column). Checkmarks indicate which environment included in each mixture.
relevant passage in the both settings.
D Full Experimental Results
Full results of the experiments in $3.1 are shown in Figure 6 and Table 7 for TriviaQA and Figure 7 and Table 8 for Natural Questions.
12 ceal
5
10
Full results of the experiments in $3.2 are shown
2
10
in Figure 8 and Table for TriviaQA and Figure 9
79.8
93.9
98.3
96.1
89.0
86.8
90.6
96.0
89.0
and Table 10 for Natural Questions.
75.8
83.1
Results of the experiments in $3.3 are shown Table for TriviaQA.
73.4
92.4
97.7
95.1
85.6
82.9
87.9
79.1
93.3
97.5
95.4
88.3
86.2
90.0
79.0
94.2
98.4
96.3
88.7
86.6
90.5
Results of the experiments in $4.2 are shown in Figure 10.
Table 6: Size of datasets used to train FiD models for the relevant passage annotation
Do,wain
Do,dev
Diusin
Di,dev 4373
30612
4411
30589
(Natural Questions)
28696
4112 28489
4115
(TriviaQA)
evant) passages was the same.
C Details of Passage Relevance
Annotation by Question Answering Model
We used FiD models pre-trained question an- swering models used in the passage relevance an- notation deseribed in $2.3, and trained those models described in Appendix A except training and development data which we describe below. For each dataset with original train set Dtrain and development set Dev. we split Dirii into four sets: Do,uain Do,dev, L.rainn- and Di,do Then, we re- spectively trained FiD model Mo or Mi with pair of train and development data (Do,train: Do,dev) (D 1,rrii D 1e.dev). Finally, annotated Do,trinn and Do,dev with M1, and D,,miin Di,dev. and Deev with Mg- See Table for statisties of the datasets used train FiD models for the relevant passage annotation.
Our preliminary experiments showed that the be- havior of FiD models differs when trained with all passages in the original dataset (AII), com- pared to when trained with only those passages containing ground-truth answer (Pos). Thus, chose stricter criterion extract relevant pas- sages. Specifically, we trained Mo and Mi and annotated passages in each of All and Pos setting, and we extracted only those passages annotated as

---

Neeal
Reeal 60
1.0
Plenii
0.8
25
0.6
_X... ................................
Vitcoin 60
0.4
0.2
2, Parral
2. Naval 25
11 aval Nocal 60
1.0
0.8
x
Meric """"""""..... Y.....
0.6
0.4
0.2
3, Ravel
8. rvat 3. Reeal
11 ovsl Recal
1.0
Exact match (EM)
x
0.8
Morre x
Aruth
0.6
0.4
0.2
mal 5 Paril
maa Tannl
maa Naxal
1.0
1-00
%
0.8
X..
0.6
0.4
0.2
evs) Hoot
mvcl
1.0
0.8
0.6
0.1
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Context quality, nttain during model training
Figure Performance of models on TriviaQA with varying training context quality. Panels represent different evaluation environments with different (n Reaal) pairs, and red dashed line shows corresponding context quality. Red stars represent the best performed models in the corresponding evaluation environments. Dotted lines show models trained on the same context quantity Prrain

---

Table 7: Exact match (EM) [%)] of FiD models trained in environment (n inann Maran) in various evaluation environment (n eval, Reval) in TriviaQA. The standard deviations for each reported value is denoted with lower subscripts.
Roal
10
25
60
Pemin
n eval
10
25
60
10
25
60
10
25
60
tnin
I
70.9a.0
70.41.
70.1
58.5a.1
60.11.
61.95.1
45.73.5
49.8
53.9s.4
2
69.01.2
72.12.
72.41.1
52.31.9
60.8
64.12.0
34.41.
49.24
56.12.
64.3
70.70.8
71.41.1
43.6u.
57.41.
61.81.7
24.50.2
43.123
52.22.
5
52.2
66.90.9
70.32.0
29.11.3
51.01.
59.22.
15.20.5
34.124
47.23,3
10
18.0p.6
55.312
63.71.s
9.5p.2
36, 36.40.5
50.31.4
6.4p.
21.20.4
36.7a.8
87.91.4
86. 86.70.6
84.9p.2
77.92.2
77.91.3
77.9..7
63.93.3
66.61.6
69.9g.1
2
89.1
88.51.0
87.10.4
76.61.0
79.82.
80.41.1
56.22.0
68.1
72.72.1
2
3
86.70.3
88.70.
87.80.
71.30.2
78.61.
79.81.1
45.96.6
63.82.5
70.21.5
5
81.2g.5
87.00.2
87.30.7
56.91.2
75.0l..
78.61.7
29.61.1
56.122
66.72.8
10
39.6p.6
80.1o.7
83.0
17.8p.6
61.7p.7
71.71.4
9.5p.3
39.0p.9
57.01,3
I
93.7a.7
I2.
90.4p.4
86.31.6
85.4n
84.5u.s
73.92.
75.51.4
77.22.2
2
95.40.1
93.60.2
92.0o.1
86.90.s
87.51.
87.0p.6
09.71.
77.5a.0
80.21.4
3
94.30.1
94.70.2
92.60.1
83.6g.5
87.60.4
87.00.5
61.90.9
75.01.
79.21.3
5
92.3n.4
93.7o.s
13.1
74.60.7
85.40.1
86.40.9
44.71.1
69.61.
76.52.1
10
59.6n.4
90.80.2
90.8p.7
27.30.4
76.10.3
81.7a.
12.8o.1
53.6p.7
68.50.8
1
97.70.3
96.6o.1
95.11.0
93.3g.s
92.36.
91.1a.1
85.418
85.6p.7
85.91.1
2
98.5p.1
97.50.4
1..4
94.8n.4
93.90.3
92.6g.0
84.2p.
87.51.
88.50.4
5
3
97.8u.1
98. 98.00.1
96.6g.6
93. g.11
94.6p.4
93.
79.5au
86.8p.9
88.30.5
97.7p.2
67.5
97.1g.2
90.
93.8pu
93.8u.1
67.2 67.2ax
83.5p.3
87.2g.8
10
83.8p.2
97 Dn.3
96.3p.2
47 47.40.6
90.0p.a
91.77..2
21.96.s
72.4p.2
82.0p.9
1
99.2p.2
98.90.1
97.8p.9
98.3a.1
97.50.1
96.3g.s
94.6aus
93.6p.1
92.90.a
2
99.3g.1
99.0u.1
98.2p.1
98.9g.
98.30.4
97.0g.3
95.2g.
95.2p.o
94.4g,2
10
3
99.0p.
99 99.
98.6g.4
98.0pu2
98.5p.1
y7.5g.4
93.00u
95.6p.s
94.9u.
5
99.2n.2
99.00.3
98.9p.3
98.1g.2
98.3n.
97.9u.3
89.60.6
91.6p.7
95.2
10
98.8p.1
98.90.2
98.40.4
81.4g.4
97.7o.4
97.5g.2
141.3..
90.9p.4
93.50.5
Table 8: Exact match (EM) [%]] of FiD models trained in environment (m+tmain Rerain) in various evaluation environment (n evaly neval) Natural Questions. The standard deviations for each reported value is denoted with lower subscripts.
irain
Revll
10
25
60
n eval
Memin
10
25
60
10
25
60
10
25
60
tainn
I
69.8n.7
70.30.3
65.8p.3
18.
62.4o.7
61.0p.1
45.50.8
54.11a
55.90.1
2
65.0p.5
69. 60.6
68. 18.s
45.41.2
58 60.5
60.7a.s
24.11.6
45.7p
53.21.2
53.91.2
65. S0.2
67.8p.1
29.0g.4
50.7o.
58.7a.2
10.6g.2
33.32.1
49.2g.3
1
82.9n.4
80.30.a
74.01.4
73.7a.2
74.
70.8us
61.9u.s
67.0p.
66.5aa
2
2
85.5p.1
83.90.
79.9p.1
71.4u.
75.7o.
74.5g.4
48.21.4
64.3p.
67.8u.7
83.4g.1
84.2n.3
81.9g.7
60.9u.
73.9n.
75.5us
29.8u.1
57.61.3
66.60,7
1
88.6n.
81.40.s
77.416
80.8p.
79.2g.2
74.410
70.30.1
72.9p.4
70.80.
2
92.2p
89.5ous
84.6p.
82.4n.s
83.20.5
79.6g.1
63.51m
73.6p.5
74.4a.
3
93.0p.1
91.2p.
87.6g.
78.803
83.70.4
82. la.
48.0uz
70.5p.5
74.8us

---

Table 9: Exact match (EM) [%]] of FiD models trained in environment (n* train Krain) various evaluation environment (nt evals keval) in TriviaQA. The standard deviations for each reported value is denoted with lower subscripts.
Reval
20
Kerain
11
1
20
5
20
5
20
train
1
90.8p.2
90.11.4
88.6p.9
77.40.3
76.92.3
77.52.5
53.50.3
57.0g
63.4s.5
2
89.00.2
89.4o.
88.30.2
69.0o.1
77.60.4
17.
32.21.1
58.71.
63.4
88.0n.4
89.40.3
87.80.8
68.81.
77.10.8
77.
34.83.5
58.31.1
63.72.
5
85. 85.8p.3
87 6o.a
65.4n.
74.00.5
33.31a
54.718
8
84.01.1
86. lo.5
64.0n.
72.80.7
35.31.9
53.92.
10
83.6
84.31.
63.9p.6
70.7
35.30.7
52.71.2
1
95.4n.2
95. 1o.4
93.7
85.
85.81.2
85.71.4
61.0aa
66.44.4
73.0
2
95.6
95.20.5
93.9uu2
81.2p,
87.0nu
85.9u.4
38.01.2
68.41.0
73.9
3
95.0p.1
94. 64.6
93.
81.11.1
86.6p.
86.3g.7
41.53.4
68.4p.7
74.82.5
2
5
93.6p.1
93.9p.s
77.90.4
84.9p.
40.21.4
65.62.1
8
92.0p.7
92.80.3
75.41.
83.
42.1
64.61.9
10
91.5n.2
91.6o.a
74.6p.2
81.31.
41.80.4
63.31.5
1
96.9p.1
96. To.3
95.0p.3
88.2p,2
89.40.
88.99..
63.5q.6
69.8g.2
76.53.5
2
97.30.3
96.60.5
95.3p.2
81.90.2
90.6p.7
89.5g.3
39.91.2
71.3p.9
77.71.2
3
97.2p..
96.20.3
95.2p.1
85.71.3
90.00.5
89.60.6
43.95.5
71.30.4
78.52.1
3
5
96. 96.2p.2
95.9u.
82.9g.4
80.2p.2
43.01.0
69.31.8
8
95. 15.
95. 0o.4
80.91.0
87-1n.3
45.42.4
68.92.4
10
94.4p.1
93.7o.s
80.2g.3
86.0p.7
45.1a.
67. 67.61.0
97.7p.2
98. 98.0v.n
96.8p.4
90.0g.3
92.0p.4
91.6g.5
2
98.0n.1
97.9u.a
96.9u.s
87.8u.
93.2p.4
92.3u.1
3
98.
97.6g 67.6
96.9u.4
88.41.1
92.3p.2
92.4u.
5
97.7p.2
97. 97.7o.1
86.5p.3
92.5n.3
8
97.10.2
97.1o.2
8/191.1
90.80.3
10
96.8n.1
96.30.2
84.10.4
90.30.s
1
18.o
98. 7o.1
47.4
91.3u.
93.70.a
93.3g2
2
98.4a 4g.1
98.50.3
97.8p.3
89. 19.1
91.5n.3
94.2
3
98. 98.6p.3
98 98.20.2
97.6p.3
90.2
93.90.4
94.5p.2
5
98.4
98.40.a
88.8n.2
94.20.4
8
97.9u.3
98. 1u.3
87.7u.7
93.30.4
10
97.9p.o
97. 07.40..
87.0p.
92.6p.4
1
98.5p.
99.00.1
98.0p.6
91.9p.4
91.40.1
94.1
2
98.6p.2
98. 98.7o.2
98.
90.2g.2
94.90.3
95.0g.5
3
98.7
98 58.a
98.0p.3
90.91.1
94.6p.4
95.
5
98.7u.
98.7o.1
89.6u.
95.0pa
8
98.3p.2
98.3k.
88.7a.7
91.0n.3
10
98.1o.1
97.90.s
88.4g.2
93.5p.s
Table 10: Exact match (EM) [%] of FiD models trained in environment (n trainy Arrin) in various evaluation environment (n+ evals kevat) in Natural Questions. The standard deviations for each reported value is denoted with lower subscripts.
train,
Reval
5
20
Kimin
11
1
20
5
20
5
20
1
87.9u.
88
18.20..
84.3p.
72.4u.
76.0p.
74.7a.
48.2am
58. 58.7pp
62.91.4
2
85.
87.0 37.60.1
83.2ps
59.51.
73.96.4
73.99.7
20.713
53.2p
62.41.
82.4g.5
85.80.3
82.3p.7
56.0j.1
70.9p.9
72.8g.
20.41.
47.5.0
60.908
1
91.8u.5
91.
87. 87.6
80.0p.s
82, 82.3n4
80.2a.s
54.9h.1
65.7p.
69.3n.
2
93.8p.o
92. 82.40.2
87.71.0
75.21.0
83.5o.
80.6g.2
25.61.5
62.4p.
70.7o.8
3
93.1p.2
92.1o.2
87.4p.9
73.9p.
82.7o.
80.5p.
26.31.1
58.9p.s
70.7o.1
1
93.3n.
92.60.a
89.2p.3
82.6us
84.30.4
81.7a.
57.3mm
68. 68.2pm
71.5ax
2
96.4p.2
94. 04.6n.1
89.3p.s
80.0p.s
86.7n.
83.4g.4
27.51.4
65.7p.4
73.814
3
96.2p.3
94.8p.2
89.5g.7
79.8g.1
87.10.3
83.3g.1
28.4n.
63.5
74.404

---

1, Neval 33
n" enal Nieaa 60
1.0
0.8
X-- Ottain
Plerann 25
0.6
60
0.4
0.2
0.0
2, Naval 25
2, Tical 80
1.0
Exact match (EM)
0.8
0.6
0.1
0.0
3, Ri
3. Recal
Reaal BD
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Context quality,
ni during model training
Herain
Figure 7: Performance FiD models on Natural Questions with varying training context quality. Panels represent different evaluation environments with different (m Reval) pairs, and red dashed line shows corresponding context quality. Red stars represent the best performed models in the corresponding evaluation environments. Dotted lines show models trained on the same context quantity Tirain

---

1.0
26 23
0.8
x
0.6
0.4
x X
0.2
9 Feest
1" cost - Avot
2e mal Reval
1.0
200% 0
0.8
X-x xx
0.6
0.4
x XXx
0.2
1.0
5000 XX
0.8
0.6
X Xxx-XXXX XX
Exact match (EM)
0.4
0.2
eva Real
1.0
0.8
0.6
0.1
0.2
1.0
100 A
0.8
0.6
0.1
0.2
12 R0
10, Asnal
1.0
X
Recain
0.8
Aecoin
5
kirain 20
0.6
0.4
0.2
20
60
Context quantity, not train
TtaiinArrna (Ruain 1), during model training
Figure 8: Performance of FiD models on TriviaQA with varying training context quantity. Panels represent different evaluation environments with different (mtar keval) pairs, and red dashed line shows corresponding context quantity. Red represent the best performed models the corresponding evaluation environments. Dotted lines show models trained on the same context quality 1+hmm"

---

Nes
1.0
0.8
0.6
Eizain
0.4
Ripin
0.2
Kgain
20
msl 2.
n: val
2.
1.0
Exact match (EM)
0.8
0.6
0.4
X
0.2
S.
25 3, Raha
1.0
0.8
0.6
0.4
X
0.2
60
Context quantity, n² "tom trommmaaa (Ruam 1), during model training
Figure 9: Performance of FiD models Natural Questions with varying training context quantity. Panels represent different evaluation environments with different (n keval) pairs, and a red dashed line shows corresponding context quantity. Red stars represent the best performed models in the corresponding evaluation environments. Dotted lines show models trained on the same context quality T=
71 cval
3, Reval
10
71 eval
3, Plrva
25
71 evaal
3, Flevsl
60
0.8
match (EM)
0.9
0.8
0.6
0.1
0.7
Perzin
Puzzi
0.6
Pursio
60
0.2
Optimal temperatur 7
8
2
0.5
106
X
X
X
X
0.125
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Context quality, aTaanee during madel training
Figure 10: Top panels: Performance of FiD models on TriviaQA with adaptation by the proposed method (solid lines) and without adaptation (dotted lines). Bottom panels: Optimal temperature parameter 7 selected for each model. Multiple T were selected for some context qualities, i.e., training environments, because selected T for each of the three models trained with different random seeds for each training environment.
Panels represent different evaluation environments with different (n+ + Reval) pairs, and red dashed line shows corresponding context quality.


Reviews:
[{"review_id": "B4M2Li2drP", "reviewer": "Reviewer_mQfV", "summary": "", "questions": "A. How do you claim the proposed hyper-parameter, i.e., temperature is better than the others, e.g., number of documents, or different sets of documents selected by heuristics like estimated relevant scores?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Questions For The Authors:\nA. How do you claim the proposed hyper-parameter, i.e., temperature is better than the others, e.g., number of documents, or different sets of documents selected by heuristics like estimated relevant scores?\n\nThe proposed temperature parameter has distinct characteristics from existing hyperparameters and works complementary rather than competitively with the existing hyperparameters.\n\nExisting hyperparameters, such as the number of documents and heuristical selection based on relevance scores, affect model performance by chainging the set of input passages itself. Since a trade-off exists when selecting input passages (e.g., between recall and quality), tuning these hyperparameters is important.\n\nNevertheless, relying solely on these hyperparameters has its drawbacks. As shown in our paper, not only context itself but also a quality gap between training and inference time influences the performance of FiD. A larger quality gap can lead to performance degradation. Therefore, unless reducing the quality gap via a costly retraining of FiD, there is a risk of underestimation of the model performance on a specific hyperparameter setting that leads to a larger quality gap, leading to suboptimally tuned hyperparameters.\n\nContrary to the existing hyperparameters, our proposed temperature parameter does not change input contexts to reduce the effect of the quality gap. Therefore, it can not only improve the performance on a given fixed context during inference but also reduce the aforementioned underestimation effect due to the quality gap without any retraining.\n\nIn conclusion, the temperature parameter works complementarily to the existing hyperparameters that adjust context itself, thereby improving their tuning performance.\n\n> It would be more significant if the paper could propose an approach to automatically estimate this temperature without tuning it by cross-validation.\n\nWe acknowledge the importance of automatically adjusting the temperature parameter, as you pointed out, and consider it a crucial direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provide insights of the effects of context quantity and quality of a retrieval augmented model. The goal of this study is well presented and the experiment designs are well aligned with the goal. I like the layout of the experiments. The analysis of cross-attention probability is interesting and comprehensive.", "reasons_to_reject": "The novelty of the proposed method, i.e., introducing a hyper-parameter temperature, could be limited. The reason is there are existing hyper-parameters that we can tune when choosing supporting documents, such as number of documents, different sets of documents selected by heuristics like estimated relevant scores. So far, I did not see why temperature is a more advanced hyper-parameter. It would be more significant if the paper could propose an approach to automatically estimate this temperature without tuning it by cross-validation.", "questions_to_authors": "A. How do you claim the proposed hyper-parameter, i.e., temperature is better than the others, e.g., number of documents, or different sets of documents selected by heuristics like estimated relevant scores?", "paper_topic_and_main_contributions": "The paper studies the effect of context quantity and quality of input documents for FiD model when training for question answering.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 395: Contray -> On the contrary\n\nFigure 5 is a bit difficult to understand."}, "extra_text": {}}, {"review_id": "6Tai26Mm6d", "reviewer": "Reviewer_t9yE", "summary": "", "questions": "- According to Section 3.3, the context quality of each sample (instance) also needs to be considered. Have you tried to address the context quality of each sample by reflecting the retrieval scores of documents (a hint of the number of relevant documents) in equation 3?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments for Reasons To Reject section**\n\n> (Minor) In my opinion, the analysis in Section 3.4.1 is actually quite obvious and not very meaningful. When there are multiple relevant documents, attention probabilities are inevitably divided for each document, so it will show a long-tailed distribution compared to a setting with only a single relevant document.\n\nWe apologize for any confusion that may have arisen due to the lack of explicit explanation in the paper, and we’d like to clarify our analysis.\n\nIn lines 389 and 395, when we compare the attention distributions, we compare distributions shown within the same panel in Fig. 3 (e.g., the blue dashed line and the red solid line). In each panel of Fig. 3, we controlled and fixed the number of relevant documents during the evaluation. The results demonstrate that even under the same evaluation conditions (i.e., the same number of relevant documents), models with a lower training context quality show more long-tailed distributions to relevant passages.\n\n> Given the experimental results in Section 3.1, it seems more accurate that the performance degradation is due to differences in the ability/need to handle multiple related documents rather than differences in context quality (line 260). ... In the absence of such evidence, the claim should be reduced to single/multiple passage.\n\nIn Fig. 7 in the appendix, we have provided additional experimental results under different inference conditions that we consider relevant to your concerns. According to the results in Fig. 7, the performance degrades in both cases when the number of relevant documents during inference is 1 and 3 if the context quality is higher than that used during training (represented by the red dashed line). In particular, focusing on the results represented by the blue dotted line in the bottom-right panel of Fig. 7, the model's performance deteriorates when the number of relevant documents during training increases from 1 to 3, even though the number of relevant documents during inference is 3. Based on these results, we consider that our claim holds regardless of the number of relevant documents.\n\n**Comments for Questions For The Authors section**\n\n> According to Section 3.3, the context quality of each sample (instance) also needs to be considered. Have you tried to address the context quality of each sample by reflecting the retrieval scores of documents (a hint of the number of relevant documents) in equation 3?\n\nWe have not conducted experiments to estimate the context quality for each sample (i.e., each question) using relevance scores (e.g., retrieval scores or FiD attention scores) and dynamically change the intervention on cross-attention (e.g., temperature parameter) for each sample. This remains an area for future work. However, in preliminary experiments, we analyzed the correlation between passage quality (i.e., how much the passage contributed positively to correctly answering the question) and cosine similarity between a question and a passage embedding, and we did not observe strong correlations. The result suggests that using relevance scores to accurately classify relevant documents and estimate their numbers might not be straightforward. Our proposed method avoids the necessity of accurately estimating quality for each sample by estimating the single temperature parameter for the entire dataset using the development set.\n\nNote: Considering the result in the bottom panels of Fig. 5, it seems possible to estimate optimal temperature if we know correct quality during training and inference. Thus, passage quality estimation would be important direction of future work to achieve automatic temperature tuning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The writing is clear and neat, and the figures and tables are well placed to help understand the content.\n\n- This paper analyzed the influence of the quality and quantity of context in the retrieval augmented model, which is helpful to the community as it can be used as a guideline in other future studies or provide insights.", "reasons_to_reject": "- (Minor) In my opinion, the analysis in Section 3.4.1 is actually quite obvious and not very meaningful. When there are multiple relevant documents, attention probabilities are inevitably divided for each document, so it will show a long-tailed distribution compared to a setting with only a single relevant document. \n\nI do not fully agree with some of the experimental settings. If I've misunderstood, I hope the authors deal with this during the response period, and if they do, I'll be happy to revise my score. \n- Given the experimental results in Section 3.1, it seems more accurate that the performance degradation is due to differences in the ability/need to handle multiple related documents rather than differences in context quality (line 260). In the center and right-hand side experiment results of Figure 1, there was only one relevant document at the time of inference. Therefore, it can be interpreted that the performance degrades as the context quality increases because the model learned to handle multiple relevant documents during training. To prove that the performance is poor due to the difference in the context quality itself as the authors argue in line 260, it should be seen that the performance drops when the context quality increases above 0.3 with n_train = 10 on the left-hand side of figure 1 (where n+_eval > 1). In the absence of such evidence, the claim should be reduced to single/multiple passage.", "questions_to_authors": "- According to Section 3.3, the context quality of each sample (instance) also needs to be considered. Have you tried to address the context quality of each sample by reflecting the retrieval scores of documents (a hint of the number of relevant documents) in equation 3?", "paper_topic_and_main_contributions": "This paper examines how the retrieval-augmented generation model is affected by the quantity (number of evidence documents) and quality (proportion of question-relevant documents among evidence documents) of context. Through experiments, the authors revealed that context quality is more influential than quantity, and that the discrepancy of context quality during training and inference causes performance degradation. Based on these observations and analyzes, the authors proposed a method to mitigate performance degradation caused by context quality discrepancy in train-inference by adjusting the cross-attention probability.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IPW6BEzdKI", "reviewer": "Reviewer_wXc7", "summary": "", "questions": "1. Can you provide intuition behind different values of r in 3.4.2. For r=0 or close to 0, the intervention makes the model\n2. Line 251 to 253, different question may have different number of relevant passages, what would you do for questions of which the number of relevant passages is smaller than a specific setting (e.g. n+=3)? \n3. Figure 2, 5, red dashed lines show the corresponding context quality in evaluation? If so, please state it clearly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments for Reasons To Reject section**\n\n> 1. The findings and conclusion might be limited to extractive open-domain QA task (in Natural Question, and TriviaQA) that is considered in this paper. More experiments should be conducted to support the findings, or the conclusion should be stated more precisely to avoid future misleading.\n\nWe agree that our findings and conclusions should be more precisely stated as focused on extractive open-domain QA tasks. We will clarify this in the revised version.\n\n> 2. Some results are confusing and needs further explanation. For example, Table 2 shows that the model trained with low-quality training environment attend more to the relevant passages (during inference?). If keep the evaluation context is of high quality, training with low-quality would be better since we would like the model to attend more to the relevant passages. However, Figure 1 (left) shows that training with high quality would be better.\n\nWe, too, initially expected that training in a low-quality environment and learning to attend more to relevant passages would always be desirable, and were surprised by the results. While further investigation is needed to determine the cause, we hypothesize that one reason might be that training in a low-quality environment leads to excessive selectivity that causes the model to focus on only a subset of relevant passages. Such excessive selectivity makes the model overlook necessary information and, as a result, fail to correctly answer the questions. In fact, looking at the results in Fig. 3, we can see more disparities among the attention probabilities to relevant passages for the models trained in low-quality environments. It may be the case that, when evaluated in a high-quality environment (i.e., where the majority of passages are relevant to the question), it is more optimal for the model to examine all passages more uniformly without being overly selective. We will include this discussion in the revised version.\n\n**Comments for Questions For The Authors section**\n\n> 1. Can you provide intuition behind different values of r in 3.4.2. For r=0 or close to 0, the intervention makes the model\n\nWhen r=0, the model completely ignores irrelevant passages, whereas when r=1, the model attends uniformly to all passages.\n\n> 2. Line 251 to 253, different question may have different number of relevant passages, what would you do for questions of which the number of relevant passages is smaller than a specific setting (e.g. n+=3)?\n\nIn all experimental settings, we have carefully selected questions that have a sufficient number of relevant passages to conduct our experiments. For example, in our experiments with Natural Questions, the maximum value for n+ is 3, and therefore, we only use questions with three or more relevant passages as training and evaluation data in the experiments (Lines 881-884).\n\n> 3. Figure 2, 5, red dashed lines show the corresponding context quality in evaluation? If so, please state it clearly.\n\nYou are correct. The red dashed lines in Figures 2 and 5 represent the corresponding context quality during evaluation. We appreciate your suggestion and will revise the captions to clearly state this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research question and Experimental designs are interesting \n2. The findings of the paper is meaningful for future research\n3. The proposed method is simple and easy to be adopted.", "reasons_to_reject": "1. The findings and conclusion might be limited to extractive open-domain QA task (in Natural Question, and TriviaQA) that is considered in this paper. More experiments should be conducted to support the findings, or the conclusion should be stated more precisely to avoid future misleading. \n2. Some results are confusing and needs further explanation.  For example, Table 2 shows that the model trained with low-quality training environment attend more to the relevant passages (during inference?). If keep the evaluation context is of high quality, training with low-quality would be better since we would like the model to attend more to the relevant passages. However, Figure 1 (left) shows that training with high quality would be better.\n", "questions_to_authors": "1. Can you provide intuition behind different values of r in 3.4.2. For r=0 or close to 0, the intervention makes the model\n2. Line 251 to 253, different question may have different number of relevant passages, what would you do for questions of which the number of relevant passages is smaller than a specific setting (e.g. n+=3)? \n3. Figure 2, 5, red dashed lines show the corresponding context quality in evaluation? If so, please state it clearly.", "paper_topic_and_main_contributions": "The paper investigates how the quality and quantity of context affects the performance of Fushion-in-Decoder for retrieval augmented generation. The authors proposes a method for measuring the quality of the context based on the ratio of the relevant of passage to the non-relevant ones. A number of experiments are designed where the main findings are: 1) the training of Fid is overfit to the quality of context; 2) the overfitting is partially due to that training with low quality context makes the model pays more attention to the relevant passages. The authors then propose a simple method to adapt the cross-attention probability during inference, which is shown to outperform FiD.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 403: Interventioal Experiment > Intervention Experiment\nFigure 4: Evaluation environment (n+, n) or Evaluation environment (n+, n-)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting analysis of the effects of context quality (ie the proportion of relevant passages in the top N documents)  on extractive open-domain QA. They found that FiD's resulting performance is strongly affected by the quality of the context during training and can overfit the training data distributions, resulting in degraded performance when test context quality differs. Based on the observations, they introduce a temperature parameter to the loss, to change the sharpness. 

Overall, the research questions and experiments are intriguing, but the main takeaway isn't really surprising to me. To summarize, the core finding (e.g., Section 3.1; *For a given evaluation context quality, models trained with similar context quality showed the highest performance*) is not surprising and can be explained by the tendency of a model performing poorly when there's distribution shift between training and test.  This point is also noted by the reviewer t9yE. On the other hand, I think some findings on attention distributions are somewhat underexplored in prior work and provide unique contributions, and may inspire future work to further investigate some mixed results in this work (e.g., Table 2). 

Another concern I have is that the analysis is limited to a single model architecture (FiD) on an extractive question-answering task, and given the experimental designs, this analysis may not be easily adapted to other architecture or tasks.
Decision: Accept-Findings