Hindsight: Posterior-guided training of retrievers for improved open-ended generation
Authors: Ashwin Paranjape
Omar Khattab
Christopher Potts
Matei Zaharia
Christopher D Manning
Venue: ICLR 2022 Poster (2022)
Keywords: retrieval
generation
retrieval-augmented generation
open-ended generation
informative conversations
free-form QA
posterior distribution
ELBo

Abstract: Many text generation systems benefit from retrieving passages from a textual knowledge corpus (e.g., Wikipedia) and using them to generate the output. For open-ended generation tasks, like generating informative utterances in conversations, many varied passages $z$ are relevant to the context $x$ but few are relevant to the observed next utterance $y$ (label). For such tasks, existing methods (that jointly train the retriever and generator) underperform: during training the top-k context-relevant retrieved passages might not contain the label-relevant passage and the generator may hence not learn a preference to ground its generated output in them. We propose using an additional guide-retriever that also conditions on the observed label $y$ and “in hindsight” retrieves label-relevant passages during training. We maximize the evidence lower bound (ELBo) to jointly train the guide-retriever $Q(z|x,y)$ with the standard retriever $P_\eta(z|x)$ and the generator $P_\theta(y|x,z)$ and find that ELBo has better inductive biases than prior work. For informative conversations from the Wizard of Wikipedia dataset, with our posterior-guided training, the retriever finds passages with higher relevance in the top-10 (23% relative improvement), the generator’s responses are more grounded in the retrieved passage (19% relative improvement) and the end-to-end system produces better overall output (6.4% relative improvement). 

Published as a conference paper at ICLR 2022
HINDSIGHT: POSTERIOR-GUIDED TRAINING OF RE- TRIEVERS FOR IMPROVED OPEN-ENDED GENERATION
Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia & Christopher D. Manning Stanford University (ashwinp: np,, okhattab] 8cs stanford.edi
ABSTRACT
Many text generation systems benefit from retrieving passages from a textual knowledge corpus (e.g., Wikipedia) and using them to generate the output. For open-ended generation tasks, like generating informative utterances in conversa- tions, many varied passages are relevant to the context but few are relevant to the observed next utterance (label). For such tasks, existing methods (that jointly train the retriever and generator) underperform: during training the top-k context-relevant retrieved passages might not contain the label-relevant passage and the generator may hence not learn a preference to ground its generated out- put in them. We propose using an additional guide-retriever that also conditions on the observed label y and "in hindsight" retrieves label-relevant passages dur- ing training. We maximize the evidence lower bound (ELBo) jointly train the guide-retriever Q= y) with the standard retriever (=(xr and the generator (o(ylar, z) and find that ELBo has better inductive biases than prior work. For in- formative conversations from the Wizard of Wikipedia dataset, with our posterior- guided training, the retriever finds passages with higher relevance in the top-10 (23% relative improvement), the generator's 's responses are more grounded in the retrieved passage (19% relative improvement) and the end-to-end system produces better overall output (6.4% relative improvement).
INTRODUCTION
In neowledge-intessvve NLP tasks, models must use open-domain knowledge to answer questions (Kwiatkowski et al.] 2019 Joshi et al., (017). fact-check claims (Thorne et al. 2018) or engage informative conversations (Dinan et al. 2019 Zhou et al.) 2018). State-of-the-ar models for open-domain question answering are eetrieval-augmente human-readable corpus (e.g.. Wikipedia) using a learned retriever and process it with a tssc-sccc reader. If the relevant passage is known (e.g., human-annotate gold passage), the retriever supervised with it. In this work we consider open-ended generation tasks where the oold-passages unknown. Figure [ illlstrates this one-to-many setting: for conversational context x. many relevant passages = (dubbed context-relevant passages) could have generated many coherent re- sponses. But only 2gold (dubbed label-relevant passage) generates the observed target output y- Had we known Egold corresponding to the target output, we could have supervised the retriever with gool and trained the generator conditioned on Zgold but but we don't!
they extract relevant passages from
be
Current methods for retrieval-augmenttd generation (Lewis et al. 2020) work well for short-answer QA-like tasks: Natural Questions (Kwiatkowski al.; 2019) or fact-checking (Thorne et al,, 2018). Lewis et al. (2020) use the generator's probability distribution Po 2) as a proxy for label relevance and train the retriever (=(xx) by marginalizing p(y(æ) over retrieved documents P(ylz) Ecetop (,,l))) Po z). However, for one-to-many tasks, this objective leads to suboptimal solutions: the generator is less grounded in the retrieved passages (Figure the retriever performance saturates at low recall (Figure 33,, and the top-k retrieved passages exclude many label-relevant passages weakening the supervision during training (T.
In our work, as a proxy for Zgold: we train a separate guide-retriever model to find label-relevant passages. The guide-retriever both the input and the output y and is represented by

---

Published as a conference paper at ICLR 2022
X
2
y
Miles Dewey Davis (May 26, 1925- September 28, 1991) was American tnumpeter, bendleader, among most influential accialmed
compoeer. He is
A: would say I like the
composttons Miles Devis
figures the history jazz and 20th century music.
A: Jazz is characterized by swing and blue nates as well call response vocals Wow, that's is so cool. Who are some popular jazz musicians?
Louis Daniel Armstrong (August 1901 July 1971), nicknamed "Satchmo". "Satch" and "Pops", was American trumpoter vocalat. among most influential figures in jazz.
A: you heard
Ammsronn?? known as "Satch"?
Elle Jane Fitzgerald (April 25, 1017 1806) was an American jazz singer, sometimas referred to as the First Lady of Song, Queen of Jazz, and Lady Ella.
15,
A: Elta Fitzgerald, the Queen of Jazzz was a really popular jazz singer
Figure 1: A conversational turn with many plausible responses. The input (blue) can be answered based on 3 equally context-relevant passages but only one possible response (yellow) is observed in the training set based on only one of the pink label-relevant passages (outlined in black).
the label-posterior distribution Q(=|x, y) that captures label-relevance in "hindsight". Model- ing the label-posterior distribution Q212z y) with a full-fledged retriever generalizes weak su- pervision approaches and retrieves label-relevant passages from the entire collection. We jointly optimize the retriever, posterior-guine and generator using the evidence lower bound (ELBo): (Q)(p) While the objective function is lower bound, it en-
(s,,Q()(o.)) [log Po(y|æ, codes biases that improve joint-training training on open-ended tasks: (1) conditioning the generator on the passages weighted by their label-relevance (from the label-posterior distribution) increases ground- ing and (2) training the retriever with a mode-seeking reverse-KI. divergence encourages it to match some modes with the guide (label-relevant passages), with a lesser penalty for matching other modes (other context-relevant passages).
Our main contribution is a complete HINDSIGHTT training system that: (I) uses guide-retriever to provide a stronger learning signal for both the generator and the retriever and (2) is amenable to aadex-updates with iterative closed-set training (Section 3). To evaluate one-to-many open-ended generation tasks, it is insufficient to just evaluate the end-to-end performance of the joint system. Thus, we also evaluate the individual models (retriever and generator) and at varying passage depths. Using HINDSIGHT on the Wizard of Wikipedia dataset of informative conversations: the retriever finds more relevant passages with a 23% relative improvement (r.i.) success@ 10 (i.c., is the label-relevant passage among the top- 10 retrieved passages?), the generator is more grounded with 19% r.i. in Novel-F1 overlap with the top-1 retrieved passage (i.e., its overlap with the retrieved passage excluding words that common or in the input) and the combined system is overall better with a 6.4% r.i. in Novel-F1 @1 overlap with the gold utterance (the best matching generation when considering top-1 retrieved passage). HINDSIGHT also improves performance the MS-MARCO NLGen dataset, a one-to-one free-form QA task.
2 BACKGROUND
Open-domain Question Answering In the reading comprehension task, passage is given and the models extract the answer span from it. In Open-domain QA (a.k.a. open-QA) no such passage given; the models are expected extract the answer from a large document corpus. Dr. QA (Chen) et al. 2017), the first neural system for factoid open-QA, used an off-the-shelf retriever (c.g.. TF- IDE, BM25) to find relevant passages and trained reader to extract the answer span. Now, trainable neural retrievers have replaced the classical term- matching retrievers. Here, pre-trained models (like BERT) embed the document corpus and the query into a single vector space and efficient nearest- neighbour search algorithms (Jegou et al. 2010: Johnson al. 2017) find the relevant passages corresponding to the query. The neural retriever can be trained variously: pretraining with the inverse cloze task then weakly supervising using span matches (Lee et al. (201)). using gold passages with in- batch negatives (Karpukhin et al. 2020), and retrieval-guii -guided supervision with span-based positives (Khattab et al. 2021).
2

---

Published as a conference paper at ICLR 2022
Open-ended Generation Natural language generation tasks provide some input (sequence of tokens, image) and expect the system to produce another sequence of tokens (or word-pieces, as output. An open-ended task accepts higher diversity of generations. Factoid suustion-ansseringg with a single correct short answer is less open-ended than free-form long answers. Machine transla- tion accepts few correct translations (Bojar et al., 2014) but they less diverse than informative dialogue, where the speakers can lead the conversation in many different directions (Dinan et al.) 2019) making more open-ended. Many more generation tasks such as summarization (Narayan et al. 2018) and story generation (Mostafazadeh et al. 2016) lie on this spectrum.
Retrieval for Language Modeling Khandelwal al. (2020) retrieve similar contexts from the training set at each time-step and increase the likelihood of tokens that were predicted in similar contexts. Guu et al. (2020) instead pre-train retrieval- -uggeeettd masked language model using salient-span masking and fine-tune it on downstream QA tasks.
Using labels for direct supervision Zheng et al. (2020) use term-overlap with the label as a heuristic to identify the gold-passage from a small passage set (~ 50) and train a reranker. Prior work has also modeled the posterior of various probabilistic models (Lian et al. 2019 Kim et al.) 2020 Zhan et al. 202T) used reinforcement learning (Zhao et al., (020) improve knowledge selection from the small passage set. In Zheng et al. (2021) the authors increase grounding by using the label to reweigh passage tokens and in Cai et al. (2019) they increase grounding by feeding corrupted version of the label to the generator as a stand-in for the label-relevant passage during training.
Retrieval- -atieeeaee-umeeeeeddd Generation Lewis al. (2020) introduce setrivall-ugmeent generation, where, for input æ and output y, retriever finds top-k passages (z) from corpus and jointly train with a generator (Pa) by maximizing the likelihood of the output marginalized over the top-k documents. In this work, we refer to this loss function as the MARGINALIZEDLOSS:
P(ylx)
Σ (P
Po(ylee)
(I)
Here Poll, z) is conceptually utilized in two roles: first, supervising the retriever (i.e., teaching the retriever to score label-relevant passages higher than other passages) and keeping the generator grounded (i.e., maximizing the probability of the target output given the context-relevant passages). In the next section we introduce a guide-retriever to capture the label-relevance and we train it using ELBOLLss, lower bound to MARGINALIZEDLLOS, that has better inductive biases.
TRAINING WITH HINDSIGHT
To identify label-relevant passages, we explicitly model the posterior distribution Qz y) with learned neural model. Unlike the retriever (=|z)) the label-posterior model has access to the target output and in hindsight can differentiate the label-relevant from other context-relevant passages. We learn the label-posterior jointly with the retriever and the generator by maximizing the evidence lower bound, ELBOLOSS, as given by the formula:
log P(y|z) > Ez-Q(()a,,) [log (vllx, DKI. (Q)).)
The ELBoLLSS has two terms with useful inductive biases. The first term maximizes the expecta- tion of the generator's ligelikeld Po over the passages sampled from the label-posterior distri- bution Q. The generator need attend only to the label-relevant passages, biasing it toward relying more on the retrieved passages rather than its internal language model. The second term is the KL divergence from the retriever to the label-posterion also referred to as the reverse KL divergence:
DKL (kL(((((1,))) P,(22[))
Σ
Q(2lz,8) (log Q(2|2, v) - log (z(2)))
Z~Q( ...)
This term is again weighted by Q(2|z, v). making it like probabilistic implication: high Q(=læ, y) implies high P(z]x), i.., label-relevance implies context-relevance but not vice-versa. In one-to- many tasks, which have many context-relevant passages but few label-relevant passages, this term captures the intuition that the retriever be penalized heavily if it doesn't retrieve the label-relevant passage but lightly if it retrieves other context-relevant passages that happen not be label-nelevant.
3

---

Published as a conference paper at ICLR 2022
Outer loop with index refresh
y Q
Closed
Fast inner loop for model training
Update P.... Q
Passage
Corpus
Index
Retrieve
Zr.
Rescore
P,(e(() (x)
Train P
Sample
using P
[()|x,)))
Po Q
Index
Retrieve
Z
An overview of iterative closed-set training: We iterate through the outer-loop and call each execution a round. At the beginning of the round we re-index the passage corpus using the latest retriever (=[æx) and guide-retriever Q(= lr, y) to create high-recall closed-set of top-r passages for each retriever and query. Then, in the fast inner loop, we train the models for multiple epochs by sampling passages from the fixed closed-set and recomputing the probability distributions. The trained models are then used in the next round.
Figure 2:
Posterior as a retriever Rather than modeling the label-posterior Q(2 y) as a re- (that merely reranks documents retrieved by the retriever P,), model it guide retriever that finds label-relevant passages from the entire corpus. We sample passages from the label-posterior distribution, and estimate the ELBoLOSs more accurately than using passages from (2 (2(ar). The guide retriever generalizes weak supervision approaches (Lee et al. 2019) Guu et al. 2020) and relevance-guiie guiiedd supervision (Khattab et al. 2021). posterior-guided supervision with learned posterior retriever rather than brittle heuristics based on word-overlap.
Iterative closed-set training Prior works (Guu et al. 2020 Khattab al. 2021) intermittently update the passage index during training. To allow for such a workflow, we organize our training into rounds (see Figure D At the beginning of each round, the outer loop, we encode the passages and the queries with various retrievers and find the highest scoring passages that we dub the closed- set. the inner loop that runs for many epochs, we sample k (= 8) passages from the closed-set (r 100). This is fast because are longer retrieving from the entire corpus in the inner loop and also sufficient because the closed-set has high recall. During the inner loop, we update the retrievers (both document and query encoders) and use the latest model parameters for computing the loss functions. A round results in trained models that are then used for the next round. We find that 2 rounds are often sufficient, with decreasing marginal utility from the third round onward.
Distributional repositioning before inference We approximate the expectation terms in EL- BoLoss by sampling passages from the closed-set (- y). which provides better super- vision than (2(2)) and leads to faster training. However, the models only ever get exposed to passages from the Q.1[z, 3) distribution, which limits their ability generalize over passages from (a(+)z during inference. To remedy this, we instead sample passages from an a-mixture of the two distributions: with probability a,
(-(x)) and with probability initial rounds we set low values of a and increase it toward the end to reposition the passage distri- bution and better match with (=(xr at test time. The retriever and the generator can be trained by sampling passages from different a- a-mixtures and we utilize this to avoid retriever overfitting (with
a, Q(.|xr y). In the
a = 1) while maintaining generator groundedness (with a = 0.25, 0.5).
Training individual models to convergence In practice, the retriever and the generator when jointly trained converge at different times. The single loss term in MARGINALIZEDDLOS (Eq. I hides the convergence of individual models; one model starts overfit while the other model still hasn't 't converged. With EL Bosss, there are two terms in Eq. 2 the first connecting Q(=|æ, y) and (oll, z). the second connecting Q(=|æ, y) and P, (z(xr. After training for few epochs, we freeze the guide and train the models independently until convergence based on their individual losses.
4 EXPERIMENTAL EVALUATION
We evaluate on two open-ended knowledge-intensive tasks: informative conversations and free-form question answering. We ask the following three research questions:
4

---

Published as a conference paper at ICLR 2022
65
Marg. Loss
0.22
60
ELBo (a=1)
passage
0.20
Successik (%)
0.18
Marg.
55
El (a- 1.0)
EL8o (a=0..5)
50
0.14
(a=0.25)
0.12
ELoo (a=0)
45
0.10
40
0.08
2
6 (K)
10
5
6
7
8
passage
Figure 3: Relevance and Groundedness of models trained on the Wizard of Wikipedia dataset: (left) success@K of retrieved passages w.r.t. rank and (right) Novel-F1 between decoded output and re- trieved passage w.r.f. retrieved passage rank. The ELBoLLss retriever is more effective at retrieving retriever, especially when consider the top- 10
the gold passage than the MARGINALIZEDLOO passages for this one-to-many task. The ELBoLOsS generators have higher overlap with top-k retrieved passages and the overlap increases decreases.
RQ: Relevance: Are the retrieved passages more relevant? (Section 4.4)
RQ2 Groundedness: Does the generator make better use of the retrieved passages? (Section 4.5
RQs Generation Quality: Does this lead better end-to-end performance? (Section 4.6
4.1
MODELS
Retriever Models We model the retriever (z(xx) and the guide-retriever Q22aæ, y) using Col- BERT (Khattab & Zaharia 2020). ColBERT encodes the query tokens qi and the document tokens d, independently using BERT, normalizes produce unit-vectors Eg. and Ed,, and defines simi- larity Sq Ej max, F. F.z,- Unlike DPR's [CLS] token embedding (Karpukhin al. 2020). with ColBERT's late-interaction paradigm the query and document tokens retain their Identities and contribute a finer- finer-grained term-wise similarity leading to state-of-the-ar retrieval results on open-domain QA benchmarks (Khattab et al. 202T). To convert similarity scores into probability distribution, calculate the softmax of the scores over the sampled passages. For the posterior- retriever, concatenate the input and the output into the query g [x y). CoIBERT pre-trained on MS-MARCO passage ranking dataset widely used for other tasks and we use it the Wiz- ard of Wikipedia task. However, the MS-MARCO NL.Gen task contains queries from the passage ranking pre-training dataset. Therefore, following Lewis et al. (2020), we use Natural Questions to pre-train ColBERT Khattab et al. (2021) for the MARCO NL task.
Following Lewis et al. (020) we use a pre-trained BART model and fine- tune for the respective tasks during training. It is conditioned both the context and the document and trained to produce the target. At test time, we decode using beam-search with a beam size 4.
Generation Model
4.2 TASKS
Informative conversations Informative conversations are open-ended because people have the ageney to drive them in different directions at every turn cone-to-many and are knowledge-intensiee because the utterances contain specific bits of world knowledge. We evaluate with Wizard of Wikipedia (WoW) dataset (Dinan et al., 2019). where "apprentice" chats (via text) with "wiz- ard", being curious about different topics, and the "wizard" grounds their response from Wikipedia. The input for this task is the conversational history the output is the wizard's utterance y and the models can retrieve individual passages (2) from all of Wikipedia (=266 mil- lion passages). We the version of this dataset provided in the KILT benchmark (Petroni al.) 2021) and report leaderboard performance on the held out test set. We use the dev set to answer the granular research questions.
sentence
5

---

Published as a conference paper at ICLR 2022
Table 1: Relevance evaluation: Our method (ELBoLoss Retriever, = strongly improves over the baseline (MARGINALIZEDLLOS Retriever) for the one-to-many Wizard of Wikipedia dataset, in particular for 5, 10. The ELBo posterior finds zgold with high success providing better supervision during training. (MRR = Mean Reciprocal Rank, Success @ k both in percentages.)
Wizard of Wikipedia
MS MARCO NLGen
Method
MRR
S@1
S@5
S@
MRR
S@1
S@5
S@
10
Gold-sup.
45.2
35.6
57.0
63.1
28.9
19.5
40.4
47.7
Marg.
43.8
38.9
49.9
52,8
30.4
19.4
43,4
53.2
BLBo (a =
49.0
41.1
58.8
63.9
32.1
21.2
45.3
54.4
Free-form Question Answering We use the MS-MARCO NLGen dataset (Nguyen et al. 2016) where the task is generate natural-sounding answers to questions. This free-form open-QA task one-to-one but more challenging than other extractive open-domain QA datasets. The dataset is a subset of MS-MARCO questions whose answers were reviewed by separate editor and rewritten if they had a high overlap with one of the provided passages (indicating that the original editor may have copied the passage directly). These "well-formed answers" meant be complete sentences (such as can be read out by conversational assistant) and are long (median length 11 words). The input for this task query x, the output well-formed answer y, and the models can retrieve from the MS-MARCO passage collection (8.8 million web passages). The public benchmark and the test set is no longer available for evaluation. Instead we split the public validation set into validation and test set and show results on the test set.
While both datasets annotate the passages referred to by the person who wrote the target output (gold passages), we only use them for evaluation and validation and not for training.
4.3 BASELINES
Apart from the two main methods - MARGINALIZEDLLOSS and ELBoLoss we train two ad- ditional baselines: gold-supervised and generator-only. For the Gold-supervised baseline, assume that the gold-passage Zgold is available during training and train a retriever by maximizing its log-likelihood. We take random passages from ELBo Retriever's closed-set as negatives (exclud- ing top-10 to avoid false negatives). We train the Gold-supervised generator by simply maximizing Po(y)z, 2gold). For the generator-only baseline, we ignore the existence of passages and directly maximize P(y with sequence to sequence model.
4.4 RELEVANCE EVALUATION
We evaluate the relevance of the retrieved passages (RQ;) using the gold passage labels supplied by each dataset. For one-to-many tasks, expect the label-relevant passage to be one of the top-k (k = {1, 5, 10}) retrieved passages. Thus, we report Success@K k (S@A for short the percentage of inputs for which any gold provenance passage is retrieved within the top-k (k: = {1, 5, (0)) passages. We also report Mean Reciprocal Rank (MRR), a common IR evaluation metric.
Our results shown in Table I With Wizard of Wikipedia, ELBoLoss retriever markedly out- performs MARCINALIZEDDLOSS Both systems have a relatively high Success@ and easily handle 38.9-41.1% of the examples, but the EiBBoLoss retriever continues to find many more relevant passages at larger retrieval depths & widening the gap to 11 points for Success@ 10. With MS MARCO NLGen, ELBoLoss outperforms MarginalizedLoss by 1-2 points across our metrics, re- flecting smaller-but nonetheless consistent- one-to-many generation task. Based on manual inspection, we find many false negatives (corrobo- rated by Arabzadeh et al. (202T), i.e. passages that contain the answer but aren't 't marked as gold, leading to a lower Success @ compared to Wizard of Wikipedia.
when compared with Wizard of Wikipedia,
With single gold passage Recall @ and Success@R @k are numerically identical and sometimes used inter- changeably; we prefer S k because it is less ambiguous and widely used in the IR community since Voorhees 2004).

---

Published as a conference paper at ICLR 2022
Table 2: Groundedness evaluation: Our method ELBoLoss (a 0.25, 0.5) shows more overlap between generated output and the retrieved passage than MARGINALIZEDLOSS and for the Wizard of Wikipedia dataset the gap increases as we consider the maximum over top-ó passages. (Novel-F1: discounts commonly occurring words and context words (z)).
Top-1
Max. of Top-5 F1 Nov-F1
Dataset
Method
F1
Nov-FI
Gold-sup. Generator
18.84
17.12
32.47
31,92
WoW
Marg. Generator
18.63
17.46
26.19
25.39
ELBo Generator (a 0.25)
21.34
20.78
34.16
34.24
Gold-sup. Generator
36.13
28.34
49.29
43.45
MSM
Marg. Generator
33.12
25.39
45.76
39.45
ELBo Generator (a 0.5)
34.52
26.49
46.91
40.47
Effect of the guide-retriever MARGINALIZEDLOSS depends on the retriever (2/22 [) to find label-relevant passages during training and therefore recall limited. MARGINALIZEDLOSS' suc- cess@) 100 Wizard of Wikipedia saturates at 55.8% (not reported in the table) without much hope of further improvement because Zzold that are never retrieved cannot provide positive exam- ples for supervision. With ELBoLLos,, the guide-retriever retrieves label-relevant passages with >85% success@ 10 (Table S for both the datasets providing better supervision than MARGINAL- IZEDLoss. Consequently, ELBoLOSS retriever's 's success @ is higher than MARGINALIZEDLOOS despite containing x0x less passages and reaches 69.3% for success 100.
Comparison with Gold-supervised retriever We find that the Gold-sup. retriever quickly over- fits during training resulting in lower performance than using ELBo loss. We also see lower performance when training with ELBoLoss while sampling purely from Q(-[x,,) (i.e., Table 7 because it low-entropy and the same label-relevant passages get repeatedly sampled. By sampling from (-r),, the KL divergence is minimized over wider and realistic support set of pas- sages. Perhaps, there are many passages that have some label-relevant phrases making them partially relevant and Q(=[z; (2, y) "teaches" (2|x) to capture these phrase-level relative differences. Sam- pling from P, (-12) leads to better generalization, has similarities to distillation and is an interesting direction for future work.
0,
Overall, we find that EL BLBLLss improves relevance of retrieved passages over MARGINALIZED- Loss for two qualitatively different tasks, with larger gains for the one-to-many generation task.
4.5 GROUNDEDNESS EVALUATION
We now examine RQ2, studying the degree to which the generator relies on the retrieved passages for producing its output. To quantify this groundedness, passage (not necessarily the gold passage) and the produced text when generation is conditioned on that passage. We get the retrieved passages for each method (except generator-only) using the corresponding retriever.
compute F1-overlap between retrieved
As analogue of Success@k, we propose Max. @l@k, the largest Fl-overlap exhibited by any generated output with the corresponding retrieved passage fed to the generator. We also propose Novel-F1, new metric that discounts words that occur frequently and words that already appear in the context v, since otherwise these tokens dominate raw F1 in practice (up to 80%, see Ap- pendix A.5) but are not indicative of grounding.
Our results are shown in Table 2and Figure 3 For the Wizard of Wikipedia dataset, we observe that our ELBoLOSS generator outperforms MARGINALIZEDLOOS by 2.7 Fl (14.5% relative improve- ment) and 3.3 Novel-F1 (19% r.i.) when given the top retrieved passage. For the MS MARCO NL- Gen dataset, we observe smaller but consistent gains in groundedness (1-2 F1, Novel-F1) with EL- BoLoss compared to MARGINALIZEDLOSS. In Figure 3 (right), MARGINALIZEDLOOS genera- tor's 's overlap decays rapidly beyond the top passage, whereas the ELBoLoss (a = 0.25) generator's overlap declines gradually. This shows that the ELBoLLss generator stays grounded beyond just

---

Published as a conference paper at ICLR 2022
the top passage, desirable property in one-to one-to-many generation systems. We also see (in Figure right) that groundedness increases as a decreases. However, a generator trained with a = 0, de- spite being maximally grounded (Appendix table has lower end-to-end performance (Appendix table () because it is unduly "trusting" of the provided passage (nearly flat line in Figure 3) and does not abstain from using irrelevant passages. We see the same effect with the Gold-supervised generator on MS- MS-MARCO NLGen: it is more grounded but has lower downstream performance.
Overall, we find that ELBoLLSS improves grounding of the generator over MARGINALIZEDLLOSS for two qualitatively different tasks, with larger gains for the one-to-many generation task.
4.6 END-TO-END EVALUATION
To evaluate the end-to-end quality of our systems, we calculate FI and Novel-F1 (defined Sec- tion 4.5) of the decoded output with the human-written gold oulput. To allow for the possibility of the generator using any part of the gold passage (and not not the human -wrttten gold output) for the Wizard Of Wikipedia task, we use Knowledge-Fi (defined by ShSusteer et al. (2021)): )): F1 between the sampled generation and the gold passage. Since is reasonable to expect the gold passage to be in the top-k for k>1 for one-to-many tasks (as in Section 4.5) retrieved passages.
also compute the max. over top-k
The results are summarized in Table For the Wizard of Wikipedia dataset, using only the top retrieved passage ELBoLOsS leads to 6.7% relative improvement in Novel-F1 @ 1. But in the one- to-many setting, the label-relevant passage is arbitrary choice from amongst the context-relevant passages, We account for that using max. overlap over the top-5 passages and see larger improve- ments for ELBoLoss, namely FI, Novel-F1 (~ 15% ri.), and 1.5 K-F1 (~ 10% ri.). For MS Marco NL.Gnn, we see a small but consistent increase due to ELBoLoss over MARGINALIIZED- LOSS: 1.5 F1 and 2 Novel-F1 across passage depths.
We also submit the above models (ELBoLoss and MARGINALIZEDLLOS) to the Wizard of Wikipedia task on the KILT leaderboard. ELBOLOSS consistently outperforms the baseline MARGINALIZEDLOSS across all metrics (see Table 4. The table also reports Recall@5, which evaluates retrieval at a coarser granularity, namely at the full Wikipedia puge level, though so far have investigated it directly at the passage level. Consistent with the results in Table [ our method also outperforms MARGINALIZEDLOSS in retrieval metrics. In fact, our ELBOLOSS model achieves state-of-the-ar performance across all the generation metrics (F1, ROUGE-L, KILT-F1, KILT-ROUGE-L) on the leaderboard, though it is not the strongest on R-Prec and Recall @5
To conclude, we have evaluated the ELBoLOSS and MARGINALIZEDLOSS using one-to-one free- form QA dataset and a one-to-many dataset of informative conversations. Our results show that our method ELBoLOSS trains a better retriever, more grounded generator and improves end-to-end performance, especially in the one-to-many setting.
5 DISCUSSION
Shuster et al. (2021) show that providing retrieved passages to a generator reduces hallucination. Our work increases grounding the retrieved pas- sage, promising to further reduce hallucination. Even though the generator is now more likely to use content from the provided passage (rather than hallucinating from parametric memory), that does not guarantee correctness. Our token-level overlap metrics that evaluate grounding do not capture this aspect either. There is scope for future work to address this gap with better training methods (and evaluation metrics) that produce (and reward) grounded and correct outputs.
Hallucination, grounding and correctness
Practical matters: Trust and control For tasks like QA, we want a "conservative" generator: it should abstain from using passage that doesn't contain the answer. For more open-ended tasks like informative conversations, we want the generator make use of diverse passages. In our work, we show that by reducing a, the distribution of the passages shifts from (,, (z) to Q( y) and the generator increasingly trusts the retrieved passages (Figure 3 System designers can use the
2 Earlier results on the KILT leaderboard for Wizard of Wikipedia should be interpreted with caution, as the KILT authors recently updated the train/dev splits due to anomalies in the preprocessing script. We have used the updated version for our model and baseline.
8

---

Published as a conference paper at ICLR 2022
Table 3: End-to-end evaluation: Our method ELBoLoss improves over MARGINALIZEDLOSS when considering Max. overlap of generated output with target output over top-5 passages for the Wizard of Wikipedia dataset and also for top-1 with MS Marco NLGen dataset. (Novel-F1: dis- counts commonly occurring words and context words (r), Knowledge-Fll: overlap of generated output with gold passage.)
Top-1 N-FI
Max. of Top-5
Dataset
Method
Fl
K-FI
F1
N-F1
K-FI
Gold-sup.
16.70
8.53
11.64
24.95
14.87
16.16
WoW
Gen. Only
16.11
5.15
8.05
Marg.
18.79
10.45
12.61
26.52
16.42
16.02
EI.Bo
18.86
11.12
13.08
27.56
18.67
17.69
Gold-sup.
59.25
36.22
71.44
55.02
MSM
Gen. Only
51.75
14.71
Marg.
60.18
37.19
72.22
56.06
ELBo
61.46
39.65
73.18
58.19
Table 4: Wizard of Wikipedia KILT leaderboard evaluation: ELBoLoss achieves SoTA on gen- eration metrics (F1, ROUGE-L, KILT-F1, KILT-ROUGE-L indicated with f) as of Oct 2021 and improves relevance over MARGINALIZEDLLOS
R-Prec
Recall
F1
ROUGE-L
KILT-FI
KILT-ROUGE-L
Re2G (prev. best)
60.10
79.98
18.90
16.76
12.98
11.39
Marg.
53.94
68.12
18.11
16.21
11.78
10.47
ELBo (curr. best)
56.08
74.26
19.19t
17.06
13.39
11.92
a-mixture tool to modulate the degree of trust placed by the generator in the retrieved passages. Further, when "trusting" generator is deployed real-life settings (e.g., in open-domain socialbots; Paranjape al. 2020). external business logic select appropriate passage from the top-k retrieved passages and effectively control the generated content.
Comparison with Fusion-in-Decode generator simultaneously and in Izacard Grave (2021b) they use the decoder's attention weights each passage for relevance supervision. However, (aacann al. (202I) use Fusion-in- iusion-in-Decoder only at inference, forgoing the decoder's attention weights and using an equivalent version of the MARGINALIZEDI.OS. for training the retriever. In our experiments ments while using indeooeerr at inference and we believe our approach of conditioning on individual passages is more precise for relevance supervision in one-to-many generation tasks.
Izacard & Grave 2021 provide multiple passages the
found marginal improve-
Inductive biases of HINDSIGHT A learned P, (z(x) that maximizes ELBoLOss (a lowerbound) can therefore seem counter-intuitive first that EL-
will also maximize MARGINALIZEDLOGS BoLoss empirically outperforms MARGINALIZEDLA
We hypothesize that it better captures the necessary inductive biases (refer Section 3 The a-mixture further loosens the bound theoretically, inducing bias toward the test distribution and improving generalization. Trading off theoretical tightness of bounds for desirable inductive biases proved fruitful for us and we hope that inspires other researchers.
this paper, we propose HINDSIGHT,, a system that introduces a guide-retriever to improve su- pervision for both the retriever and the generator for retrieval-ause aarrival-augmennted, open-ended generation. During training, the guide retriever uses the target output of each example in order to find relevant passages, leading to better retrieval and more grounded generation. The resulting system achieves considerable empirical improvements over existing work, improving retrieval quality by up to 23%, grounding by up to 19%, and end-to-end output quality by up 6.4%.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers and Stanford colleagues: Eric Mitchell, Nandita Bhaskhar, Shikhar Murthy, Amelia Hardy and Antoine Bosselut for their valuable feedback. This research was supported in part by Stanford HAI, Samsung Electronics Co., Ltd, and affiliate mem- bers and other supporters of the Stanford DAWN project-Ant Financial, Facebook, Google, and VMware-as well as Cisco, Virtusa, SAP, and the NSF under CAREER grant CNS-1651570. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
REPRODUCIBILITY STATEMENT
Section contains key details of our method. The code for recreating these experiments along with hyperparameters will be released at htttttss/ggtthttt www. ccom/AswwinPaaaajaaaaa hindsight
ETHICS STATEMENT
Any generative system that produces free-form text and interacts with humans needs to have safe- guards in place against harmful speech. In our work, try reduce hallucination and increase grounding which a step in the right direction. It also helps restrict the content and exert control over the generation. But as we point out in the discussion (Section (5) increased grounding does imply correctness. These systems by themselves are not ready to be deployed in the real world and future work needs to address outstanding issues.
REFERENCES
Negar Arabzadeh, Alexandra Vtyurina, Xinyi Yan, and Charles LA Clarke. Shallow pooling for sparse labels. arXiv preprint arXiv:2 (ttt::2000.00002., 2021.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Lev- eling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Pro- ceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12-58, Baltimore, Mary- land, USA. June 2014. Association for Computational Linguistics. doi: URL https: http:///aaaaattooococcoo
2agg/W14-333 -302
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. Retrieval- guided dialogue response generation via matching to to-generation framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference Natural Language Processing (EMNIP-IJCNI) pp. 1866- 1875, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DD 10133//1/D11-111 11D19-1195, htttss/t:::///tt acl antho ogy- org/ollgll0 9-11199911115
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1.1.8665////11--1777. URL https:
//aclanthology. ort/
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge- powered conversational agents. In International Conference on Learning Representations, 2019.
net wett/
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3938. PMLR, 2020. URL http: http:::/tooceeeeeeettet /ttp://eccccee.g..... mlr. wttpss/wwl//gpie////p html
10

---

Published as a conference paper at ICLR 2022
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open
domain question answering. In Proceedings of the 16th Conference of the European Chapter of
the Association for Computational Linguistics: Main Volume, pp. 874 880, Online, April 2021a.
Association for Computational Linguistics. URL https: / antha org/2021.
(eacl ...in. 74
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question
answering. In International Conference Learning Representations, 2021b. URL https:
//openreview. net
http:e/.nn//oormmmmeeeeee fttr//orrmmmidde Neea--TTE---555 Siw---d---EE---yydd bwysdb
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-122, 2010.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. arXiv preprint arXiv: 1702iv:170..00 (tt.::/772.875555 2017.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp- 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 100118633/1//117-11
URL https::///// http:///ccaaaaooooooorrr//eeeee
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6788 Online, November 2020. Association for Computational Linguisties. doi:
http:////////....mmmmaeemmm 2020. emnep-mmpp-mmmmm main. 550
URL httpp:::/www..al.b...ooo//////// wtt.://wwww.........oo.ooo//////
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learn- ing Representationss 2020.
http////ppeeeeeee//e////e net forum 1d=thm?dd=dkkkkeeeeeka
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '20, pp. 39-48, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164, 3401075. URL https: http://ti..oggg//// http://...111333311
doi: 10.1145/3397271.
www.e1114333991 000112277.344175 3401075
Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guides Supervision for OpenQA
with ColBERT. Transactions of the Association for Computational Linguistics, 9:929 944, 09 2021. ISSN 2307-387X. doi: 10. 10.1162/tacl a 004055.
116./
tacl a 004005
Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. Sequential latent knowledge selection for knowledge-grounded dialogue. In International Conference Learning Representations, 2020.
URL BRRlhtttps:///pereviiee
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang. Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086 6096, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10.18653/v1//19-11 P19- 612
URL ht http::////aaaaahoooooo.ooooo. org/
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian

---

Published as a conference paper at ICLR 2022
Riedel, and Douwe Kiela. Retrieval-augmmentd generation for wwwllede-iittnssing NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp.
Curran Asso-
ciates, Inc., 2020. URL httpps///prrccccdgggg. neurips ips. cas.cc/peee ccc/paper/2022/F11ee
pdf
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. Learning to select knowledge for response generation in dialog systems. In IJCAI International Joint Conference on Artificial Intelligence, pp. 5081, 2019.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v 1/N5// 1098. URL
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P Processing, pp. 1797-1807, Brussels, Belgium, URL BttLL/tpee/////aanooooooooo///// 1LL18653/11/D1--11
2018. Association for Computational Linguistics. doi:
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@NIPS, 2016. URL paper9.pdF
17whooo1117771110 / 2016
Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D. Manning. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. In Alexa Prize Socialbot Grand Challenge 3 Special Journal, pp. 31-57, 2020.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v I. mawcl-main.200 wwwpo/ooooooorr//2222// naaci maaci-main. 200
URL
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamil- ton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I1th International Joint Conference on Natural Language Processing (Volume Online, August 2021. Association for Computational Linguis-
1: Long Papers), pp. tics. doi: 18.13 ao11........ URL https: https::/aalaanthllogee org.//o01..rg/222111111/ acl sol-long.519
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 809-819, New Orleans, Louisiana, URL
June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1000 httpp:://cce wtw.acaaantttoooooo antho wtw.ooy..ooooooooo..o........o.o
Ellen M. Voorhees. Overview of the TREC 2004 robust track. In Proceedings of the Thirteenth Text Retrieval Conference (TREC 2004), 2004. NIST Special Publication SP 500-261.
12

---

Published as a conference paper at ICLR 2022
Haolan Zhan, Lei Shen, Hongshen Chen, and Hainan Zhang. CoLV: A collaborative latent vari- able model for knowledge knowledge-grounded dialogue generation. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 2250-2261, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/022 2ttp3////221.mmmmmmamm.... emn 1p main. 172
URthttps: //acl wentthlooy..rgg//o/
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. Knowledge- grounded dialogue generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3377-3390, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/11/2020,
emnlp-main.272. UBL/nttps://ccaaatooooy..oo wttp://looo.ogg//2....ooo emnlp main. 272
Wen Zheng, Natasa Milic-Frayling. and Ke Zhou. Approximation of response knowledge
trieval in knowledge-grounded dialogue generation. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pp. 3581-3591, Online, November 2020. Association for Computational Linguistics. doi: 11.18653/11/2020 http:////////200../ninnn///////o findinng--emmpp.22..
URL https:
//aclanthology.oo wwwpetthoooooyo.g/222200000 fereoo.fiiniie findings emnlp. 321
Wen Zheng, Natasa Milic-Frayling, and Ke Zhou. Knowledge- gooligeeegrudedddddee dialogue generation with term-level de-noising. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pp. 2972-2983, Online, August 2021. Association for Computational Linguis- tics. doi:
findings-acl.262. http:tss: petttpss//acllantoloogge org/2021.
Findings-acc.2662
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. A dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 708-713, Brussels, Belgium, October-November 2018. Association for Compu- tational Linguistics. doi: 10.18653/v1/D18. 1076. URL https: /wtp:///ooo...nooooogo..... org/
D18-1076
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 IS IIGGIER GROUNDING PURELY DUE TO A BETTER RETRIEVER?
To test this hypothesis, we ran an ablation test, where we used the best retriever we had (from Hindsight, a = 0) and trained generator using Marginalized loss while keeping the retriever itself fixed. This simulates the situation where we improved retrieval independently and we want to test that is sufficient to explain increased grounding. We see in table that the better retriever ablation (Marg. Gen. with fixed ELBo Ret.) leads to increased grounding compared to the generator trained using MARGINALIZEDL.OM demonstrating that the increased grounding is also due to ELBoLOSS.
However, the ablation is worse than ELBo Gen. (a 0.25),
A.2 WITHOUT DISTRIBUTIONAL REPOSITIONING
ELBoLoss doesn't perform as well without distributional repositioning.
Table Additional Groundedness evaluation for Wizard Wikipedia: Models in descending order of groundedness. ELBoLOSS (a 0.25) has best tradeoff between groundedness and end-to-end evaluation.
Top-1
Max. of Top-5
Method
Fl
Nov-F1
F1
Nov-F1
ELBo Gen. (a 0)
21.48
21.18
35.04
35.27
ELBo Gen. (a = 0.25)
21.34
20.78
34.16
34.24
Marg. Gen. with fixed ELBo Ret.
20.17
19.28
31.08
31.09
Marg. Gen.
18.63
17.46
26.19
25.39
Table 6: Additional End-to-end evaluation for Wizard of Wikipedia (Novel-F1: discounts com- monly occurring words and context words (x), Knowledge-Fl: overlap of generated output with gold passage)
Top-1 N-F1
Max. of Top-5
Method
K-F1
F1
N-F1
K-FI
Marg.
18.79
10.45
12.61
26.52
16.42
16.02
ELBo (Aiget = a'gen = 0.25)
18.86
11.12
13.08
27.56
18.67
17.69
ELBo (Ayer agen 0)
18.41
11.03
12.93
27.04
18.13
17.61
Gen. Only
16.11
5.15
8.05
Table 7: Additional Relevance evaluation: ELBolLoss Retriever with = is better than 0.
Sampling passages with temperature helps with overfitting but still performs worse than a = 1. (MRR Mean Reciprocal Rank, Success @k both in percentages)
Wizard of Wikipedia S@5
Method
MRR
S@
S@ 10
Marg. Retriever
43.8
38.9
49,9
52.8
ELBo Retriever (a
49.0
41.1
58.8
63.9
BLBo Retriever (a = 0, temperature=4)
43.7
35.4
53.5
60.4
Gold-sup. Retriever
45.2
35.6
57.0
63.1
ELBo Posterior
78.5
72.4
86.0
88.4
14

---

Published as a conference paper at ICLR 2022
Table 8:
The ELBo posterior finds Egold with high success providing better supervision during training. (MRR = Mean Reciprocal Rank, Success@u both in percentages)
Wizard of Wikipedia
MS MARCO NLGen
Method
Dist.
MRR
S@1
S@5
S@ 10
MRR
S@
S@5
S@
10
Gold-sup.
(,(2)z)
45.2
35.6
57.0
63.1
28.9
19.5
40.4
47.7
Marg.
(z(z))
43.8
38.9
49.9
52.8
30.4
19.4
43.4
53,2
ELBo (a (2((2)
49.0
41.1
58.8
63.9
32.1
21.2
45.3
54.4
ELBo (a = 1) Q 2)
78.5
72.4
86.0
88.4
67.8
56.7
81.9
86.2
Passages retrieved by ELBoLoss retriever while talking about Italian Cuisine. They include passages about various ingredients (rank=2), cheeses (rank=4), dishes (rank=5) alongside more information about Bucatini Pasta (rank=1,3).
Table 9:
rank
text
1.0 Bucatini Abstract Bucatini, also known perciatelli, is thick spaghetti-lik pasta with hole running through the center. The name comes from meaning "hole", while "bucato" or its Nea...
2.0
Pasta con le sarde > Ingredients. The principal ingredients are olive oil, onions, pasta and finely chopped mixture of sardines and anchovy. Various types of pasta used for the dish, but b...
3.0
Bucatini Preparation. Standard pasta machines will roll out sheets of flat pasta which are then cut into ribbons to make flat, ribbon-style pasta like fettuccine, tagliatelle, or pappardelle. ...
4.0
Bocconcini Abstract This cheese is described by its Italian name, which means "small mouthfuls".
made in the "pasta filata" manner by dipping curds into hot whey, and kncading, pulling.... ...
5.0 Carbonara > Abstract Carbonara () is an Italian pasta dish from Rome made with egg, hard cheese, guanciale (or pancetta), and black pepper. The dish arrived at its modern form, with its current...
A.3 EXAMPLES OF RETRIEVED PASSAGES
A.3.1
CONVERSATION I: ITALIAN CUISINE
Other Ooh like that! Stick some nice spicy arrabbiata sauce with it, ahhhh! Have you ever had bucatini before?
Self Oh yeah! I love that spicy garlic and tomato sauce. No I have not had bucatini. Is that type cheese?
Other Now you're re speakin' my language. No no, it's a style of noodle, like a really long straw. Bucatini amatraciana is insanely good.
15

---

Published as a conference paper at ICLR 2022
Table 10: Passages retrieved by MARGINALIZEDDLOS retriever while talking about Italian Cuisine. All passages talk about Pastas.
rank
text
1.0
Bucatini Abstract Bucatini also known perciatelli thick spaghetti-like pasta with hole running through the center. The name comes from meaning "hole", while "bucato" or its Nea...
2.0
Bucatini > Preparation. Standard pasta machines will roll out sheets of flat pasta which are then cut into ribbons to make flat, ribbon-style pasta like fettuccine, tagliatelle, or pappardelle.
3.0 Rotini Abstract Rotini a type of helix- or workserew-shaped pasta. The name comes from 17th-century Italian word meaning "small wheels". Rotini is related to fusilli, but has a tighter he...
4.0 Vermicelli History.:The Americas. The "fideo" is type of noodle, produced in Eu- rope ever since the Roman times, best known as "fideus" or "fidelis", and then spread to Mexican and Latin Amer...
5.0 Rollatini > Abstract Rollatini (sometimes also spelled rolatini or rolletini) is an Italian- style dish (called "rollatini di melanzane" in faux Italian) that usually made with thin slices of ...
16

---

Published as a conference paper at ICLR 2022
A.3.2 CONVERSATION 2: ROCK AND ROLL
Self Do you mean Elvis Aaron Presley, the American singer and actor? Other That's the one. think his nickname was the king of rock 'n roll.
Self had just heard of him being "The King". There probably would not have been Sun Records if not for Elvis and Sam Phillips.
Other He was revolutionary for his time. Many older people thought he was straight from the devil.
Table 11: Passages retrieved by ELBoLoss while talking about Rock and Roll. Relevant passages about cultural impact of Elvis Presley (rank=2) and details about his career (rank=4) alongside introductary paragraphs of other musicians
rank
text
1.0 Sam Phillips Abstract Samuel Cornelius Phillips (January 5, 1923 July 30, 2003) was an American record producer who played an important role in the development of rock and roll during the 19...
2.0
Cultural impact of Elvis Presley > Abstract Since the beginning of his career, Elvis Presley has had an extensive cultural impact. According to "Rolling Stone", "it was Elvis who made rock 'n' r... Freddie King > Abstract Freddie King (September 3, 1934 - December 28, 1976) was American blues guitarist and singer. He recorded several hits for Federal Records in the carly 1960s. His soul...
3.0
4.0 Elvis Presley Abstract With series of successful network television appearances and chart-topping records, he became the leading figure of the newly popular sound rock and roll. His energ...
5.0 Elvis Presley Abstract Elvis Aaron Presley (January 8, 1935 - August 16, 1977), also known mononymously as Elvis, was an American singer, musician, and actor. Regarded as one of the most signi...
Table 12:
Passages retrieved by MARGINALIZDDLOSS while talking about Rock and Roll. All passages are the introductory paragraphs from various related artists
rank
text
1.0
Elvis Presley Abstract Elvis Aaron Presley (January 8, 1935 August 16, 1977), also known mononymously as Elvis, was an American singer, musician, and actor. Regarded as one of the most signi...
2.0
Sam Phillips > Abstract Samuel Cornelius Phillips (January 5, 1923 July 30. 2003) was an American record producer who played an important role in the development of rock and roll during the 19...
3.0
Johnny Otis Abstract Johnny Otis (born Ioannis Alexandres Veliotes; December 28, 1921 1221-Jame January 17, 2012) was American singer, musician, composer, arranger, ban- dleader, talent scout, disc j...
4.0 Carl Perkins Abstract Called "the King of Rockabilly". he was inducted into the Rock and Roll Hall of Fame, the Rockabilly Hall Fame, the Memphis Music Hall of Fame, and the Nashville Songw...
5.0 Chubby Checker > Abstract Chubby Checker (born Ernest Evans; October 3, 1941) is American rock 'n roll singer and dancer. He is widely known for popularising many dance styles including the t...
A.4 EXAMPLES OF GENERATED OUTPUTS
We look at conversation that illustrates the differences in grounding between MARGINALIZED- Loss generator and ELBoLoss generator. We see that the MARGINALIZEDL cinates in 4 out of 5 utterances and only retrieved passage content in 2 out of 5 utterances. On the other hand ELBoLoss generator doesn't hallucinate and uses retrieved passage content for all utterances.
generator hallu-
17

---

Published as a conference paper at ICLR 2022
Prior turns:
Self I'm still working on getting better proper nutrition (Cheetos get me) but moderate vigorous exercise feels great
Other Yes, we have a swimming pool so I've been able to get bit more exercise, while enjoying the cool water during this hot summer.
Utterances generated by MARGINALIZEDLOSS generator based on top 5 retrieved pas- sages (passage text omitted for brevity). Phrases in the utterance that are based on the retrieved passage are marked in bold. Hallucinated phrases are italicised.
Table 13:
passage title
text
Physical fitness > Abstract
That's great! I love swimming too. It's greal way to slay hydruted and prevent hypokinetic diseases.
Outline of Exercise > Abstract
That's great! I love swimming too. It's great way to keep your immune system healthy and prevent diseases like heart disease and diabetes.
Underwater hockey >
Abstract
That's great! I love swimming too. It's a great way to keep your body hydrated.
Exercise Trends > Abstract Callanetics > Abstract
That's great! I love swimming too. It's a great way to keep your body hydrated. I've never been pool before.
That's 's great! love swimming pools too. They re great for exercise, especially in the summer.
Table 14: Utterances generated ELBoLoss generator based on top 5 retrieved passages (passage text omitted for brevity). Phrases in the utterance that are based on the retrieved passage are marked in bold. Hallucinated phrases are italicised.
passage title
text
Physical fitmess
> Abstract
That's s great! I'm also dedicated to physical fitness, which is state of health and well-being. It's important to be healthy, resist hypokinetic diseases.
Exercise > Abstract
That's s awesome! I've been doing it for a long time. It's a great way to improve sirength, prevent aging, develop muscles and the cardiovascular system.
Bodybuilding Abstract
That's s awesome! I'm also into bodybuildn building, which is the of progressive resistance exercise to control and de- velop one's 's musculature for aesthetic purposes. That's great! I'm also into general fitness training, which is focused on broad goals of overall health and well- being.
General fitness training Abstract
Strength training > Abstract
That's awesome! I'm also into strength training, which type of physical exercise specializing in the use of resistance to induce muscular contraction.
A.5 NOVEL-F1
Rationale We conducted small experiment with the generated output on Wizard of Wikipedia dataset using top-8 retrieved passages. We removed the gold passage and computed overlap of the generated output with the target output. We consistently found (across models and passage ranks) the Fl overlap to be close to 15. This meant that by conditioning arbitrary passages the generator (likely by ignoring them altogether) is able to achieve around 80% of the Fl-overlap of the best performing models (~ 19 F1). This can be confounding factor for selecting models based high Fl overlap. A model that simply copies content from the input x can achieve high Fl-overlap but fail to using the retrieved passage generate the output. Removing commonly occurring words
18

---

Published as a conference paper at ICLR 2022
reduces it to FI, but removing words from input context reduces it further down to 4 F1. Thus we find Novel-F1 to be the cleanest measure of overlap as it discounts two confounding factors and only looks at "Novel" words, words that are rare and were not in the input text z..
We construct the list of common words based on their frequency in the training corpus. We sort words by frequency and take the most frequent words that contribute: 50% of the probability mass toward Wizard of Wikipedia utterances (amounting to 121 words) following [Shuster et al.] (202T). However, we found that using the same heuristic for MS-MARCO NLGen answers included num- bers and rarer tokens that could potentially be in the answer span. So we instead use only 33% of the probability mass (amounting 55 words). We also ran evaluation using 50% of the probability mass but found the trends to be consistent.
MS Marco NLGen list of common words (sorted by frequency)
is, of, in, to, and, for, or, are, that, on, from, as, by. you, with, it, county, can, at, per, was, your, average, cost, be, between, which, used, one, united, states, there, years, located, name, not, new, have, takes, number, has, means, days, when, blood, system, year, should, no, most, first, hours, up, minutes, 1
Wizard of Wikipedia list of common words (sorted by frequency)
is, of, in, to, and, for, or, are, that, on, from, as, by, you, with, it, county, can, at, per, was, your, average, cost, be, between, which, used, one, united, states, there, years, located, name, not, new, have, takes, number, has, means, days, when, blood, system, year, should, no, most, first, hours, up, minutes, 1 i, and, of, in, is, to, it, that, are, you, they, have, was, but, for, as, its, like, with, on, so, be, or, not, yes, do, can, from, there, by, well, also, one, my, know, has, some, he, their, love, most, people, think, really, all, about, just, too, them, im, which, sure, more, been, at, would, many, were, good, very, dont, when, thats, no, yeah, what, other, great, if, because, used, actually first, since, lot, me, even. your, how, we, time, different, world, use, get, called, only, out, much, over, had, though, music, around, popular, his, am, made, than, such, back, up, us, make, usually, who, favorite, new, food, oh, long, she, now, did, pretty, any, where, years, this, way, go
B INTUITION BEHIND IMPROVEMENTS DUE TO ELBoLoss
To understand the intuition behind suboptimality of MARGINALIZEDLOSS for open-ended gen- eration tasks consider the following: We would want a good retriever to assign similar but high probabilities to all context-relevant passages because they are similarly relevant but a good genera- tor to only assign high probabilities when using label-relevant passages because only label-relevant passages are pertinent to the target output. But the training signal to a model (partial derivative w.r.t the model and passage) is modulated by the probability of the other model:
d Po(v)
aP(yl=) = (2(22)
Since context-relevant passages have similar ((zj(z) the gradient encourages the generator to assign equal probabilities to the target output using all context-relevant passages. We see this issue play out empirically when using MARGINALIZEDLOSS for two different tasks: Open-Domaie QA (Natural Questions by Kwiatkowski et al. (019)) and informative dialogue (Wizard of Wikipedia by Dinan et al. (2099) (Figure 4). We see that the Natural Questions dataset, where there is typically one correct answer, the generator produces distribution with a sharp peak that can potentially serve as an accurate proxy for relevacee and in turn train good retriever. But on the Wizard of Wikipedia dataset, the generator produces a flatter distribution which is a bad proxy for label-relevance. This provides weaker supervision for the retriever which learns a flatter probability distribution as well and is less able to differentiate context-relevant from irrelevant passages.
We see in Figure 5 that for the Wizard of Wikipedia dataset with ELBoLoss we obtain sharp distribution for (2 y) (nearly as good as Po(y/z, 2) on NQ from Figure 4 and that the P, (z(a) and Po(ylar, =) are now sharper than MARGINALIZEDL.CM imply better retriever and generator (they may still assign high probability to the wrong passage), a flatter distribution limits their potential. As we will see in Section 4 ELBoLoss indeed utilizes the potential and trains better retriever and more grounded generator.
While a sharper distribution does not
19

---

Published as a conference paper at ICLR 2022
1.00
Cumulative
0.50
0.50
0.25
Natural Questions
0.25
Natural Questions Wikipedia
Wikipedia
0.00
rank
rank
Figure 4: With MARGINALIZEDLOSS, the generator (s(y)x, 2) learns sharp distribution for Nat- ural Questions (NQ) dataset (right) but learns flatter distribution for a one-to-many open-ended generation task using the Wizard of Wikipedia dataset (WoW). The flatter distribution in the of WoW Generator shows that it has not learned label-relevance as well. Consequently, for WoW weaker retriever (left) that has flatter distribution than NQ. (Left) Cumulative probability (=|æz) w.r.t. rank for passages. (Right) Assuming a uniform prior P(z[z), the cumulative proba- bility Po (y]l,,, w.r.t rank for passages, plotted P(=|æ, y) (() The gray dotted line shows hypothetical model that assigns equal probabilities to all passages.
1.00
1.00
Cumulative
0.75
0.50
0.50
Marginalized retriever ELBo prloorretriever
0.25
0.25
Marginalized Generator
ELBo pasteriar retriever
ELBo Generator
0.00
0.00
10
50
20
40
rank
rank
Figure 5: Analogous plots to Figure 4 but with ELBoLOSS on the one- one-to-many Wizard of Wikipedia (WoW) dataset. Training with ELBoLoss produces a sharp distribution for Q(2|21 y) and subse- quently sharper (2(0x) and Po (o(ulr, (z, z) than MARGINALIZEDL.OSS
20


Reviews:
[{"review_id": "uklxj3juEQY", "reviewer": "Reviewer_uUxg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of this paper is:\nThey novelly proposed to train a dedicated module, \"guide-retriever\", to predict the knowledge passage probability with extra information (target) and use that to guide (via KL divergence) the retriever. It seems to be a reasonable approach. \nThe evaluation and experiments are also relatively comprehensive.\n\nThe weakness is that this paper only compares their method with one baseline (MARGINALIZEDLOSS), but there're more baselines are designed for the similar tasks and should be analyzed: e.g., \"REALM: Retrieval-Augmented Language Model Pre-Training\" by Guu et al., and \"Retrieval-augmented generation for knowledge-intensive nlp tasks\" by Lewis et al.", "summary_of_the_paper": "This work focuses on the knowledge grounded text generation tasks. They argue that multiple passages can be valid and relevant to the context, but not all of them are observed/used in the target response. Therefore they propose that, during training, the target response should be utilized to train a \"guide-retriever\",  which predicts P(passage|context, response) and provides passage weight for the generator. A retriever is jointly trained with the generator and the guide-retriever, and the KL divergence between the retriever and the guide-retriever is included in the loss function.\n\nThe contribution of this paper is that they proposed a potential solution to the challenge that the model may be not effectively trained in case that multiple passages are valid but only one of few are used.", "summary_of_the_review": "This work proposes a novel approach to the knowledge grounded text generation tasks via the design of a new \"guide-retriever\".\nThe methodology is reasonable and the experiments show significant improvement over one baseline. However, this work can be improved if comparison with more baselines is provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hF5ULAL6vVv", "reviewer": "Reviewer_kJHS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The problem in this paper is important and underexplored --- despite recent advances in neural retrieval, knowledge-grounded open-ended generation is less studied compared to other problems like short answer generation. As the paper motivates, there are many reasons that prior work focusing on short answer generation are not suitable for open-ended generation.\n* The proposed method is novel, and shows very strong empirical results.\n\nWeaknesses:\n* One major concern is that the pipeline training baseline ---  the approach that trains the retriever and the generator separately --- is missing, although it is (arguably) the most widely-used approach and is a potentially stronger baseline. Pipeline training (Karpukhin et al, Khattab et al and more) has been much more widely used in recent retrieval work compared to joint training due to its simplicity, stability, and better empirical results (e.g., as shown in [1] which compares joint training and pipeline training under the same condition). Although I agree with the motivation for joint training in the paper, there is no justification for not adding a pipeline training baseline, especially given that the datasets come with annotated gold passage so it is pretty straightforward to train two models separately.\n* Three research questions for evaluation in Section 4 all look reasonable to me, but how the specific metric used in the paper measures those is not entirely clear to me. For example, is F1 / Nov-F1 between retrieved evidence and generated response a standard metric to measure the groundedness? If not, there should be some evaluation that shows that F1 / Nov-F1 correlates with human evaluation of the groundedness of the generated response.\n\nMinor comments / questions\n* Is “Success @ k” an existing term taken from prior work? This metric is identical to “Recall @ k” which is a standard, long-standing term. Unless “Success” is another widely-used term as an alias of “Recall” that I am unaware of, it should be replaced to “Recall”.\n* Is there a reason that ColBERT is pretrained on MS-MARCO or NQ? Has there been any ablation on the impact of this pretraining? Does any of findings in the paper change if the model is not pretrained on another dataset?\n\n[1] https://arxiv.org/pdf/2101.00408.pdf", "summary_of_the_paper": "The paper tackles a problem of knowledge-grounded open-ended generation and proposes a new model that is an extension of an end-to-end training of the retrieval and the generator. In particular, there is another model called “posterior-guide” that is jointly trained with the other two models using ELBo. Intuitively, this posterior-guide is similar to the retriever that scores the evidence, but conditioned on not only the question but also the response. It is used as a weight of each evidence in the generation in order to encourage the generator to ground more to the evidence that is more relevant to the response. The retriever is also trained to be close to the posterior-guide (minimizing KL-divergence). To my understanding, this is a very clever way to give supervision to the model when it is not easy to obtain distant supervision data (unlike in short answer generation where the gold evidence is easily obtained by whether the short answer is included in the evidence or not).\n\nExperiments are done on Wizard of Wikipedia and MS Marco NLGen. The model achieves significant improvements over the baseline retriever, based on three evaluation metrics: relevance, groundedness and generation quality.\n", "summary_of_the_review": "Overall a strong paper, tackling important and underexplored problem of knowledge-grounded open-ended generation, proposing novel objective, significant empirical improvements on two datasets in multiple metrics. There are a few concerns like absence of a potentially stronger baseline, justification of evaluation metrics, choice of terms, choice of implementation details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "fY4UlphaKGN", "reviewer": "Reviewer_hgu8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--------------\nStrengths\n--------------\n1. Improvement in performance over a baseline RAG model by use of the posterior for training the retriever   \n2. Experiments on aspects related to retrieval as well as response generation\n--------------\nWeakness\n---------------\n1. Limited evaluation - no experiments to study hallucination/memorization. This is a big weakness because BART is a powerful generation model. \n\n----------------\nQuestions:\n----------------\n1. The paper discusses using an alpha-mixture but doesn't clearly explain the need for the same. I'd imagine there's a risk of not seeing the ground-truth document when sampling from P_eta (because of a poorly trained prior network) and that is probably why alpha is set low initially, but what happens if you only sample from Q? Perhaps experiments that demonstrate the benefit of using an alpha-mixture over just using the posterior Q could be useful.\n2. What happens if the ground-truth document is never sampled; the KL divergence term will not add anything meaningful in that case? Is there a risk of posterior collapse? Do you do anything in your training to address this problem? This is also tied to my comment about hallucination; in the event that the model does not see the \"correct\" document while training the retrievers, it will force the generator to learn without grounding. \n3. There is contemporaneous unpublished work submitted with perhaps the exact same approach (with minor differences in how the approximation is done):  https://openreview.net/forum?id=vdTCHL1nyEq Were the authors aware of this work? \n4. Why was pre-training of the retrievers done differently? ", "summary_of_the_paper": "In this paper authors describe an approach to use the responses/ answers to guide retrieval during training of document grounded response generator. By having the retriever being trained using the posterior (p(document|response,context)), the model could learn a better (supervised) retrieval network (based on ColBERT) which could be used to guide the training of the prior ( p(document|context)).  Documents are retrieved using the dialog context and the top-k documents are used for training the posterior as well as the prior. Since the expectation for ELBOLoss cannot be computed exactly (due to the large document set),  it is computed by sampling documents from the top-r retrieved documents -- from either the posterior or the prior, guided by a parameter called alpha which governs the sampling proportion. The idea of doing variational training using a posterior network isn't particularly novel but the authors have made it work for open-ended response generation using this approximation for computing the ELBOLoss. The networks use BART for language generation and CoLBERT for modeling the retrievers. Experiments have been presented using the Wizard of Wikipedia dataset and the MSMARCO NLGEN. Experiments show an improvement over the baseline model (RAG - referred to as MarginalizedLoss) in both retrieval (success@k, MRR) as well as response generation (text-F1 overlap between response and grounded document and textF1 overlap between response and ground-truth output response). Overall this is a well written easy to read paper\n\n", "summary_of_the_review": "This is a simple and interesting idea which shows promise in performance. However, the experimental results are a little in weak -- in particular there is no study about hallucination/memorization in these models and some of the modeling choices aren't well studied.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ebNAy8Z6EqD", "reviewer": "Reviewer_eXap", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Main Review\n\n### Strengths\n\n- The authors indeed find improvement over a strong baseline (using a marginalized loss instead) in all three points across two unrelated datasets.\n- The proposed models are straightforward and not overly complex, with a proper diagram easily describing the training procedure, thus making the methods reproducible.\n- The findings are relevant and important particularly for those interested in what the authors describe as “one-to-many” generation tasks, where several plausible outputs can stem from a single input; the proposed model helps to narrow the “many” possible generations to the one most similar to the gold human response.\n- The submission is quite clear; methods (and the model architecture) are proposed clearly and concisely, providing the reader with appropriate background and technical knowledge to understand the model. The results are plainly displayed in tables in figures and are clearly described in the text.\n\n### Weaknesses\n- I think the authors failed to consider the model on an important subset of tasks that, in my opinion, would benefit most from this procedure: tasks without **gold retrieved passages available**. Though the authors compare to a “MarginalizedLoss” baseline, as it is used in some strong models in the literature, the real ablation to test is retrievers trained **_directly with the gold passage as target_**. WoW and MS-MARCO both provide such passages, and yet in the paper they are only used for evaluation of trained models. A more interesting task would be to apply these models to datasets without such provided targets.\n- Utilizing the target output to improve knowledge-grounded dialogue is not particularly novel in the literature:\n    - [1] proposes a system (TPPA) that learns to retrieve knowledge both from the input context and the response\n    - [2] utilizes the responses to inform the model where to pay attention to in the given knowledge, and thus can obtain models that ground more appropriately in the knowledge\n- The authors claim that a main contribution of the method is a model that produces generations more grounded in the retrieved passages than other methods. However, one could argue this naturally follows from the simply improving retrieval in the first place, in that the model will ground more often if it receives less loss from doing so, and it will receive less loss if it uses more relevant passages. So the overarching contribution here is improving retrieval through posterior-guided supervision, the effectiveness of which could be better supported if other situations were considered, see above..\n\n# Questions + Feedback\n### Questions\n1. Did you experiment with not updating the index, and instead just updating query encoders as in [3]? \n2. Why call your metric “success@10”, rather than “recall@10”? Are these fundamentally different metrics?\n3. Why were different pre-trained ColBERT models used for each task?\n4. In section 5, paragraph “Comparison to Fusion-in-Decoder”, you mention that “Fusion-in-Decoder is uniquely useful for QA style tasks”, however it is in fact used directly on the Wizard of Wikipedia task in [4] and achieves the best results reported there (better, e.g., than marginalized-loss methods such as RAG).\n\n### Minor Feedback\n1. Typo end of section 2: “we describe our method we separates those two concerns”\n2. Section 3, “Posterior as Retriever Paragraph”: “...and estimate the ELBoLoss ‘more’ accurately…”\n3. End of section 4.2: “... only use them for evaluation and validation and not for training nor for validation”, this sentence is unclear\n4. It would be nice to introduce “r.i.” as an abbreviation of “relative improvement” before it is introduced in section 4.4\n\n\n# References\n[1] Zheng et. al. 2020, Approximation of Response Knowledge Retrieval in Knowledge-grounded Dialogue Generation\n\n[2] Zheng et. al. 2021, Knowledge-Grounded Dialogue Generation with Term-level De-noising\n\n[3] Lewis et. al. 2021, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\n\n[4] Shuster et. al. 2021, Retrieval Augmentation Reduces Hallucination in Conversation\n", "summary_of_the_paper": "The paper tackles the problem of open-ended knowledge-grounded natural language generation, in the context of free-form QA or knowledge-grounded dialogue, where models must ground their generations on passages relevant to the input context. Specifically, the authors explore improving the retrieval component of retrieval-augmented systems by utilizing posterior signal from the label. A “guide retriever” learns the relevant passage to retrieve by including the target output in its input context, and then, using an ELBo loss, provides this signal to the normal, in-system retriever. The authors find that their model improves three-fold over baselines: 1) it retrieves relevant passages more frequently; 2) its generations are more grounded in retrieved passages; and 3) its generations are closer to human generations.\n", "summary_of_the_review": "The authors show improvements in two downstream tasks when improving retrieval via posterior-guided supervision of a retriever in a retrieval-augmented generation setup. However, the datasets considered are not explored to their maximum potential, with appropriate baselines missing from the paper (training the retrievers directly on the target passages); and, datasets where this solution appears to be most applicable are not considered at all (those without target passages). The paper indeed supports its claims, as mentioned in the summary, though some work is missing that could bolster the proposed efficacy of the method. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study the problem of open-ended knowledge-grounded natural language generation, in the context of free-form QA or knowledge-grounded dialogue, focusing on improving the retrieval component of the retrieval-augmented system. By retrieving more relevant passages, the generations are more grounded in retrieved passages. 

Pros:
+ The paper is clearly written and motivated.
+ Presents a straightforward approach that shows improvement over a  strong baseline.  
+ A strong paper focuses on a rather under-explored problem of knowledge-grounded open-ended generation, proposing novel objective, significant empirical improvements on two datasets in multiple metrics.
+ The authors included human evolution results to support their findings. 
+ The authors did a good job addressing several questions raised during review period and added several new experiment results and discussions to strengthen their findings. The reviewer team was generally satisfied.

Cons:

+ Several related work on knowledge guided dialog response generation is missing in the paper. Although the paper's focus is on retrieval based QA systems, the main focus is on open domain generation, which has overlaps with dialog response generation research. So the authors should cite the following papers in their paper: [1] Dinan, Emily, et al. "Wizard of wikipedia: Knowledge-powered conversational agents." arXiv preprint arXiv:1811.01241 (2018). [2] Zhou, Kangyan, Shrimai Prabhumoye, and Alan W. Black. "A dataset for document grounded conversations." arXiv preprint arXiv:1809.07358 (2018). [3]Zhan, Haolan, et al. "CoLV: A Collaborative Latent Variable Model for Knowledge-Grounded Dialogue Generation." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. [4]Zhao, Xueliang, et al. "Knowledge-grounded dialogue generation with pre-trained language models." arXiv preprint arXiv:2010.08824 (2020). 
+ There are several related work concerning with generation of a response given a relatively small set of evidence text such as the following ones:  [5]Lian, Rongzhong, et al. "Learning to select knowledge for response generation in dialog systems." arXiv preprint arXiv:1902.04911 (2019). [6]Kim, Byeongchang, Jaewoo Ahn, and Gunhee Kim. "Sequential latent knowledge selection for knowledge-grounded dialogue." arXiv preprint arXiv:2002.07510 (2020). Although these work do not include a retrieval part, the authors should cite and discuss similarities and differences to [5] & [6] in their paper.
Decision: Accept (Poster)