SlowGuess/NAACL_2021_OCR huggingface dataset:
Kalpesh Krishna\*
Aurko Roy
Mohit Iyyer
$\spadesuit$ University of Massachusetts Amherst, Google Research
{kalpesh,miyyer}@cs.umass.edu
aurkor@google.com
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train/ validation overlap, as at least  $81\%$  of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.
Long-form question answering (LFQA) integrates the retrieval component of open-domain QA, which involves searching a large external knowledge source for documents relevant to a given question, with a text generation component to produce paragraph-length answers. Significant progress has been made on open-domain QA datasets such as Natural Questions (Kwiatkowski et al., 2019),
whose questions are answerable with short phrases and entities, by leveraging dense retrieval techniques like ORQA (Lee et al., 2019), REALM (Guu et al., 2020), and DPR (Karpukhin et al., 2020; Lewis et al., 2020c; Izacard and Grave, 2020). Methods inspired by these results have recently been combined with pretrained language models (Lewis et al., 2020b; Petroni et al., 2020) and applied to the Reddit-derived "Explain Like I'm Five" (ELI5) dataset (Fan et al., 2019), which is the only publicly-available large-scale LFQA dataset.
The recently proposed KILT benchmark (Petroni et al., 2020), which compares retrieval-augmented models across a variety of knowledge-intensive tasks including ELI5, automatically evaluates LFQA models by the quality of both generated answers (ROUGE-L against reference answers) and retrieved documents (R-precision against human-annotated relevant documents). In this paper, we build a state-of-the-art system for ELI5 by using a sparse Transformer variant (Roy et al., 2020) to condition over Wikipedia paragraphs returned by a REALM-style retriever (Guu et al., 2020).
However, despite its success on the KILT leaderboard, our system does not actually use the documents that it retrieves! To measure the effect of retrieval on generation quality, we design a control experiment in which retrieved documents are replaced with randomly-sampled documents at inference time. Results from both human A/B tests and automatic metrics like ROUGE-L demonstrate that conditioning on random documents has almost no effect on generated answer quality (Figure 1c). We recommend that future LFQA research report the results of such control experiments in addition to reporting generation and retrieval quality.
How can a system using random retrieval per
* Work done during an internship at Google Research.  
<sup>1</sup>Resources accompanying our paper can be found in https://github.com/martiansideofthemoon/hurdles-longform-qa
${}^{2}$  State-of-the-art as of April 3,2021-the "Google Research & UMass Amherst" team entry on https://evalai.cloudcv.org/web/challenges/challenge-page/689/leaderboard/1908
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940-4957 June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) Many held-out questions are paraphrased in the training set. Best answer to similar train questions gets 27.4 ROUGE-L
(b) Simply retrieving answers to random unrelated training questions yields relatively high ROUGE-L, while actual gold answers underperform generations
(c) Conditioning answer generation on random documents instead of relevant ones does not measurably impact its factual correctness. Longer outputs get higher ROUGE-L
(d) Annotators find it difficult to judge long answers (with repetition) & correctness of technical content
Val Q: Can you protect electronics from EMPs/solar flares? If so, how?
Train Q1: How does an EMP ruin electronics? What does it do? How would they be fixed? Can It be protected against? How?
Train Q2: If Earth were hit with a massive EMP, would all of our currently technology be completely unusable permanently?
Train Q3: Whenever a electromagnetic pulse (EMP) is released what does it do to electronics to disable them?
Train Q4: If earth was hit with an EMP, could we ever restore electricity? If not, why?
Train Q5: What are solar flares and why does it impact our electronics?
Train Q6. When an EMP goes off, can the electronics affected be replaced?
Random Train Ans, 19.4 ROUGE-L
The fast lane/slow lane is a bit of a misnomer. It gives the impression that new, faster lanes are being built. In reality, normal speed will be...
Gold Answer, 18.6 ROUGE-L
I'll start with the grounding question, because that's the easiest to answer: Doesn't help a bit. All that matters is that the metal container is conductive and doesn't have gaps...completely seal your Faraday cage. Consider soldering the lid on to that paint can... look at little baggie it comes in. Sealed mylar. That protected that chip from air travel at 35,000 feet, land travel through rural, urban, and suburban areas, and all the electromagnetic radiation that the trip entails... No lead shielding. No safes....
Generation using predicted retrievals, 19.0 ROUGE-L
Yes, you can shield them. But it's a slow process... Also, the equipment that's powered by them is a lot more expensive than you'd think, so it's hard to make sure that you're not just shielding them from your remote control.
Generation using random retrievals, 24.8 ROUGE-L
Yes, you absolutely can, in fact you can build a Faraday cage around your electronics, and protect them from solar flares... This is what is done with the Faraday cage around your electronics, which is the problem. The reason it is expensive is because it requires a huge amount of power and is expensive to replace... designed to shield your electronics from solar flares, you will have to pay for the protection. This is because you have to buy a piece of equipment that is designed to shield your electronics from solar flares, and that is expensive. ... This is also expensive, but not as expensive as the protection you need to shield your electronics from solar flares... designed to be as cheap as possible...
Figure 1: A summary of the major hurdles (a-d) to progress in long-form question answering with ELI5.
form well on ELI5? Our analysis reveals that this result is partially due to significant train / validation overlap in the ELI5 dataset (Figure 1a), which eliminates the need for external retrieval. A human study shows that at least  $81\%$  of validation questions have a paraphrase in the training set, and almost all validation questions are topically similar to a training set question. While Fan et al. (2019) attempted to identify and remove question overlap using TF-IDF similarity, more complex semantic matching methods & human verification is needed to address this issue in future LFQA datasets.
Digging deeper, we identify fundamental issues with using ROUGE-L to evaluate generated answer quality (Figure 1b). Simple baselines such as just repeatedly copying the question, or choosing a random training set answer, can outperform LFQA systems such as RAG (Lewis et al., 2020c) in terms of ROUGE-L. On the other hand, our system achieves higher ROUGE-L than reference human-written answers, which is misleading since human A/B testers strongly prefer reference answers to our system's. We conclude that ROUGE-L is not a reliable metric to evaluate LFQA due to its large and relatively unconstrained output space (e.g., compared to translation or summarization), and we offer suggestions for better automatic & human evaluations to enable meaningful progress on this task.
2 A state-of-the-art LFQA system
The ELI5 task (Fan et al., 2019) asks models to generate paragraph-length answers to open-ended questions in English that often rely on world knowledge (e.g., how do jellyfish function without brains or nervous systems?). LFQA systems thus benefit from conditioning answer generation on relevant documents from the web (such as the Wikipedia article about jellyfish). While large-scale pretrained language models store surprising amounts of world knowledge within their parameters (Petroni et al., 2019; Roberts et al., 2020), external document retrieval not only augments this intrinsic knowledge but also grounds model outputs in a knowledge source, which provides interpretability.
In this section, we describe our proposed LFQA system, which conditions answer generation on Wikipedia articles identified by a pretrained retriever. We use a dense retriever trained by scaling up a distantly supervised algorithm from Jernite (2020). Since retrieved articles can be quite long and often exceed the maximum sequence length of pretrained models like BERT (Devlin et al., 2019), we use a sparse-attention variant of the Transformer to allow modeling over longer sequences. While our system sets a new state-of-the-art on ELI5, we question the significance of this result in Section 3.
2.1 Retriever
We begin by specifying our dense retriever ("contrastive REALM" or c-REALM), which returns documents related to an input question. Consider a corpus of long-form questions and answers, represented by  $(q_{i},a_{i})_{i = 1}^{N}$ . Our retriever uses  $q_{i}$  as a query to retrieve  $K$  documents  $(r_{i,j})_{j = 1}^{K}$  from a knowledge corpus (Wikipedia), which is enabled by an encoder network that projects both questions and candidate documents to a 128- $d$  shared embedding space. Like REALM (Guu et al., 2020), our encoder is a BERT-base Transformer (Devlin et al., 2019) with a final projection layer.
Since the ELI5 dataset does not include gold retrievals, we train our retriever by scaling up a method recently introduced by Jernite (2020) that uses gold answers for distant supervision. The key idea is to push the encoded vector for a question close to a vector representation of its groundtruth answer(s), but away from all other answer vectors in the mini-batch (negative examples). Intuitively, this method works because both ELI5 answers and external documents are of paragraph length (documents are paragraph-length chunks from Wikipedia). Concretely, we optimize the loss,
$$
\log = - \sum_ {(q _ {i}, a _ {i}) \in B} \log \frac {\exp \mathbf {q} _ {i} \cdot \mathbf {a} _ {i}}{\sum_ {a _ {j} \in B} \exp \mathbf {q} _ {i} \cdot \mathbf {a} _ {j}}
$$
where  $B$  is the mini-batch and  $\mathbf{q}_i$ ,  $\mathbf{a}_i$  are the encoded vector representations for  $(q_i, a_i)$ . This objective is based on contrastive learning, a method that has been used effectively for semi-supervised learning (Chen et al., 2020) and dense retriever training (Karpukhin et al., 2020). Scaling up from Jernite (2020), who used a mini-batch size of 512 and initialized their retriever with BERT, we use much large mini-batches of size 12,288 (and hence, many more negative examples) and initialize our retriever with a strong pretrained retriever, the REALM model (Guu et al., 2020) trained on the Common Crawl News (CC-News) corpus. These design decisions greatly improve retriever quality, as we observe in an ablation study (see Appendix A.2). During inference, we perform a maximum inner-product search (MIPS) with the ScaNN library (Guo et al., 2020) to efficiently find the top  $K$  documents. In all our experiments we use  $K = 7$ , following the setup in Guu et al. (2020).
2.2 Generator
We next describe our generator model, which conditions its generated answers on retrieved documents returned by c-REALM. We use the Routing Transformer (RT) from Roy et al. (2020), which is the current state-of-the-art in long-form language modeling. The RT is a sparse attention model that employs local attention as well as mini-batch  $k$ -means clustering to better model long-range dependencies in sequences (attention maps in Appendix A.1). Long-form language models such as RT are well-suited to ELI5 as the task requires conditioning answer generation not only on a short question but also many lengthy retrieved documents.
We pretrain our RT model on PG-19, a long-form language modeling benchmark (Rae et al., 2020) created from approximately 28,000 Project Gutenberg books published before 1919. PG-19 has 1.9B tokens and an average context size of 69K words. While this data is out-of-domain for ELI5, we choose it to encourage long & coherent generation. Our RT is a 22-layer model with 1032 hidden units (486M parameters), maximum sequence length of 8192 tokens, and a vocabulary of 98K subwords. We fine-tune our model in a decoder-only fashion (Liu et al., 2018; Wolf et al., 2018) by concatenating the top  $K$  retrieved documents to the question  $[r_{i,K}, r_{i,K-1} \dots r_{i,1}, q_i, a_i]$  and training the model to predict tokens of the answer  $a_i$ . We do not backpropagate gradients through the retriever. Retrievals slightly improve perplexity (18.1 vs 17.8) as seen in Wang and McAllester (2020), but do not improve generations ( $\S 3.1$ ).
2.3 Main Experiments
Dataset & Evaluation details: We evaluate our model on the KILT validation & test subsets of ELI5 (Petroni et al., 2020), since the original ELI5 dataset does not have human annotations to measure retriever performance. We downloaded the ELI5 dataset (Fan et al., 2019) from the KILT Github repository. This version of the dataset has 272,634 training examples, 1,507 validation examples and 600 test examples. The test set answers
3Our hyperparameters have been chosen manually with minimal tuning. See Appendix A.1 for details.
4We tried training the retriever jointly with RT using the attention bias scheme proposed in MARGE (Lewis et al., 2020a). This improved perplexity only in autoencoding settings where the gold answer itself is used as a retrieval query (like the setup in Lewis et al., 2020a), which is not valid in LFQA.
5github.com/facebookresearch/KILT
Model	Retrieval	Generation
RPr.	R@5	F1	R-L	KRL
T5-base	0.0	0.0	16.1	19.1	0.0
BART	0.0	0.0	19.2	20.6	0.0
RAG	11.0	22.9	14.5	14.1	1.7
BART + DPR	10.7	26.9	17.9	17.4	1.9
p = 0.9
RT + REALM	6.7	15.5	25.1	21.5	1.4
RT + c-REALM	10.2	24.4	25.4	21.5	2.1
p = 0.6
RT + REALM	6.7	15.7	23.1	23.4	1.5
RT + c-REALM	10.7	24.6	22.9	23.2	2.4
Table 1: Results on the KILT test set for ELI5 for (1) retrieval performance, using R-precision and Recall@5 (RPrec, R@5), and (2) generation quality, using ROUGE-L (R-L). These scores are combined to produce the final metric KILT R-L (KRL). We outperform prior work on both generation & combined scores.
are hidden, and hosted on a public leaderboard in the EvalAI platform (Yadav et al., 2019).
Answer quality is measured by the maximum overlap of generations with a set of gold answers in terms of unigram F1 score and ROUGE-L (Lin, 2004). Petroni et al. (2020) collected human annotations of Wikipedia articles which support ELI5 gold answers, which enables measuring retrieval quality by computing R-precision (if the top-1 retrieval matches the annotation) and Recall@5 using the top-5 retrievals. Finally, the KILT benchmark combines R-prec. and ROUGE-L to measure the overall performance of the system by "KILT ROUGE-L". This metric is similar to ROUGE-L, but assigns a score of 0 whenever the top-1 retrieval does not match the gold annotation.
Baselines: We compare our model with the other entries on the ELI5 KILT leaderboard which are either generation-only, like T5-base (Raffel et al., 2020) and BART (Lewis et al., 2020b), or variants of BART using retrieval such as RAG (Lewis et al., 2020c) and BART + DPR (Petroni et al., 2020). These systems are based on massive pretrained language models, with similar number of parameters as our model (details in Appendix A.3).
Results: Table 1 contains our results on the test set of the ELI5 (also on the public KILT leaderboard). We present four variants of our system, using a different retriever during inference (REALM or C-REALM), and different nucleus sampling  $p$  values (Holtzman et al., 2020). All variants outper
Q: Why are almost all boats white?
A: Boats are generally painted white for obvious reasons, as they are inexpensive and easy to clean. Ships are also often painted to provide a visible visual contrast to the sea. Boats also often have high visibility ports to allow for easy international trade. In addition, many boats have an air-breathing function that makes them easier to spot and stay afloat, which is a good thing if you’re a pirate or trying to get into some country’s waters. Lastly, many ships are built in the last few decades, so they are not as exotic looking as they were thirty years ago. And lastly, many ships are made out of materials that are easy to work with, and so many of them are painted white as a practical matter. There are many, many more reasons for why ships are white, but those are the ones I know of. I’m sure there are many other reasons, but those are the ones I know of. Edit: I forgot to mention that the other answers about visibility and easy to see ports are also true. Ships are painted to appear to be larger, and more expensive. That makes them appear more impressive, and so they are more likely to be bought by people who want to show off their wealth and status.
Figure 2: Example generation from our LFQA system with  $p = 0.9$ . Generations are long & coherent, but suffer from repetition towards the end. (more in Appendix A.4 and attached data supplementary material).
form prior work in generation quality, with lower-entropy models ( $p = 0.6$ ) performing best. $^{6}$  C-REALM performs competitively to RAG and DPR despite being only distantly supervised, and outperforms REALM. Our proposed RT+C-REALM system achieves a new state-of-the-art on combined performance (KILT R-L). Generations from our model are provided in Figure 2 and Appendix A.4.
In this section, we conduct a thorough analysis of our model's usage of retrievals (Section 3.1), the impact of overlap in ELI5's train / validation / test folds (Section 3.2), issues with ROUGE-L and performance bounds (Section 3.3), and the difficulty in human evaluation for this task (Section 3.4). At the end of each section, we provide short takeaways with suggestions for future work.
3.1 Are generations grounded in retrieval?
While our retrieval-augmented system achieves state-of-the-art performance, we find little evidence that it is actually using the retrieved documents. To measure this, we run an ablation study where at inference time we replace retrieved paragraphs with
As in Holtzman et al. (2020), a human study reveals that higher entropy ( $p = 0.9$ ) answers are slightly more coherent and sensible, but lower entropy answers ( $p = 0.6$ ) are more relevant to the question (details in Appendix A.5).
R-L	vs predicted retr.	vs random retr.
1-g	2-g	1-g	2-g
Predicted	24.42	52.3	9.0	38.8	3.9
Random	24.20	51.2	8.5	38.5	3.9
Gold Ans	-	54.1	9.1	40.2	3.8
Table 2: Comparison of generations (with  $p = 0.6$ ) conditioned on predicted retrievals (Predicted) and randomly chosen retrievals (Random). Notice small differences in: (1) ROUGE-L vs gold answers (R-L); (2)  $n$ -gram overlap ( $n$ -g) with predicted retrievals (vs predicted retr.). Gold answers also have a similar overlap with predicted retrievals. To control for stopwords, we show overlaps with the random retrievals.
A	B	Prefer A	Prefer B	Tie
For p = 0.6
pred.	random	40% (78)	33% (64)	27% (51)
pred.	gold ans.	14% (29)	68% (138)	18% (36)
For p = 0.9
pred.	random	31% (52)	37% (63)	32% (54)
pred.	gold ans.	17% (49)	72% (203)	11% (31)
randomly sampled paragraphs from Wikipedia. We compare this Random baseline with our original system (Predicted) in terms of generation quality as well as the  $n$ -gram overlap between the generation and the retrieved paragraphs.
Generations are similar irrespective of type of retrievals: We present our results in Table 2. Despite not being conditioned on any meaningful retrievals, the Random retrieval model has similar ROUGE-L scores as our Predicted system. Moreover, generations from the Random and Predicted models have similar amounts of 1-gram and 2-gram overlap with the paragraphs retrieved by C-REALM, despite the fact that the Random model does not actually see the retrieved paragraphs. $^7$
The  $n$ -gram overlaps are possibly overestimates due to stopwords (e.g., prepositions, punctuation) and entities which are copied from the question.
Table 3: Human evaluation results with exact number of ratings shown in  $(\cdot)$ . Annotators are shown a question along with two answers (A, B) in random order and ask them to choose one (details in Appendix A.5). For both model variants  $(p = 0.6, 0.9)$ , we see (1) little difference between generations conditioned on predicted (pred.) or random (rand.) retrievals; (2) strong preference for gold answers over generations.
vs qn.	vs predicted retr. but not in qn.	vs random retr. but not in qn.
(lemmatized nouns, proper nouns, numbers only)
Predicted	13.4%	34.4%	11.9%
Random	13.7%	31.7%	12.1%
Gold Ans	8.3%	28.8%	15.1%
Table 4: A fine-grained version of Table 2 measuring the unigram overlap of nouns/numbers in the generations with the input question (vs qn.), retrievals predicted by c-REALM (vs predicted retr.) and randomly sampled retrievals (vs random retr.). Similar to Table 2, notice very little difference with and without retrieval.
To tackle this issue, in Table 4 we measure the fractions of lemmatized nouns, proper nouns and numbers in the generated answer which are present in the predicted retrievals but not in the question. We notice similar trends as before, with only small differences between the two systems. Finally, there is almost no correlation (Spearman  $\rho = 0.09$ ) between the Predicted model's generation quality and the amount of unigram overlap between its outputs and the retrieved documents (scatter plots in Appendix A.7), strengthening our hypothesis that generations are not grounded in retrievals. $^{8}$
Human evaluation validates our findings: As ROUGE-L and  $n$ -gram overlap have major limitations for LFQA (Section 3.3), we perform additional human A/B testing on the output of Random and Predicted. Specifically, we ask human volunteers to choose between answers generated by the two systems (presented in random order). As seen in Table 3, humans struggle to choose which of the two answers is more relevant to the question. For both model variants ( $p = 0.6$ , 0.9), there is a less than  $7\%$  preference for a particular answer type, with humans preferring answers (by  $6\%$ ) from the Random model for  $p = 0.9$ .
Other systems also have this issue, possibly due to source-reference divergence and train-validation overlap: We note that this issue is not unique to our system — other systems on the KILT leaderboard like BART + DPR and RAG actually perform worse than their no-retrieval counterpart (BART) in generation quality, as
7Corresponding experiments with the  $p = 0.9$  variant of our model are presented in Appendix A.7.
8 All these trends persist even on questions for which our retriever predicts the ground-truth document (Appendix A.7)
<sup>9</sup>Details of our experimental setup in Appendix A.5.