SlowGuess/NAACL_2021_OCR huggingface dataset:
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159-174.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Association for Computational Linguistics.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a. Pre-training via paraphrasing. Advances in Neural Information Processing Systems.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020b. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020c. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of Advances in Neural Information Processing Systems.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2020d. Question and answer test-train overlap in open-domain question answering datasets. arXiv preprint arXiv:2008.02637.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In Proceedings of the International Conference on Learning Representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Roektaschel, et al. 2020. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of Empirical Methods in Natural Language Processing.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, et al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of Empirical Methods in Natural Language Processing.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. In Transactions of the Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the Association for Computational Linguistics.
Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 21-29.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations.
Hai Wang and David McAllester. 2020. On-the-fly information retrieval augmentation for language models. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 114-119.
Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. In Proceedings of Empirical Methods in Natural Language Processing.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2018. Transfertransfo: A transfer learning approach for neural network based conversational agents. In NeurIPS CAI Workshop.
Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvi-jit Chattopadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra. 2019. Evalai: Towards better evaluation systems for ai agents. arXiv preprint arXiv:1902.03570.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9054-9065.
Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2020. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the Association for Computational Linguistics.
Chunting Zhou, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad. 2020. Detecting hallucinated content in conditional neural sequence generation. arXiv preprint arXiv:2011.02593.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.
A Appendices for "Hurdles to Progress in Long-form Question Answering"
A.1 Training & Model Details
All our models are developed and trained using TensorFlow 1.15 (Abadi et al., 2016) and Tensor2Tensor (Vaswani et al., 2018). Our implementations are based on the open-source codebases of REALM  $^{19}$  and the Routing Transformer. Similar to the REALM implementation, we use separate processes to run the retriever and generate training data (using a MIPS search). Since our retriever is frozen, we do not use the document index refresher available in their codebase.
Retriever: Our retriever is trained on 64 Google Cloud TPUs for a total of 4k steps and a batch size of 12288. We do early stopping on the validation data (with a smaller batch size of 512 due to smaller P100 GPU memory). Our model converges quite fast, reaching its best performance in 1.5k steps (in 43 minutes) and needing 103 minutes for the full set of 4k steps.
Generator: Our generator is trained on 64 Google Cloud TPUs, for a total of 100k steps on the ELI5 training set. We use the pg19_local_cluster8k configuration available in the Routing Transformer implementation. Besides the default hyperparameters, setting  $15\%$  input, attention and ReLU dropout was critical to prevent overfitting on the training set. We use a learning rate of 5e-5. Our retrievals, questions and answers are truncated / padded to 288 subword tokens (using the PG19 subword tokenizer). We use a minibatch size of 128 QA pairs, which corresponds to 332k tokens per mini-batch (of which, the loss is computed over the last 288 answer tokens, or 37k total tokens). We do not compute loss over padded tokens, and use special symbols to separate different parts of the input context. We reverse the retrieved paragraphs in context since the model uses local attention layers, and we wanted higher ranked retrievals to appear closer to the answer tokens. Our models take about 30 hours to finish 100k steps (0.92 steps / second).
Attention Maps: We show the 2D plots of our generator's attention maps in Figure 3.
(a) Local attention
(b) Routing attention
Figure 3: Figures (from Roy et al., 2020) showing 2-D attention schemes for the sparse attention mechanism used in Routing Transformer. Lower layers pool in local information via sliding window local attention (Sub-figure 3a) while upper layers gather global information for every token via clustering (Sub-figure 3b).
Hyperparameter Choices: We experimented with several different pretraining strategies (using Wikipedia), smaller model variants and hyperparameter choices manually in preliminary experiments. All these experiments performed quite poorly on ELI5, producing very short and sometimes incoherent responses. Finally, switching to a Routing Transformer model which was pretrained on a longform language modeling dataset (PG-19) significantly improved generation quality. Hyperparameters for this pretrained model (like hidden size / number of layers) were manually chosen with model capacity in mind. For our final experiments with this pretrained model we did not perform any hyperparameter search during training, primarily due to the expensive setup required to train the system. During inference, we tuned the nucleus sampling value from 0.0 to 1.0 in increments of 0.1, choosing the value with the best validation set performance. Our hyperparameter choices for contrastive learning on the retriever have been justified in an ablation study in Appendix A.2. Notably, we use very large minibatches of 12,288 to scale the number of negative examples. To train this model, we used the standard trick of data parallelism across 64 hardware accelerators. This resulted in an effective mini-batch size of 192 per chip, which is small enough to fit a BERT-base sized model on a TPU v3 chip's memory. To accumulate information across different chips before the final softmax, we used the tf.tpu.cross Replica_sum function (using an open-source wrapper found here).
$^{19}$ https://github.com/google-research/language/tree/master/language/realm $^{20}$ https://github.com/google-research/google-research/tree/master/routing_transformer
A.2 Ablation Study of C-REALM
One of our contributions is scaling up a distantly supervised objective for training retrievers on ELI5, originally described in Jernite (2020). This method uses in-batch negative sampling, making minibatch size a critical hyperparameter for better constrastive learning. We perform controlled experiments initializing our retrievers with REALM-CCNews (Guu et al., 2020) and varying batch size and keeping all other hyperparameters consistent. In Table 8, we notice a steady increase in performance as minibatch size is increased, with the largest gains coming by doubling the batch size in Jernite (2020) from 512 to 1024. Finally, in preliminary experiments we saw no benefit of more intelligent negative sampling schemes.
Batch size	R-Prec	Recall@5
REALM (pretrained)	6.6	14.9
256	6.2	11.0
512 (Jernite, 2020)	6.8	12.6
1024	11.5	21.0
12288 (Ours)	13.3	21.2
Next, we investigate the effect of initialization on the training of C-REALM. Unlike Jernite (2020) who initialize their model with BERT, before training we initialize our retriever with a pretrained self-supervised retriever. As a baseline, we initialize our model with ICT, a weaker self-supervised retriever introduced in Lee et al. (2019). Both models are trained with minibatch sizes of 12228. In Table 9, we notice a large improvement in performance when using a better initialization, confirming our design decisions.
A.3 Number of trainable parameters
In Table 10 we present the number of trainable parameters in our model compared to baselines on the leaderboard. Our generator is slightly larger than the models used in prior work, but we utilize a smaller retriever due to the shared query and candidate encoders in REALM. Overall, our system has a similar total number of parameters as baseline models like RAG and BART + DPR.
Table 8: The effect of minibatch size on the validation performance of c-REALM. As a baseline, we also add the retrieval performance of the REALM pretrained model which is used as an initialization.
Initialization	R-Prec.	R@5
REALM (pretrained)	6.6	14.9
ICT (Lee et al., 2019)	9.3	16.5
REALM (Guu et al., 2020)	13.3	21.2
Table 9: The effect of initialization on C-REALM. As a baseline, we also add the retrieval performance of the REALM-CCNews pretrained model without any fine-tuning on ELI5.
Model	Generator	Retriever	Index
T5-base	220M	-	-
BART	406M	-	-
RAG	406M	220M	15B
BART + DPR	406M	220M	15B
RT + C-REALM	486M	110M	15B
Table 10: The number of parameters used by our model and baselines. Our generator is slightly bigger than other submissions on the leaderboard, but we use a smaller retriever with a similar sized index.
A.4 Generations from our System
More generations have been provided (along with retrievals, highlighted to show  $n$ -gram overlap) in the supplementary material (data) as HTML files. We also present a few samples in Table 16.
A.5 Human Evaluation Setup
We conducted several A/B tests between variants of our model using human annotators. We asked a total of 20 participants for help who voluntarily agreed to help with the annotation process. Most participants were English-speaking graduate students in computer science. In every test, participants were shown a question along with two answers (generated by different systems) presented in a random order. They were then asked to choose which generation (1) answered the question better / which answer was more relevant to the question; (2) was more coherent / had less repetition; (3) was more factually correct. Since some annotators had a limited time, we asked them to prioritize question (1) over (2) / (3). Annotators were allowed to select "Tie" if they could not choose between the systems. We also permitted them to use search engines, but suggested restricting search to Wikipedia. We present all our results in Table 15. We also interviewed some participants after the annotation process and discuss our findings in Section 3.4. Note that while these A/B tests help us understand
which system is relatively better, they do not provide an absolute measure of performance (Celikyilmaz et al., 2020) — annotators reported that there were cases where both answers were very good and other cases where both were very poor. This is a limitation of A/B testing.
A.6 Effect of length on ROUGE-L
In this section we measure the effect of outputs lengths on ROUGE-L scores. To conduct this experiment, we truncate generations by our system to a fixed fraction of tokens across all instances. As we see in Table 11 in the Truncate column, shorter generations tend have lower ROUGE-L. To disentangle the effects of length and content, we also measure the generation quality by repeating the truncated generations several times until it matches the original generation length. In the Repeat  $1/f$  times column, we notice a gap between our model's original generation (24.4 ROUGE-L) and the equal-length truncated generations with repetition. These results indicate that while length helps improve ROUGE-L scores, simple repetition is insufficient.
Fraction f	# Tokens	Truncate	Repeat 1/f times
0.1	18.2	17.4	18.2
0.2	37.0	20.8	21.1
0.3	55.7	22.2	22.4
0.4	74.4	22.9	23.1
0.5	93.4	23.4	23.6
0.6	112.0	23.9	23.9
0.8	149.4	24.2	24.3
1.0	187.3	24.4	24.4
Table 11: Effect of truncating generations (Truncate) from the  $p = 0.6$  model to keep the first  $f$  fraction of tokens, and then repeating the truncated generations  $1/f$  times to match the original length (Repeat ...). Notice a consistent increase in ROUGE-L with longer outputs, but a gap between the original generations (24.4) and equal-length generations formed by repeating truncations (Repeat  $1/f$  times column).
A.7 More experiments on measuring retrieval grounding of generations
In this section we provide some more experiments testing the grounding of generations in retrieved documents. Overall, trends are consistent with our observations in Section 3.1.
Scatter plots between generation quality and unigram overlap with retrievals: We present this scatter plot in Figure 4. There is virtually
Figure 4: Scatter plot for generations from the  $p = 0.6$  model between generative quality (ROUGE-L vs reference on X-axis) and grounding with retrieval (unigram overlap with retrieved documents on Y-axis). The plot shows no correlation between the two quantities.
no correlation between the two quantities, with Spearman  $\rho = 0.09$ .
Instances with correct predicted retrieval: In Table 12, we present results similar to Section 3.1 considering only those instances where at least one retrieved document matched the gold annotation (roughly  $23\%$  instances). We also present a scatter plot on the same set of instances in Figure 5 and note a low correlation of  $\rho = 0.13$ .
R-L	vs predicted retr.	vs random retr.
1-g	2-g	1-g	2-g
p = 0.6, correct retrieval examples
Predicted	23.74	54.4	10.0	39.7	4.3
Random	23.91	52.5	9.6	38.8	4.0
p = 0.9, correct retrieval examples
Predicted	22.40	54.9	9.2	40.9	4.3
Random	22.22	54.7	9.2	41.1	4.2
Table 12: Comparison of generations conditioned on retrievals from C-REALM (Predicted) and randomly chosen retrievals (Random), for those cases where C-REALM predicted the correct retrieval. Notice very small differences in generation quality (R-L) as well as the fraction of  $n$ -grams ( $n-g$ ) in the generation overlapping with retrievals predicted by C-REALM (vs predicted retr.). To control for overlap due to stopwords, we also add  $n$ -gram overlaps with the randomly sampled retrievals.
Experiments with  $p = 0.9$ : We conduct additional experiments studying our model variant with
Figure 5: Scatter plot for generations from the  $p = 0.6$  model between generative quality (ROUGE-L vs reference on X-axis) and grounding with retrieval (unigram overlap with retrieved documents on Y-axis). Unlike Figure 4, this plot only considers those cases where C-REALM predicted the correct retrieval. The plot shows very little correlation between the two quantities (Spearman  $\rho = 0.13$ ).
R-L	vs predicted retr.	vs random retr.
1-g	2-g	1-g	2-g
Predicted	22.62	53.9	8.7	40.7	4.1
Random	22.56	53.1	8.4	40.7	4.1
Gold Ans	-	54.1	9.1	40.2	3.8
higher nucleus sampling values. As we saw in Section 2.3, these generations tend to be more fluent and coherent, but less relevant to the question. In Table 13 and Table 14 we find consistent trends as Section 3.1, with very little difference between models conditioned on retrievals from C-REALM and random retrievals.
Table 13: Comparison of generations (with  $p = 0.9$ ) conditioned on retrievals from c-REALM (Predicted) and randomly chosen retrievals (Random). Notice very small differences in: (1) ROUGE-L vs gold answers (R-L); (2)  $n$ -gram overlap ( $n$ -g) with retrievals predicted by c-REALM (vs predicted retr.). Gold answers also have a similar overlap with predicted retrievals. To control for overlap due to stopwords, we also add  $n$ -gram overlaps with the randomly sampled retrievals.
vs qn.	vs predicted retr. but not in qn.	vs random retr. but not in qn.
(lemmatized nouns, proper nouns, numbers only)
Predicted	9.1%	32.4%	12.0%
Random	9.4%	30.2%	12.3%
Gold Ans	8.3%	28.8%	15.1%
Table 14: A fine-grained version of Table 13 measuring the unigram overlap of nouns/numbers in the generations with the input question (vs qn.), retrievals predicted by C-REALM (vs predicted retr.) and randomly sampled retrievals (vs random retr.). Similar to Table 13, notice very little difference with and without retrieval.
A	B	Question	Prefer A	Prefer B	Tie
Experiment 1: A comparison between nucleus sampling p values (0.6, 0.9), conditioning on predicted retrievals (pred.).
Result: Lower entropy more relevant to question, but higher entropy more coherent and lesser repetition.
p = 0.6, pred.	p = 0.9, pred.	Which generation answers the question better?	41% (65)	30% (48)	29% (46)
Which answer is more coherent?	27% (42)	50% (79)	23% (37)
Which ans. is more factually correct + sensual?	30% (47)	37% (58)	33% (52)
Experiment 2: A comparison between generations conditioned on predicted (pred.) and random retrievals (rand.).
Result: Little difference in generation quality / coherence / relevance to question, high amounts of tie.
p = 0.6, pred.	p = 0.6, rand.	Which generation answers the question better?	40% (78)	33% (64)	27% (51)
Which answer is more coherent?**	55% (12)	27% (6)	18% (4)
Which ans. is more factually correct...**	48% (10)	9% (2)	43% (9)
p = 0.9, pred.	p = 0.9, rand.	Which generation answers the question better?	31% (52)	37% (63)	32% (54)
Which answer is more coherent?	32% (26)	36% (30)	32% (26)
Which ans. is more factually correct + sensual?	28% (23)	35% (29)	37% (30)
Experiment 3: A comparison between generations conditioned on predicted retrievals (pred.) and the longest gold answer.
Result: Strong preference for gold answers over generations.
p = 0.6, pred.	gold answer	Which generation answers the question better?	14% (29)	68% (138)	18% (36)
Which answer is more coherent?	7% (8)	71% (77)	21% (23)
Which ans. is more factually correct + sensual?	2% (2)	76% (65)	22% (19)
p = 0.9, pred.	gold answer	Which generation answers the question better?	17% (49)	72% (203)	11% (31)
Which answer is more coherent?	13% (14)	61% (65)	25% (27)
Which ans. is more factually correct + sensual?	6% (6)	72% (78)	22% (24)
Table 15: Human evaluations experiments with exact number of ratings shown in  $(\cdot)$ . Differences greater than  $10\%$  with more than 50 total samples have been bold marked. The experiments marked with \*\* have less than 50 samples, so it is difficult to draw meaningful conclusions.
Q: What causes the trail behind jets at high altitude?
A: The same thing that causes the clouds you see at the start of a storm to form in the first place. The air pressure changes as the jet moves through the air, causing it to cool down and condense into water droplets. These droplets are in fact a lot bigger than the water droplets in the clouds, so when the plane passes through them, they can cause the air pressure to change, causing the water droplets to condense into clouds. This is what you see when you look at a cloud formation from the ground. In addition to this, when the plane passes through the clouds, it also moves through the air, causing it to change its temperature, causing the water droplets to condense and form clouds. These clouds are often quite large, and can block out the sun. This is what you see when you look at a contrail. Finally, the water vapor that forms at high altitudes is cooled by the air above, and as it cools, it condenses into clouds. This is what you see when you look at a cumulonimbus cloud. These clouds are even larger than the water droplets in the clouds, and they block out the sun.
Q: How do shampoo and conditioner work?