SlowGuess/NAACL_2022_OCR huggingface dataset:
SBERT (Reimers and Gurevych, 2021): SBERT is a sentence-BERT-based encoder model with down-project layers that outperforms DPR on the MS MARCO dataset.
<sup>6</sup>https://github.com/dorianbrown/rank_bm25
We report results from baseline retrievers and our trained models in Table 4. We also conduct a human evaluation to compare retrieved examples from EGRET (L) to examples generated by the state-of-the-art LFQA Routing Transformer model of Krishna et al. (2021).
While all models outperform random baselines, our trained EGRET models (Fan et al., 2019) consistently outperform both the lexical baseline (BM25) and other neural baselines. Among the baseline models, SBERT model, trained on MSMARCO dataset, consistently outperforms other models and DPR model lags behind the lexical baseline. $^7$
Including context after the exemplifying unit improves recall: As a sanity check, we observe that including more context  $(\mathbf{L} + \mathbf{R})$  significantly boosts recall for both EGRET and all baselines compared to including just context before the exemplifying unit (L). In addition to providing more constraints over the exemplifying unit, we also observe improvements on multi-sentence exemplifying units due to term matching; as this  $\mathbf{L} + \mathbf{R}$  setting is not particularly realistic, we analyze only the L configuration moving forward.
Pretraining improves example retrieval: Pretraining on out-of-distribution Books3 examples substantially boosts EGRET's performance, with the best left-context-only model achieving a recall@1 of  $21.1\%$  compared to  $13.0\%$  without pretraining. In fact, an EGRET model pretrained on Books3 without fine-tuning on in-domain ELI5 data ( $19.3\%$  recall@1) outperforms the ELI5-only EGRET. While Figure 1 shows that the distribution over exemplification types differs based on the dataset/domain, our results suggest that many aspects of exemplification apply generally to wide forms of writing, and that the pretrained Books3 EGRET could be useful for many other applications.
4.2 Human evaluation of retrieved examples vs. generated examples
How do the examples retrieved by EGRET compare to examples generated by a state-of-the-art LFQA model? In theory, generative LFQA models can produce examples tailored to any input context; in practice, however, they struggle to generate relevant and informative examples. On the other hand, retriever models will always produce human-written examples, but it may not always be possible to retrieve a relevant example for an arbitrary held-out context. To explore this trade-off, we conduct a human evaluation by providing Mechanical Turk workers with an ELI5 question and context, and asking them to both rank and rate (on a 5 point Likert scale) three candidate exemplifying units: (1) the ground-truth; (2) the top-ranked retrieval from EGRET, restricted to only cases where this retrieval is not the ground-truth; and (3) a generated output from the state-of-the-art c-REALM-RT model of Krishna et al. (2021).
Task setup: In the ranking task, we ask workers to produce a ranking of the three choices (e.g.,  $1 > 2 > 3$ ). We allow equality (e.g.,  $1 = 2 > 3$ ) since multiple candidates can be equally valid for a given context. In the rating task, we ask workers to evaluate how well each example fits with the given context on a scale of 1 to 5. For both tasks, we collect three annotations per item for 100 total items, and we pay $0.35 per item for an estimated hourly wage of $21 per hour. While a completely fair comparison of EGRET to c-REALM-RT is infeasible due to differences in training objective and architecture, we choose to focus only on sentence-level exemplifying units that begin with "For example". We provide the question, left context, and "For example" marker to c-REALM-RT and decode using nucleus sampling with  $p = 0.9$  until a sentence boundary (e.g., period) is generated. For the retrieved output, we use EGRET(Books3 + ELI5, L) since the RT model has access to only the left context.
EGRET retrievals are preferred over generated exemplifying units: In both tasks, crowdworkers exhibit a clear preference for exemplifying units re
This model is pretrained on the PG-19 dataset (Rae et al., 2020) and fine-tuned on ELI5, conditioned on retrieved Wikipedia documents.
9We restrict workers to those in English speaking countries who have completed at least 1000 HITs with an acceptance rate of  $97\%$
7Models that outperform others in recall@  $k$  when  $k$  is small (less than 100) but underperform others when  $k$  is large (greater than 1000) typically score poorly in the average rank (e.g. ColBERT). More details about the variation in these retrieval evaluation metrics can be found in the Appendix  $\S B$ .
Left context	Ground-truth	EGRET-retrieved vs others	Analysis
Evolution is not a force to-wards the optimum, it's a force towards the minimum neces-sary.	For example, if grass was poi-sonous, it would be better for its survival, as less animals would come eat it.	[EGRET-retrieved]: For example, we move incredible slowly when compared to the maximum speed allowed in the universe. (R:0.111 / H:5.0)
[Highest ROUGE-L]: For example, if a certain percent-age of snakes are venomous, then the more snakes in a given area the more venomous. (R:0.22)	ROUGE is not a viable evalu-tion for example quality. Our EGRET's retrieved ex-ample was rated a 5/5 by all three crowdworkers but achieves lower ROUGE than an irrelevant example.
... You're brain is asleep and not paying any attention to your body so it ignores all of these stimuli unless they be-come too hard to ignore.	For example if the touching turns to slapping, the talking turns to yelling, or the light in the eyes turns to really bright light in the eyes.	[EGRET-retrieved]: For example, if you're in a room with a clock ticking you don't notice the ticking after a while. (H:4.0)
[c-REALM-RT Generated]: For example, its not just that your brain is dead. (H:2.0)	The EGRET-retrieved exam-ple effectively illustrates the phenomenon in the context and receives a higher average rating from crowdworkers than the generated example and even the ground-truth (3.3).
... Multiple births mean less time per offspring. Each individual offspring therefore has a lower chance of survival, ... Seems like the larger mammals tend to have single births.	For example, polar bears and elephants usually have single births.	[EGRET-retrieved]: For example, in mammals, a typical litter will be one offspring per pair of nipples as this is as many individuals a female can reasonably sustain. [c-REALM-RT Generated]: For example, ok, this is as close I can get to explaining in ELI5 terms.	EGRET retrieves an example based on a key entity from the context (mammals) but fails to address the concept to be exam-plified (“single births”)
Table 5: Instances where  $EGRET$  retrieves exemplifying units that are rated highly by Humans but have low ROUGE score with the ground-truth example (top); where model retrievals are rated as more meaningful than those generated by the  $c$ -REALM-RT model (middle); and where  $EGRET$  fails to retrieve a relevant example by relying too much on lexical overlap and  $c$ -REALM-RT fails by producing an overly-generic output (bottom).
RatingSTD(↑)	Krip. α
c-REALM-RT	2.800.775	0.058
EGRET (Books3 + ELI5, L)	3.550.636	0.125
Ground-truth	3.700.597	0.128
Table 6: Crowdworkers rate EGRET retrievals higher (on a scale of 1 to 5 of how well the exemplifying unit fits with the context) than the SOTA generative LFQA model.
RankingSTD(↓)	Krip. α
c-REALM-RT	2.260.271	0.168
EGRET (Books3 + ELI5, L)	1.880.252	0.154
Ground-truth	1.710.284	0.200
Table 7: On our ranking task (1=best, 3=worst), crowd-workers prefer EGRET retrievals over the generative LFQA model.
trieved by EGRET compared to those generated by c-REALM-RT. While both tasks are fairly subjective, as shown by the low interannotator agreement measured by Krippendorf's alpha, the results indicate that as of now, exemplification in LFQA is better handled by retrieval models than generative models, and that research into hybrid generation/retrieval LFQA models is a promising direction.
One limitation of our retrieval approach is that it will fail when the candidate set contains no relevant examples for a given context, which is not (at least in theory) an issue with generative models. How-
ever, we observe from our error analysis (Table 8) that both approaches at times produce seemingly relevant but incorrect examples. Furthermore, our human evaluations show that generative model fails to produce relevant examples for  $79\%$  of the contexts and consistently receive lower ratings than the retrieval model. This gap indicates that effectively incorporating retrieved information into the answer generation process is an important future research direction.
Our human study is coarse, judging the overall quality of the example, and future experiments could perform more fine-grained ratings of properties such as grammaticality and relevance.
Linguistic studies of exemplification: Early work (Kurohashi and Nagao, 1994) studies automatic detection of discourse relations including exemplification. Several works have studied exemplification in the domains of academic writing and teaching (Hyland, 2007; Oliveira and Brown, 2016; Triki, 2021). They proposed the following categorization of examples: general example (an instance of a general category); analogy (a parallel or similar case); counterexample (example that opposes or contradicts the anchor) and extreme example (boundary cases, in the sense of being more of an unusual instance than a representative, generic case). During our initial investigation, we
Left context	Ground truth	Retrieved/Generated	Error Analysis
Dog's do not pass a mirror test so it's highly unlikely they have a sense of self.[...]They do recognize there names, it's a bit of an illusion though.	For example if you have two dogs, and you give one of them a treat when you say Fido, and the other when you say Clifford, they learn that the respective words only apply to them.	[EGRET] For example people can identify their own dog.	EGRET retrieved relevant but semantically incorrect example (peo-ple identify their dog, instead of how dog identify themselves).
An economist would say healthcare has a positive externality[...]There are some things you can buy that make everyone better off.	For example: going to the doctor every time you are sick will make you less likely to make other people sick.	[EGRET] For example, a butterfly house, a free cinema, games con-soles etc.	EGRET retrieved examples related to the immediate preceding context ("some things you can buy...") but failed to retrieved examples based on earlier context (about healthcare).
4 billion - Economic and Military aid for Pakistan, Egypt, and Jordan. The goal is to have a few people in the mideast who call us allies. Es-entially, we buy their cooperation. That cooperation is sometimes use-ful.	For example, when we killed Osama Bin Laden, we sent troops into Pak-istan. Normally, countries don't tolerate troops from other countries. The Pakistaniis did complain a little, but they didn't do anything about it.	[c-REALM-RT] For example, for everyone here talking about how a lot of aid works: If we put money to-wards helping foreign countries re-build, we are imposing restrictions on domestic activity. [...]	c-REALM-RT generated an on-topic hypothetical example, which contra-dicts with the context.
You don't usually work on the same files because everything is split up between the departments. I haven't used USD yet but I have encountered the following workflow in different studios (using Maya).	For example: a character that has been rigged by one (or more, but not at the same time) rigger goes to the animators. Every animator works with the same character rig BUT each animator works on his/her own shot.[...]	[c-REALM-RT] For example, as a starting point: I'm a post graduate and work in the final sector of the project, not the project itself. Most of the work is done with other studios around the world who are made up of multiple departments.[...]	c-REALM-RT generated a personal example that is irrelevant to the context.
Table 8: Error analysis of retrieved and generated examples.
noticed that majority of the examples in our dataset are general examples, which aligns with the findings in Hyland (2007). The dominance of general examples could also be due to the choice of our exemplification markers – all examples given by Hyland (2007) for analogy have the exemplification marker "like", which we found noisy for automatic extraction. Li and Nenkova (2016) examine the closely-related instantiation discourse relation, where one text span explains in further detail the events described in another text span.
Long-form question answering: Our work studies exemplification mainly within the task of long-form question answering (LFQA), which involves generating paragraph-length answers to open-ended questions. Previous work has approached this problem using retrieval-augmented generation (Fan et al., 2019; Lewis et al., 2020), while Nakano et al. (2021) set up an interactive language model that learns LFQA through human interaction. Krishna et al. (2021) demonstrate that lexical overlap metrics such as ROUGE are not meaningful for this task. With a series of human annotations, Xu et al. (2022) study the discourse structures of long-form answers and identify exemplification as one of the functional roles commonly present in different types of long-form answers.
Neural retrieval models: Our EGRET retriever builds on recently-developed neural models that
retrieve evidence documents for open-retrieval question answering (Karpukhin et al., 2020; Guu et al., 2020) and fact-checking (Samarinas et al., 2021). These models demonstrate superior performance compared to non-neural methods like BM25 (Robertson et al., 1995); that said recent sparse/dense hybrid retrievers (Luan et al., 2021) could be interesting to explore on our exemplification task in the future.
In this work, we present the first computational study of exemplification in long-form question answering. We perform a detailed annotation over the use of exemplification across various domains and observe different distributions over complex exemplification types and units. While existing LFQA systems are conditional language models that do not give special treatment to exemplification, we propose to retrieve examples based on their context instead of generating them. We develop EGRET, a simple dual encoder trained with a contrastive learning objective, that outperforms a diverse set of baselines on this task of example retrieval, which we can meaningfully evaluate using simple ranking metrics instead of unsuitable metrics like ROUGE. We hope that our work spurs researchers to consider separately modeling and evaluating the fine-grained linguistic and discourse phenomena found in LFQA data.
We make use of pretrained language models to both generate and retrieve text in this work. Representations from pretrained language models are known to cause ethical concerns, such as perpetuating racial or gender bias (Field et al., 2021; Gala et al., 2020). We advise using caution and adopting a post-processing strategy to filter potentially offensive text produced by pretrained language models before releasing text content to users. Additionally, we note that most existing LFQA datasets (including the ELI5 dataset used in this work) and benchmarks are collected from English text sources. We hope future works can explore the use of exemplification in other languages.
We are grateful to the UMass NLP group for many helpful discussions. We thank Yapei Chang, Kalpesh Krishna, Katherine Thai for sharing their retriever models with us, as well as Tu Vu and Andrew Drozdov for sharing their compute resources. We would also like to thank the anonymous reviewers for their insightful feedback. SW and MI were supported by awards IIS-1955567 and IIS-2046248 from the National Science Foundation.
Annelie Adel. 2006. Metadiscourse in 11 and 12 english.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference of Machine Learning.
Barbara Fine Clouse. 2013. The student writer. McGraw-Hill.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A survey of race, racism, and
anti-racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1905-1925, Online. Association for Computational Linguistics.
Dhruvil Gala, Mohammad Omar Khursheed, Hannah Lerner, Brendan O'Connor, and Mohit Iyyer. 2020. Analyzing gender bias within narrative tropes. In Workshop on NLP and CSS at EMNLP.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. ArXiv, abs/2002.08909.
Ken Hyland. 2007. Applying a gloss: exemplifying and reformulating in academic discourse. Applied Linguistics, 28:266-285.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of Empirical Methods in Natural Language Processing.
Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In North American Association for Computational Linguistics.
Sadao Kurohashi and Makoto Nagao. 1994. Automatic detection of discourse structure by checking surface information in sentences. In *COLING* 1994 Volume 2: The 15th International Conference on Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005.11401.
Junyi Jessy Li and Ani Nenkova. 2016. The instantiation discourse relation: A corpus analysis of its properties and improved detection. In *NAACL*.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9:329-345.
Charles F. Meyer. 1992. Apposition in contemporary english.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332.
Alandeom W. Oliveira and Adam Oliver Brown. 2016. Exemplification in science instruction: Teaching and learning through examples. Journal of Research in Science Teaching, 53:737-767.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations.
Nils Reimers and Iryna Gurevych. 2021. The curse of dense low-dimensional information retrieval for large index sizes. In Proceedings of the 59th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 605-611, Online. Association for Computational Linguistics.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
Chris Samarinas, Wynne Hsu, and Mong Li Lee. 2021. Improving evidence retrieval for automated explainable fact-checking. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 84-91, Online. Association for Computational Linguistics.
Katherine Thai, Yapei Chang, Kalpesh Krishna, and Mohit Iyyer. 2022. Relic: Retrieving evidence for literary claims. In Association of Computational Linguistics.
Nesrine Triki. 2021. Exemplification in research articles: Structural, semantic and metadatacutive properties across disciplines. Journal of English for Academic Purposes, 54:101039.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. *ArXiv*, abs/1807.03748.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762.
Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. 2022. How do we answer complex questions: Discourse structure of long-form answers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Long paper.
A Evaluation Interface on Mechanic Turk
Given a question, a partial answer (context) and three candidate examples, a worker is asked to rate all three candidate examples, according to their fit with the question and the given context.
Please read the question, the (unfinished) answer, and rate each candidate example according to its fit as an EXAMPLE for the answer.
Please try to give each candidate example a different rating. If you cannot decide between two candidate examples, you may give them equal rating. However, please DO NOT give all three candidates the same rating.
Question: Why does  $100\%$  humidity not mean water everywhere?