SlowGuess/NAACL_2022_OCR huggingface dataset:
Shufan Wang $^{1}$ , Fangyuan Xu $^{2}$ , Laure Thompson $^{1}$ , Eunsol Choi $^{2}$ , Mohit Iyyer $^{1}$
<sup>1</sup> College of Information and Computer Sciences, University of Massachusetts Amherst
$^{2}$  Department of Computer Science, The University of Texas at Austin
{shufanwang, laurejt, miyyer}@cs.umass.edu
{fangyuan, eunsol}@utexas.edu
Exemplification is a process by which writers explain or clarify a concept by providing an example. While common in all forms of writing, exemplification is particularly useful in the task of long-form question answering (LFQA), where a complicated answer can be made more understandable through simple examples. In this paper, we provide the first computational study of exemplification in QA, performing a fine-grained annotation of different types of examples (e.g., hypotheticals, anecdotes) in three corpora. We show that not only do state-of-the-art LFQA models struggle to generate relevant examples, but also that standard evaluation metrics such as ROUGE are insufficient to judge exemplification quality. We propose to treat exemplification as a retrieval problem in which a partially-written answer is used to query a large set of human-written examples extracted from a corpus. Our approach allows a reliable ranking-type automatic metrics that correlates well with human evaluation. A human evaluation shows that our model's retrieved examples are more relevant than examples generated from a state-of-the-art LFQA model.
When an author introduces a complicated concept, they commonly follow it up with a concrete example to help clarify their intended meaning. This process, known as exemplification, occurs in diverse forms, including hypothetical examples, personal anecdotes, and analogies (Clouse, 2013). Exemplification is particularly common within the NLP task of long-form question answering (LFQA), where an author wants to communicate a concept unknown to the question asker. Consider the following QA pair from the r/explainlikeimfive subreddit (Fan et al., 2019):
Q: How does the ground not cave in while being under the heavy weight of cities?
A: It's all about what they're building on, and occasionally they get it wrong... For example,
San Francisco's Millennium Tower, built on mud and sand, has already sunk 18 inches into the ground.
Here, the answerer uses a specific example to emphasize the importance of building on a solid foundation. In general, explaining via exemplification is a fundamental technique in these kinds of pedagogical scenarios (Hyland, 2007), and it warrants separate study due to its importance in LFQA and the challenges in evaluating it. However, existing work on building LFQA models (Fan et al., 2019) does not give special treatment to exemplification or any other discourse phenomena, choosing instead to evaluate model outputs against reference answers using metrics such as ROUGE (Lin, 2004) that are not meaningful for this task (Krishna et al., 2021). In the above QA pair, any other structurally-unstable building (e.g., the Leaning Tower of Pisa) could serve as a valid example, but an LFQA model would be unfairly penalized for generating one of these acceptable alternatives.
In this paper, we first conduct a detailed study of exemplification across three different domains: Wikipedia, web articles, and community answers to questions from the ELI5 LFQA dataset. We extract sentences and clauses associated with exemplification by matching on explicit markers such as "for example" and "(e.g., ...) and annotate 300 examples from this dataset. Our analysis reveals significant variation in occurrence frequencies of different forms of exemplification (e.g., hypothetical vs. specific examples) across the three domains.
Next, we focus on improving the modeling and evaluation of the subtask of exemplification within LFQA. We propose to treat it as a retrieval problem rather than a generation problem: given a question and a prefix of a reference answer (in the above QA pair, all of the text before "For example"), a model must retrieve the ground-truth example that follows the prefix (the sentence about the Millenium Tower) from the set of all exemplifying sentences
Human Language Technologies, pages 2079 - 2092
and clauses in the dataset. We can use retrieval metrics such as recall@  $k$  to evaluate a model's ability to select the ground-truth example, which are more informative than metrics such as ROUGE.
We demonstrate that pretraining our retriever on a large-scale dataset of exemplifying units extracted from the Books3 corpus (Gao et al., 2020) and then fine-tuning it on ELI5 examples results in substantial improvements on these ranking metrics. Finally, we crowdsourced human evaluation comparing our retriever's outputs with those generated by the state-of-the-art ELI5 model of Krishna et al. (2021) and find that workers prefer the retriever's outputs far more frequently than those of the generation model. We hope that our work spurs more research into the evaluation and modeling of exemplification and other complex discourse phenomena present in LFQA. To facilitate future research, we publicly release the code and trained models from our work.
2 Exploring Exemplification
In this section, we first describe our data extraction process, which we use to mine instances of exemplification from three datasets (and domains): ELI5 (Fan et al., 2019), Natural Questions (Kwiatkowski et al., 2019) and Books3 (Gao et al., 2020). This process involves matching on a set of "exemplification markers" and collecting both the text of the matching example (a sentence or clause) as well as the surrounding context on both sides. We then conduct a fine-grained human annotation study on the extracted data, breaking exemplification down into different types and exploring how they are used across the different domains.
2.1 Extracting a Dataset of Exemplification
Exemplification markers: Hyland (2007) annotated a diverse collection of articles from multiple disciplines with a variety of rhetorical practices<sup>2</sup> and found that more than  $75\%$  of "examples" are signalled parenthetically or lexically with the use of the three most frequent "exemplification markers": "such as", "for example", "e.g.". Empirically, we find that the "such as" marker is noisy at signalling exemplification and often leads to ambiguous cases where it is hard to automatically detect the example boundary. Hence, we take the other two most
ELI5	NQ	Books3
# training examples	65,157	1,209	2,848,171
# validation examples	1,185	52	712,043
avg. # context words	123.7	74.0	155.7
avg. # example words	23.3	33.1	27.1
avg. # right words tokens	135.3	54.8	107.5
Table 1: Statistics of extracted example-in-context data.
frequent exemplification markers, namely "for example" and "e.g.", and extract the parentheses-enclosed clauses and sentences that contain these exemplification markers as examples.
Examples from Diverse Domains Using these two exemplification markers, we extract a dataset of examples in context from two popular LFQA datasets that come from different domains, ELI5 (Fan et al., 2019, Reddit answers) and Natural Questions (Kwiatkowski et al., 2019, Wikipedia passages). To study the exemplification phenomenon from a more diverse perspective, we also extract examples along with their surrounding context from the Books3 Corpus (Gao et al., 2020), a large-scale 100GB collection of books spanning a variety of topics and genres. Table 1 contains detailed statistics for the extracted example-in-context datasets.
2.2 Fine-grained Annotation Study
With the extracted dataset of examples, we conduct an in-depth analysis to understand different uses of exemplification in various domains. We (the authors) annotate a total of 300 examples extracted using exemplification markers from Natural Questions, ELI5 and Books3 as below. Fifty examples are annotated by two annotators (for purposes of computing agreement, reported in Section 2.3) and the rest are annotated by one annotator.
Given an extracted example and its left and right context, we first filter out around  $7\%$  of the extracted examples, either because they are extraction artifacts or because the marker is used for functions other than exemplification (e.g., referring to a figure or table). After this basic check, we annotate both structural information about the example (e.g., discourse units such as the anchor of the example) and semantic information about how it is used
$^{1}$ https://github.com/north125ptlm/lfqa-retrieval
2The annotation included texts from physics, biology, mechanical & electric engineering, philosophy, sociology, applied linguistics, and marketing.
<sup>3</sup>We consider questions with only a long answer span (i.e. paragraph answer) since they cannot be addressed solely by entity names or a boolean, and are suitable for studying LFQA.
Dataset	Valid	Extracted	% Valid
ELI5	87	93	94%
NQ	89	95	94%
Books3	85	94	90%
Total	261	282	93%
in the context. Table 2 contains statistics of the annotated subset. $^4$
2.2.1 Discourse units
Exemplification is usually expressed through three discourse units (Meyer, 1992; Triki, 2021): the anchor (also known as “exemplified unit”), the exemplification marker, and the example text itself (“exemplifying unit”). We annotate the anchor (marked as **bold**) and example (marked as **italics**). Concretely, in example (1) below, the anchor is “euryhaline species”, the exemplifying marker is “e.g.”, and the exemplifying unit is “Flounder”. As in the study of Triki (2021), we find that these units mainly come in two forms: (1) nominal groups that refer to entities, or (2) clauses that represent statements.
(1) However, some fish show a tremendous ability to effectively osmoregulate across a broad range of salinities; fish with this ability are known as euryhaline species, e.g., Flounder.
(2) Players earn you points, depending on their performance. For example, your QB might throw for 200 yards, earning 1 point per 10 yards, for 20 points.
During our initial investigation, we also noticed examples (3) which are signalled implicitly (i.e. without exemplification markers).<sup>5</sup> Identifying such examples automatically is beyond the scope of our study and warrants future work.
(3) The biggest driver of disparity in tech jobs is cost of living. If it costs 2000 a month to live in Boston, and 200 a month to live in India, then salaries will reflect that.
Table 3 shows that the length of the discourse units is roughly the same across the three datasets, which supports our experimental decision in Section 3 of using sentences as the base unit for example retrieval. We also find that all of the anchors we
Table 2: Statistics of annotated examples.
Dataset	# samples	Anchor	Example
ELI5	87	1.1/16.6	1.3/29.2
NQ	89	1.0/15.4	1.4/25.1
Books3	85	1.1/17.0	1.2/25.1
Type
Real	209	1.1/14.5	1.3/24.6
Hypothetical	52	1.2/23.4	1.2/33.7
Personal	13	1.1/18.1	2/49.6
Not-personal	248	1.1/16.3	1.2/25.2
Table 3: Length of the discourse units per dataset and type, presented as average # sentences / # words.
annotated occur before the examples, suggesting that using preceding context to retrieve examples gives the model sufficient information.
2.2.2 Real vs. Hypothetical Examples
One notable categorization found during our investigation and also identified by Triki (2021) is whether the examples are real, specific scenarios, or hypothetically-constructed scenarios. We detail the definition of the two types here:
Real examples: These examples are either real entities (4) or specific scenarios (5) that are constructed as fact clauses.
(4) CEOs lead a range of organizations, including public and private corporations, non-profit organizations and even some government organizations (e.g., Crown corporations).
(5) For a given pressure, different liquids boil at different temperatures. For example, water boils at  $100^{\circ}C$ $(212^{\circ}F)$  at sea level, but at  $93.4^{\circ}C$ $(200.1^{\circ}F)$  at 2,000 metres (6,600 ft) altitude.
Hypothetical examples: In contrast, hypothetical examples are scenarios constructed by the author. According to Triki (2021), hypothetical examples often come with the use of conditional clauses if or signalled via assume. These examples are generally more complicated and are specifically constructed for the purpose of exemplification.
(6) The reasoning is that if you share your life with someone then anything you do during that time is made possible by their support. For example, if your wife is a stay at home wife and your a business man making lots of money, the reasoning is that you would not have the same amount of time to dedicate to work if you had to look after your own house and/or children.
We observe a different distribution of the two types of examples in the three datasets (Figure 1, top). ELI5 contains more hypothetical examples
4 Annotated examples in the three datasets can be found in Table A1 in the appendix.
5Recent study on long form QA (Xu et al., 2022) contains manually identified example sentences, including those signalled implicitly.
Figure 1: Distribution of different types of examples across three datasets.
(32%) than the other two datasets (16% for NQ, 12% for Books3), showing that hypothetical examples are commonly used to explain complicated concepts. We also note that hypothetical examples are generally longer than real ones (33.7 v.s. 24.6 words, as seen in Table 3), aligning with our observation that these examples are more complicated.
2.2.3 Personal Information
Previous work found that exemplification is "rarely personalized" in academic texts (Adel, 2006), referring to the uncommon use of first person pronouns or reference to the author. In contrast, we found a consistent presence of personal information in the examples we examined, in the form of either personal anecdotes (7) or an example situated in the author's own circumstance (e.g., in my city...). We thus also annotate whether the exemplifying units contain personal information or not.
(7) ... But they also give advice a doctor might have forgotten. For example about 6 months ago I went to the urgent care center for ear pain and was prescribed an ear drop antibiotic. ...
We observe differences in the presence of personal information across the three datasets (Figure 1, bottom) - ELI5 answers, which are written by users from online community forum, contain a substantial portion of examples with personal information  $(12\%)$ , while such information is rarely present in the other two datasets. There is also a notable length difference between examples with and without personal information (49.6 v.s. 25.2 words), showing that more detailed description is provided for personal anecdotes. The observation that ELI5 contains many personal examples raises concerns that language models trained on such datasets will generate personal examples that cannot be verified or meaningfully interpreted.
2.3 Annotation agreement
We report agreement for the three annotation tasks performed on the 50 two-way annotated samples. For discourse unit annotation, we find high unigram overlap between the two annotations (0.81 for the anchor and 0.92 for the example). For annotation of real vs. hypothetical examples and the presence of personal information, we find a modest to high agreement with a Cohen's kappa of 0.48 for both. Additionally, we observe annotations of real v.s. hypothetical example to be split when the example refers to abstract actions, such as the one below (8).
(8) Carpets are used for a variety of purposes, including insulating a person's feet from a cold tile or concrete floor, making a room more comfortable as a place to sit on the floor (e.g., when playing with children or as a prayer rug), reducing sound from walking (particularly in apartment buildings) and adding decoration or colour to a room.
3 Retrieving Examples in LFQA
Our annotation and analysis of the extracted exemplification dataset reveals the diversity and complexity of this discourse phenomenon. In this section, we shift our focus towards building retrieval-based models to produce examples based on a given LFQA context. First, we define our example-retrieval task formally and introduce our evaluation metrics. Then, we describe our contrastive-learning based retriever model, which closely follows the retriever implementation in Thai et al. (2022), and baseline retrieval models, and report their performances.
Given a context (part of the answer to a given LFQA question) with a masked out exemplifying unit, model is asked to retrieve that masked unit from a retrieval corpus. We consider two settings for context: (1) concatenation of left and right contexts surrounding the exemplifying unit and (2) left context preceding the exemplifying unit only (as in Figure 2). Both left and right contexts are truncated to 256 tokens surrounding the exemplifying unit. We use all 66K exemplifying units extracted from the 272K QA instances in the training and development portion of ELI5 dataset (Petroni et al., 2021) as the retrieval corpus. To illustrate the input format, consider the below answer to the question "Why didn't anyone discover dinosaur fossils before the 1800s?"
Figure 2: Our EGRET model uses dual RoBERTa encoders to embed (1) the context surrounding an exemplifying unit and (2) all 66K exemplifying units extracted from ELI5. A contrastive objective is then used to move the context embedding close to the embedding of the ground-truth exemplifying unit that occurs within that context, and far from all other negative exemplifying units.
Prior to the 1800's, when people dug up fossils (and more frequently, subfossil bones from ice age animals, which are more common and easier to find) they tended to interpret them in light of their existing myths and legends. [MASK] Because fossils are almost always found as a jumble of bones rather than a neat skeleton and because they are incomplete... nobody looked at dinosaur skeletons and realized what the animals that made them actually looked like.
where the [MASK] token corresponds to
For example, when a wooly rhino skull was dug up near Klagenfurt, it was thought to be the skull of a dragon.
The first quote block showing the masked answer will be used as a query to retrieve the exemplifying unit in the second quote block.
This retrieval task is challenging due not only to the size of the candidate set but also because of the topical similarity between many ELI5 questions, which was previously noted by Krishna et al. (2021). Retrieving based on lexical overlap, as in BM-25 and other string-matching based approaches, cannot identify exemplifying units that are relevant but share little overlap with the context.
3.2 Evaluation Data / Metric
For evaluation data, we use 1,185 context-example pairs automatically identified from 1,507 QA instances in the ELI5 development set (Petroni et al., 2021) by our discourse marker heuristics. We evaluate the performance of our retriever by measuring how reliable it is at retrieving ground-truth examples from the candidate example set. Concretely,
given the candidate set of examples, each model should output a ranked list of all examples according to their fit for the context. We evaluate these rankings using recall@k of the ground-truth example (where  $k = 1,3,5,10,50,100$ ) from the set of all 66K examples in ELI5. By using retriever-based evaluation metrics, we can directly measure a model's ability to understand and exemplify a context, in contrast to string overlap metrics like ROUGE (Lin, 2004) which are uninformative for this task.
We first introduce our model (EGRET) and describe baseline retrieval methods. We use all baseline models in a zero-shot manner (without additional fine-tuning on exemplification dataset).
EGRET: an example retriever for LFQA We train an example retriever model (EGRET) on our extracted exemplification dataset from training portion of ELI5 (Fan et al., 2019), which contains  $65K$  extracted context-example pairs. Our retriever consists of dual Transformer encoders (Vaswani et al., 2017), one to encode the context query and the other to encode candidate examples (Figure 2). Both encoders are initialized as pretrained RoBERTa-base models (Liu et al., 2019), as in the dense-RELiC model of Thai et al. (2022). To obtain a query embedding  $\pmb{c}_i$ , we feed the query encoder the surrounding context of a masked-out example, as in Figure 2. Similarly, we use the other encoder to compute embeddings of the ground-truth example  $(e_i^+)$  as well as negative examples sampled
Model	Context	Recall@k(↑)	Avg rank (↓)
1	3	5	10	50	100
[zero-shot] baselines, including pretrained dense retrievers
Random	0.0	0.0	0.01	0.02	0.08	0.15	33171.0
BM25 (Robertson et al., 1995)	L	4.6	9.5	12.1	16.2	25.6	30.4	24940.1
DPR (Karpukhin et al., 2020)	L	2.7	5.2	7.1	9.7	20.3	27.5	7796.9
ColBERT (Khattab and Zaharia, 2020)	L	6.0	11.8	14.3	18.2	31.2	36.3	9948.6
SBERT (Reimers and Gurevych, 2021)	L	5.7	11.6	15.0	20.4	34.3	42.2	5122.7
BM25 (Robertson et al., 1995)	L+R	8.7	16.1	20.0	24.8	38.0	42.7	13968.8
DPR (Karpukhin et al., 2020)	L+R	4.4	8.6	11.1	15.7	27.2	33.5	5684.0
ColBERT (Khattab and Zaharia, 2020)	L+R	8.9	16.4	18.9	23.3	36.3	42.2	8636.0
SBERT (Reimers and Gurevych, 2021)	L+R	9.2	16.3	21.2	27.1	44.2	51.3	3216.7
our models, trained on exemplification data
EGRET (ELI5)	L	13.0	22.8	29.3	36.5	55.2	64.0	807.2
EGRET (Books3 only)	L	19.3	30.4	36.8	44.1	63.1	69.0	2427.6
EGRET (Books3 + ELI5)	L	21.1	33.5	39.2	46.8	66.7	73.0	609.8
EGRET (ELI5)	L+R	23.5	35.6	41.9	51.0	71.0	77.6	300.6
EGRET (Books3 only)	L+R	33.4	47.8	53.8	61.5	76.5	82.5	632.9
EGRET (Books3 + ELI5)	L+R	36.9	50.6	58.2	64.8	80.2	85.8	188.4
Table 4: Our EGRET model outperforms pretrained (or non-parametric) baselines on the example retrieval task, indicating that exemplification cannot be solved by term matching or coarse query-context similarity alone. Pretraining EGRET on out-of-distribution examples from Books3 results in large improvements in recall@k. Finally, including context to the right of the exemplifying unit significantly boosts performance.
from other contexts, forming a set  $E$  of example embeddings. We fine-tune both encoders in EGRET with a contrastive learning objective (van den Oord et al., 2018; Chen et al., 2020):
$$
\mathcal {L} (\theta) = - \sum_ {\left(c _ {i}, e _ {i}\right) \in E} \log \frac {\exp c _ {i} \cdot e _ {i} ^ {+}}{\sum_ {e _ {j} \in E} \exp c _ {i} \cdot e _ {j}} \tag {1}
$$
This objective places the context vector  $\boldsymbol{c}_i$  close to that of the ground-truth example vector  $e_i^+$  of an example, and far from other examples  $e_j$  in the batch  $E$  ("in-batch" negative samples). We train both the left-context-only and the left-and-right-context models on a single RTX-8000 GPU for 10 epochs, using the Adam optimizer (Kingma and Ba, 2015) with learning rate initialized at  $1e - 5$  for 10 epochs with early stopping. Both models converge in 4 epochs of training over the ELI5 dataset.
While the EGRET model described above is trained on the ELI5 dataset, exemplification is pervasive in many kinds of texts, as shown by our annotation in Section 2. Thus, we also experiment with a transfer learning scenario by pretraining EGRET on a dataset of 3.5 million examples extracted from Books3 (Gao et al., 2020), and then fine-tuning the resulting model on the ELI5 examples. We perform Books3 pretraining for both left-context-only and left-and-right-context models on a single
RTX-8000 GPU for 5 epochs using Adam with learning rate initialized at  $1e - 5$ . Both models converge after one epoch of fine-tuning over the ELI5 dataset.
Baselines: We compare EGRET to a term matching method as well as three publicly-available pretrained dense retrievers.
BM25 (Robertson et al., 1995): BM25 retrieves text via a scoring function reliant on lexical overlap. We use the implementation from the rank_bm25 library, with the default BM25Okapi as the similarity algorithm.
DPR (Karpukhin et al., 2020): DPR is a retriever model trained on Natural Questions (Kwiatkowski et al., 2019) that computes dense representations of queries and evidence paragraphs for retrieval.
ColBERT (Khattab and Zaharia, 2020): ColBERT similarly uses pretrained language models to embed text, but contextualizes query and candidate documents using late interaction and was trained on MS MARCO (Bajaj et al., 2018).