Venue: NAACL 2022

# Modeling Exemplification in Long-form Question Answering via Retrieval

Shufan Wang\({}^{1}\), Fangyuan Xu\({}^{2}\), Laure Thompson\({}^{1}\), Eunsol Choi\({}^{2}\), Mohit Iyyer\({}^{1}\)

\({}^{1}\) College of Information and Computer Sciences, University of Massachusetts Amherst \({}^{2}\) Department of Computer Science, The University of Texas at Austin {shufanwang, laurejt, miyyer}@cs.umass.edu

###### Abstract

Exemplification is a process by which writers explain or clarify a concept by providing an example. While common in all forms of writing, exemplification is particularly useful in the task of long-form question answering (LFQA), where a complicated answer can be made more understandable through simple examples. In this paper, we provide the first computational study of exemplification in QA, performing a fine-grained annotation of different types of examples (e.g., hypotheticals, anecdotes) in three corpora. We show that not only do state-of-the-art LFQA models struggle to generate relevant examples, but also that standard evaluation metrics such as ROUGE are insufficient to judge exemplification quality. We propose to treat exemplification as a _retrieval_ problem in which a partially-written answer is used to query a large set of human-written examples extracted from a corpus. Our approach allows a reliable ranking-type automatic metrics that correlates well with human evaluation. A human evaluation shows that our model's retrieved examples are more relevant than examples generated from a state-of-the-art LFQA model.

+
Footnote †: Q: How does the ground not cave in while being under the heavy weight of cities?**

**A: It’s all about what they’re building on, and occasionally they get it wrong... For example, 2079**

## 1 Introduction

When an author introduces a complicated concept, they commonly follow it up with a concrete example to help clarify their intended meaning. This process, known as _exemplification_, occurs in diverse forms, including hypothetical examples, personal anecdotes, and analogies [3]. Exemplification is particularly common within the NLP task of _long-form question answering_ (LFQA), where an author wants to communicate a concept unknown to the question asker. Consider the following QA pair from the r/explainlikeimfive subreddit [4]:

Here, the answerer uses a specific example to emphasize the importance of building on a solid foundation. In general, explaining via exemplification is a fundamental technique in these kinds of pedagogical scenarios [5], and it warrants separate study due to its importance in LFQA and the challenges in evaluating it. However, existing work on building LFQA models [6] does not give special treatment to exemplification or any other discourse phenomena, choosing instead to evaluate model outputs against reference answers using metrics such as ROUGE [8] that are not meaningful for this task [11]. In the above QA pair, any other structurally-unstable building (e.g., the Leaning Tower of Pisa) could serve as a valid example, but an LFQA model would be unfairly penalized for generating one of these acceptable alternatives.

In this paper, we first conduct a detailed study of exemplification across three different domains: Wikipedia, web articles, and community answers to questions from the ELIS LFQA dataset. We extract sentences and clauses associated with exemplification by matching on explicit markers such as _"for example"_ and _"(e.g., ...)"_ and annotate 300 examples from this dataset. Our analysis reveals significant variation in occurrence frequencies of different forms of exemplification (e.g., hypothetical vs. specific examples) across the three domains.

Next, we focus on improving the modeling and evaluation of the subtask of exemplification within LFQA. We propose to treat it as a _retrieval_ problem rather than a generation problem: given a question and a prefix of a reference answer (in the above QA pair, all of the text before _"For example"_), a model must retrieve the ground-truth example that follows the prefix (the sentence about the Millenium Tower) from the set of all exemplifying sentences

and clauses in the dataset. We can use retrieval metrics such as recall@\(k\) to evaluate a model's ability to select the ground-truth example, which are more informative than metrics such as ROUGE.

We demonstrate that pretraining our retriever on a large-scale dataset of exemplifying units extracted from the Books3 corpus [14] and then fine-tuning it on ELI5 examples results in substantial improvements on these ranking metrics. Finally, we crowdsource human evaluation comparing our retriever's outputs with those generated by the state-of-the-art ELI5 model of krishna2021learning and find that workers prefer the retriever's outputs far more frequently than those of the generation model. We hope that our work spurs more research into the evaluation and modeling of exemplification and other complex discourse phenomena present in LFQA. To facilitate future research, we publicly release the code and trained models from our work.1

Footnote 1: [https://github.com/north125ptlm/Ifqa-retrieval](https://github.com/north125ptlm/Ifqa-retrieval)

## 2 Exploring Exemplification

In this section, we first describe our data extraction process, which we use to mine instances of exemplification from three datasets (and domains): ELI5 [6], Natural Questions [9] and Books3 [14]. This process involves matching on a set of "exemplification markers" and collecting both the text of the matching example (a sentence or clause) as well as the surrounding context on both sides. We then conduct a fine-grained human annotation study on the extracted data, breaking exemplification down into different types and exploring how they are used across the different domains.

### Extracting a Dataset of Exemplification

Exemplification markers:Hyland [7] annotated a diverse collection of articles from multiple disciplines with a variety of rhetorical practices2 and found that more than 75% of "examples" are signalled parenthetically or lexically with the use of the three most frequent "exemplification markers": _"such as"__, "for example"__, "e.g."_. Empirically, we find that the _"such as"_ marker is noisy at signalling exemplification and often leads to ambiguous cases where it is hard to automatically detect the example boundary. Hence, we take the other two most frequent exemplification markers, namely _"for example"_ and _"e.g."_, and extract the parentheses-enclosed clauses and sentences that contain these exemplification markers as examples.

Footnote 2: The annotation included texts from physics, biology, mechanical & electric engineering, philosophy, sociology, applied linguistics, and marketing.

Examples from Diverse DomainsUsing these two exemplification markers, we extract a dataset of examples in context from two popular LFQA datasets that come from different domains, ELI5 [6, Reddit answers] and Natural Questions [9, Wikipedia passages].3 To study the exemplification phenomenon from a more diverse perspective, we also extract examples along with their surrounding context from the Books3 Corpus [14], a large-scale 100GB collection of books spanning a variety of topics and genres. Table 1 contains detailed statistics for the extracted example-in-context datasets.

### Fine-grained Annotation Study

With the extracted dataset of examples, we conduct an in-depth analysis to understand different uses of exemplification in various domains. We (the authors) annotate a total of 300 examples extracted using exemplification markers from Natural Questions, ELI5 and Books3 as below. Fifty examples are annotated by two annotators (for purposes of computing agreement, reported in Section 2.3) and the rest are annotated by one annotator.

Given an extracted example and its left and right context, we first filter out around 7% of the extracted examples, either because they are extraction artifacts or because the marker is used for functions other than exemplification (e.g., referring to a figure or table). After this basic check, we annotate both structural information about the example (e.g., discourse units such as the anchor of the example) and semantic information about how it is used

\begin{table}
\begin{tabular}{l r r r} \hline  & ELI5 & NQ & Books3 \\ \hline \# training examples & 65,157 & 1,209 & 2,848,171 \\ \# validation examples & 1,185 & 52 & 712,043 \\ \hline avg. \# context words & 123.7 & 74.0 & 155.7 \\ avg. \# example words & 23.3 & 33.1 & 27.1 \\ avg. \# right words tokens & 135.3 & 54.8 & 107.5 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of extracted example-in-context data.

 

in the context. Table 2 contains statistics of the annotated subset.4

Footnote 4: Annotated examples in the three datasets can be found in Table A1 in the appendix.

#### 2.2.1 Discourse units

Exemplification is usually expressed through three discourse units Meyer (1992); Triki (2021): the anchor (also known as "exemplified unit"), the exemplification marker, and the example text itself ("exemplifying unit"). We annotate the anchor (marked as **bold**) and example (marked as _italics_). Concretely, in example (1) below, the anchor is "eurylahline species", the exemplifying marker is "e.g.", and the exemplifying unit is "Flounder". As in the study of Triki (2021), we find that these units mainly come in two forms: (1) nominal groups that refer to entities, or (2) clauses that represent statements.

\begin{tabular}{l l r}   (1) However , some fish show a tremendous ability \\ to effectively osmoregulate across a broad range \\ of salinities ; fish with this ability are known as \\ **evaryhaline species** , e.g., _Flounder_. \\ \end{tabular}

\begin{tabular}{l l}   (2) Players earn you points, depending on their \\ performance. For example, _your QB might throw \\ for 200 yards, earning 1 point per 10 yards, for 20 points_. \\ \end{tabular}

During our initial investigation, we also noticed examples (3) which are signalled _implicitly_ (i.e. without exemplification markers).5 Identifying such examples automatically is beyond the scope of our study and warrants future work.

Footnote 5: Recent study on long form QA Xu et al. (2022) contains manually identified example sentences, including those signalled implicitly.

\begin{tabular}{l l}   (3) The biggest driver of disparity in tech jobs \\ is cost of living. _If it costs 2000 a month to live in Boston, and 200 a month to live in India, then salaries will reflect that_. \\ \end{tabular}

Table 3 shows that the length of the discourse units is roughly the same across the three datasets, which supports our experimental decision in Section 3 of using sentences as the base unit for example retrieval. We also find that all of the anchors we annotated occur _before_ the examples, suggesting that using preceding context to retrieve examples gives the model sufficient information.

#### 2.2.2 Real vs. Hypothetical Examples

One notable categorization found during our investigation and also identified by Triki (2021) is whether the examples are real, specific scenarios, or hypothetically-constructed scenarios. We detail the definition of the two types here:

Real examples:These examples are either realentities (4) or specific scenarios (5) that are constructed as fact clauses.

\begin{tabular}{l l}   (4) CEOs lead a range of organizations , including public and private corporations, non-profit organizations and even **some government organizations** (e.g., _Crown corporations_). \\ \end{tabular}

\begin{tabular}{l l}   (5) For a given pressure , different liquids boil \\ at different temperatures. For example, water \\ boils at 100\({}^{\circ}\) C (212\({}^{\circ}\) F) at sea level, but at 93.4\({}^{\circ}\) \\ C (200.1\({}^{\circ}\) F) at 2,000 metres (6,600 ft) altitude. \\ \end{tabular}

Hypothetical examples:In contrast, hypothetical examples are scenarios constructed by the author. According to Triki (2021), hypothetical examples often come with the use of conditional clauses _if_ or signalled via _assume_. These examples are generally more complicated and are specifically constructed for the purpose of exemplification.

\begin{tabular}{l l}   (6) The reasoning is that if you share your life \\ with someone then anything you do during that \\ time is made possible by their support. For example, if your wife is a stay at home wife and \\ your a business man making lots of money, the \\ reasoning is that you would not have the same \\ amount of time to dedicate to work if you had to \\ look after your own house and/or children. \\ \end{tabular}

We observe a different distribution of the two types of examples in the three datasets (Figure 1, top). ELI5 contains more hypothetical examples

\begin{table}

 

(32%) than the other two datasets (16% for NQ, 12% for Books3), showing that hypothetical examples are commonly used to explain complicated concepts. We also note that hypothetical examples are generally longer than real ones (33.7 v.s. 24.6 words, as seen in Table 3), aligning with our observation that these examples are more complicated.

#### 2.2.3 Personal Information

Previous work found that exemplification is "rarely personalized" in academic texts Adel (2006), referring to the uncommon use of first person pronouns or reference to the author. In contrast, we found a consistent presence of personal information in the examples we examined, in the form of either personal anecdotes (7) or an example situated in the author's own circumstance (e.g., _in my city..._). We thus also annotate whether the exemplifying units contain personal information or not.

* [noitemsep]
* But they also give advice a doctor might have forgotten. For example _about 6 months ago I went to the urgent care center for ear pain and was prescribed an ear drop antibiotic_. ...

We observe differences in the presence of personal information across the three datasets (Figure 1, bottom) - ELI5 answers, which are written by users from online community forum, contain a substantial portion of examples with personal information (12%), while such information is rarely present in the other two datasets. There is also a notable length difference between examples with and without personal information (49.6 v.s. 25.2 words), showing that more detailed description is provided for personal anecdotes. The observation that ELI5 contains many personal examples raises concerns that language models trained on such datasets will generate personal examples that cannot be verified or meaningfully interpreted.

### Annotation agreement

We report agreement for the three annotation tasks performed on the 50 two-way annotated samples. For discourse unit annotation, we find high unigram overlap between the two annotations (0.81 for the anchor and 0.92 for the example). For annotation of real vs. hypothetical examples and the presence of personal information, we find a modest to high agreement with a Cohen's kappa of 0.48 for both. Additionally, we observe annotations of real v.s. hypothetical example to be split when the example refers to abstract actions, such as the one below (8).

* [noitemsep]
* Carpet s are used for a variety of purposes , including insulating a person 's feet from a cold tile or concrete floor , making a room more comfortable as a place to sit on the floor (_e.g. , when playing with children or as a prayer rug_ ) , reducing sound from walking ( particularly in apartment buildings ) and adding decoration or colour to a room .

## 3 Retrieving Examples in LFQA

Our annotation and analysis of the extracted exemplification dataset reveals the diversity and complexity of this discourse phenomenon. In this section, we shift our focus towards building retrieval-based models to produce examples based on a given LFQA context. First, we define our example-retrieval task formally and introduce our evaluation metrics. Then, we describe our contrastive-learning based retriever model, which closely follows the retriever implementation in Thai et al. (2022), and baseline retrieval models, and report their performances.

### Task Definition

Given a context (part of the answer to a given LFQA question) with a masked out exemplifying unit, model is asked to retrieve that masked unit from a retrieval corpus. We consider two settings for context: (1) concatenation of left and right contexts surrounding the exemplifying unit and (2) left context preceding the exemplifying unit only (as in Figure 2). Both left and right contexts are truncated to 256 tokens surrounding the exemplifying unit. We use all 66K exemplifying units extracted from the 272K QA instances in the training and development portion of ELI5 dataset Petroni et al. (2021) as the retrieval corpus. To illustrate the input format, consider the below answer to the question "Why didn't anyone discover dinosaur fossils before the 1800s?":

Figure 1: Distribution of different types of examples across three datasets.

 

Prior to the 1800's, when people dug up fossils (and more frequently, subfossil bones from ice age animals, which are more common and easier to find) they tended to interpret them in light of their existing myths and legends. [**MassK**] Because fossils are almost always found as a jumble of bones rather than a neat skeleton and because they are incomplete... nobody looked at dinosaur skeletons and realized what the animals that made them actually looked like.

where the [**mask**] token corresponds to

For example, when a wooly rhino skull was dug up near Klagenfurt, it was thought to be the skull of a dragon.

The first quote block showing the masked answer will be used as a query to retrieve the exemplifying unit in the second quote block.

This retrieval task is challenging due not only to the size of the candidate set but also because of the topical similarity between many ELI5 questions, which was previously noted by Krishna et al. (2021). Retrieving based on lexical overlap, as in BM-25 and other string-matching based approaches, cannot identify exemplifying units that are relevant but share little overlap with the context.

### Evaluation Data / Metric

For evaluation data, we use 1,185 context-example pairs automatically identified from 1,507 QA instances in the ELI5 development set Petroni et al. (2021) by our discourse marker heuristics. We evaluate the performance of our retriever by measuring how reliable it is at retrieving ground-truth examples from the candidate example set. Concretely, given the candidate set of examples, each model should output a ranked list of all examples according to their fit for the context. We evaluate these rankings using recall@\(k\) of the ground-truth example (where \(k=1,3,5,10,50,100\)) from the set of all 66K examples in ELI5. By using retrieved-based evaluation metrics, we can directly measure a model's ability to understand and exemplify a context, in contrast to string overlap metrics like ROUGE Lin (2004) which are uninformative for this task.

### Models

We first introduce our model (EgRet) and describe baseline retrieval methods. We use all baseline models in a zero-shot manner (without additional fine-tuning on exemplification dataset).

EgRet: an example retriever for LFQAWe train an example retriever model (EgRet) on our extracted exemplification dataset from training portion of ELI5 Fan et al. (2019), which contains \(65K\) extracted context-example pairs. Our retriever consists of dual Transformer encoders Vaswani et al. (2017), one to encode the context query and the other to encode candidate examples Figure 2). Both encoders are initialized as pretrained RoBERTa-base models Liu et al. (2019), as in the dense-RELiC model of Thai et al. (2022). To obtain a query embedding \(c_{i},\) we feed the query encoder the surrounding context of a masked-out example, as in Figure 2. Similarly, we use the other encoder to compute embeddings of the ground-truth example (\(e^{+}_{i}\)) as well as negative examples sampled

Figure 2: Our EgRet model uses dual RoBERTa encoders to embed (1) the context surrounding an exemplifying unit and (2) all 66K exemplifying units extracted from ELI5. A contrastive objective is then used to move the context embedding close to the embedding of the ground-truth exemplifying unit that occurs within that context, and far from all other negative exemplifying units.

 

from other contexts, forming a set \(E\) of example embeddings. We fine-tune both encoders in EqRet with a contrastive learning objective (van den Oord et al., 2018; Chen et al., 2020):

\[{\mathcal{L}}(\theta)=-\sum_{(c_{i},e_{i})\in E}\log\frac{\exp c_{ i}\cdot e^{+}_{i}}{\sum_{e_{j}\in E}\exp c_{i}\cdot e_{j}} \tag{1}\]

This objective places the context vector \(c_{i}\)_close_ to that of the ground-truth example vector \(e^{+}_{i}\) of an example, and _far_ from other examples \(e_{j}\) in the batch \(E\) ("in-batch" negative samples). We train both the left-context-only and the left-and-right-context models on a single RTX-8000 GPU for 10 epochs, using the Adam optimizer (Kingma and Ba, 2015) with learning rate initialized at \(1e-5\) for 10 epochs with early stopping. Both models converge in 4 epochs of training over the ELIS dataset.

**Pretraining**EgRet **on a huge set of examples:** While the EgRet model described above is trained on the ELIS dataset, exemplification is pervasive in many kinds of texts, as shown by our annotation in Section 2. Thus, we also experiment with a _transfer_ learning scenario by pretraining EGRet on a dataset of 3.5 million examples extracted from Books3 (Gao et al., 2020), and then fine-tuning the resulting model on the ELIS examples. We perform Books3 pretraining for both left-context-only and left-and-right-context models on a single RTX-8000 GPU for 5 epochs using Adam with learning rate initialized at \(1e-5\). Both models converge after one epoch of fine-tuning over the ELIS dataset.

Baselines:We compare EgRet to a term matching method as well as three publicly-available pretrained dense retrievers.

**BM25 (Robertson et al., 1995):** BM25 retrieves text via a scoring function reliant on lexical overlap. We use the implementation from the rank_bm25 library,6 with the default BM25Okapi as the similarity algorithm.

Footnote 6: [https://github.com/dorianbrown/rank_bm25](https://github.com/dorianbrown/rank_bm25)

DPR (Karpukhin et al., 2020): DPR is a retriever model trained on Natural Questions Kwiatkowski et al. (2019) that computes dense representations of queries and evidence paragraphs for retrieval.

ColBERT (Khattab and Zaharia, 2020): ColBERT similarly uses pretrained language models to embed text, but contextualizes query and candidate documents using late interaction and was trained on MS MARCO Bajaj et al. (2018).

SBERT (Reimers and Gurevych, 2021): SBERT is a sentence-BERT-based encoder model with down-project layers that outperforms DPR on the MS MARCO dataset.

\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline
**Model** & \multicolumn{1}{r}{**Context**} & \multicolumn{7}{c}{**Recall@\& \(\boldsymbol{k}\) (\(\uparrow\))**} & \multicolumn{3}{c}{**Avg rank (\(\downarrow\))**} \\ \cline{3-8}  & & 1 & 3 & 5 & 10 & 50 & 100 & \\ \hline Random & & 0.0 & 0.0 & 0.01 & 0.02 & 0.08 & 0.15 & 33171.0 \\ BM25 (Robertson et al., 1995) & L & 4.6 & 9.5 & 12.1 & 16.2 & 25.6 & 30.4 & 24940.1 \\ DPR (Karpukhtin et al., 2020) & L & 2.7 & 5.2 & 7.1 & 9.7 & 20.3 & 27.5 & 7796.9 \\ ColBERT (Khattab and Zaharia, 2020) & L & 6.0 & 11.8 & 14.3 & 18.2 & 31.2 & 36.3 & 9948.6 \\ SBERT (Reimers and Gurevych, 2021) & L & 5.7 & 11.6 & 15.0 & 20.4 & 34.3 & 42.2 & 5122.7 \\ BM25 (Robertson et al., 1995) & L+R & 8.7 & 16.1 & 20.0 & 24.8 & 38.0 & 42.7 & 13968.8 \\ DPR (Karpukhin et al., 2020) & L+R & 4.4 & 8.6 & 11.1 & 15.7 & 27.2 & 33.5 & 5684.0 \\ ColBERT (Khattab and Zaharia, 2020) & L+R & 8.9 & **16.4** & 18.9 & 23.3 & 36.3 & 42.2 & 8636.0 \\ SBERT (Reimers and Gurevych, 2021) & L+R & **9.2** & 16.3 & **21.2** & **27.1** & **44.2** & **51.3** & **3216.7** \\ \hline \end{tabular} 
\begin{tabular}{l l r r r r r r} \hline \hline \multicolumn{8}{c}{_our models, trained on exemplification data_} \\ \hline EqRet (ELIS) & L & 13.0 & 22.8 & 29.3 & 36.5 & 55.2 & 64.0 & 807.2 \\ EqRet (Books3 only) & L & 19.3 & 30.4 & 36.8 & 44.1 & 63.1 & 69.0 & 2427.6 \\ EqRet (Books3 + ELIS) & L & 21.1 & 33.5 & 39.2 & 46.8 & 66.7 & 73.0 & 609.8 \\ EqRet (ELIS) & L+R & 23.5 & 35.6 & 41.9 & 51.0 & 71.0 & 77.6 & 300.6 \\ EqRet (Books3 only) & L+R & 33.4 & 47.8 & 53.8 & 61.5 & 76.5 & 82.5 & 632.9 \\ EqRet (Books3 + ELIS) & L+R & **36.9** & **50.6** & **58.2** & **64.8** & **80.2** & **85.8** & **188.4** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Our EgRet model outperforms pretrained (or non-parametric) baselines on the example retrieval task, indicating that exemplification cannot be solved by term matching or coarse query-context similarity alone. Pre-training GeRet on out-of-distribution examples from Books3 results in large improvements in recall@\(k\). Finally, including context to the right of the exemplifying unit significantly boosts performance.

 

Results & Analysis

We report results from baseline retrievers and our trained models in Table 4. We also conduct a human evaluation to compare retrieved examples from EgRet (**L**) to examples _generated_ by the state-of-the-art LFQA Routing Transformer model of Krishna et al. (2021).

### Automatic Evaluation

While all models outperform random baselines, our trained EGRET models Fan et al. (2019) consistently outperform both the lexical baseline (BM25) and other neural baselines. Among the baseline models, SBERT model, trained on MSMARCO dataset, consistently outperforms other models and DPR model lags behind the lexical baseline.7

Footnote 7: Models that outperform others in recall@\(k\) when \(k\) is small (less than 100) but underperform others when \(k\) is large (greater than 1000) typically score poorly in the average rank (e.g. ColBERT). More details about the variation in these retrieval evaluation metrics can be found in the Appendix §B.

Including context after the exemplifying unit improves recall:As a sanity check, we observe that including more context (**L+R**) significantly boosts recall for both EgRet and all baselines compared to including just context before the exemplifying unit (**L**). In addition to providing more constraints over the exemplifying unit, we also observe improvements on multi-sentence exemplifying units due to term matching; as this **L+R** setting is not particularly realistic, we analyze only the **L** configuration moving forward.

Pretraining improves example retrieval:Pretraining on out-of-distribution Books3 examples substantially boosts EGRET's performance, with the best left-context-only model achieving a recall@1 of 21.1% compared to 13.0% without pre-training. In fact, an EgRET model pretrained on Books3 _without_ fine-tuning on in-domain ELI5 data (19.3% recall@1) outperforms the ELI5-only EgRet. While Figure 1 shows that the distribution over exemplification types differs based on the dataset/domain, our results suggest that many aspects of exemplification apply generally to wide forms of writing, and that the pretrained Books3 EgRET could be useful for many other applications.

### Human evaluation of retrieved examples vs. generated examples

How do the examples retrieved by EgRET compare to examples generated by a state-of-the-art LFQA model? In theory, generative LFQA models can produce examples tailored to any input context; in practice, however, they struggle to generate relevant and informative examples. On the other hand, retriever models will always produce human-written examples, but it may not always be possible to retrieve a relevant example for an arbitrary heldout context. To explore this trade-off, we conduct a human evaluation by providing Mechanical Turk workers with an ELI5 question and context, and asking them to both _rank_ and _rate_ (on a 5 point Likert scale) three candidate exemplifying units: (1) the ground-truth; (2) the top-ranked retrieval from EgRet, restricted to only cases where this retrieval is _not_ the ground-truth; and (3) a generated output from the state-of-the-art c-REALM-RT model of Krishna et al. (2021).8

Footnote 8: This model is pretrained on the PG-19 dataset Rae et al. (2020) and fine-tuned on ELI5, conditioned on retrieved Wikipedia documents.

Task setup:In the ranking task, we ask workers to produce a ranking of the three choices (e.g., 1>2>3). We allow equality (e.g., 1=2>3) since multiple candidates can be equally valid for a given context. In the rating task, we ask workers to evaluate how well each example fits with the given context on a scale of 1 to 5. For both tasks, we collect three annotations per item for 100 total items, and we pay $0.35 per item for an estimated hourly wage of $21 per hour.9 While a completely fair comparison of EgRET to c-REALM-RT is infeasible due to differences in training objective and architecture, we choose to focus only on sentence-level exemplifying units that begin with _"For example"_. We provide the question, left context, and _"For example"_ marker to c-REALM-RT and decode using nucleus sampling with \(p=0.9\) until a sentence boundary (e.g., period) is generated. For the retrieved output, we use EgRET(Books3 + ELI5, **L**) since the RT model has access to only the left context.

Footnote 9: We restrict workers to those in English speaking countries who have completed at least 1000 HITs with an acceptance rate of 97%.

EgRet retrievals are preferred over generated exemplifying units:In both tasks, crowdworkers exhibit a clear preference for 

trieved by EgRet compared to those generated by c-REALM-RT. While both tasks are fairly subjective, as shown by the low interannotator agreement measured by Krippendorf's alpha, the results indicates that as of now, exemplification in LFQA is better handled by retrieval models than generative models, and that research into hybrid generation/retrieval LFQA models is a promising direction.

One limitation of our retrieval approach is that it will fail when the candidate set contains no relevant examples for a given context, which is not (at least in theory) an issue with generative models. However, we observe from our error analysis (Table 8) that both approaches at times produce seemingly relevant but incorrect examples. Furthermore, our human evaluations show that generative model fails to produce relevant examples for 79% of the contexts and consistently receive lower ratings than the retrieval model. This gap indicates that effectively incorporating retrieved information into the answer generation process is an important future research direction.

Our human study is coarse, judging the overall quality of the example, and future experiments could perform more fine-grained ratings of properties such as grammaticality and relevance.

## 5 Related Work

**Linguistic studies of exemplification:** Early work Kurohashi and Nagao (1994) studies automatic detection of discourse relations including exemplification. Several works have studied exemplification in the domains of academic writing and teaching Hyland (2007); Oliveira and Brown (2016); Triki (2021). They proposed the following categorization of examples: _general example_ (an instance of a general category); _analogy_ (a parallel or similar case); _counterexample_ (example that opposes or contradicts the anchor) and _extreme example_ (boundary cases, in the sense of being more of an unusual instance than a representative, generic case). During our initial investigation, we

\begin{table}
\begin{tabular}{l r r} \hline \hline  & **RatingSTQ (\(\uparrow\))** & **Krip. \(\alpha\)** \\ \hline c-REALM-RT & \(2.80_{0.775}\) & 0.058 \\ EGret Books3 + ELI5, L) & \(3.550_{0.636}\) & 0.125 \\ Ground-truth & \(3.70_{0.597}\) & 0.128 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Crowdworkers rate EgRet retrievals higher (on a scale of 1 to 5 of how well the exemplifying unit fits with the context) than the SOTA generative LFQA model.

\begin{table}
\begin{tabular}{c l l} \hline \hline
**Left contest** & **Ground-truth** & **EGRET-retrieved vs others** & **Analysis** \\ \hline Evolution is not a force towards the optimum, it’s a force saysr. & For example, if grass was poisonious, it would be better for us survival, as less animals would come eat i. & **IGRET-retrieved:** For example, we move incredible slowly when compared to the maximum speed allowed in the universe. Our EGRET-retrieved examplex was rated a 5/5 by all three crowdworkers but achieves lower ROUGE than an irrelevant example. \\ \hline – YouTube brain is alsoep and not paying any attention to your body so it ignores all these stimuli unless they become too hard to ignore. & For example if the touching turns to slapping, the talking units to yelling, or the light in the turns to yellow, or the light in the eyes. & **EGRET-retrieved:** For example, if you’re in a room with a clock ticking you don’t notice the ticking after a while. (**H**:4) \\ \hline – Multiple births mean less time per offspring. Each individual offspring therefore has a lower chance of survival, — Stern’s like the larger mammals tend to have **single births.** & **EGRET-retrieved:** For example, if you’re in a room with a clock ticking you don’t notice the ticking after a while. (**H**:4) \\ \hline – Multiple births mean less time per offspring. Each individual offspring thereafter has a lower chance of survival,— Stern’s like the larger mammals tend to have **single births.** & **EGRET-retrieved:** For example, in mammals, a typical filter will be one offspring per pair of nipsles as this is “as many individuals female can reasonably sustain [eERET-ERF]: For example, do, this is as close I can get to explaining in ELI5 terms. & The **EGRET-retrieved** example, if you are able to give them on a key entity from the context (summals) but fails to address the concept to be exemplified (“**single births”) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Instances where EGRET retrieves exemplifying units that are rated highly by Humans but have low ROUGE score with the ground-truth example (top); where model retrievals are rated as more meaningful than those generated by the _–READ-RT_ model (middle); and where _EGRET_ fails to retrieve a relevant example by relying too much on lexical overlap and _–REAL-MR_’ tails by producing an overly-generic output (bottom).

 

noticed that majority of the examples in our dataset are _general examples_, which aligns with the findings in Hyland (2007). The dominance of _general examples_ could also be due to the choice of our exemplification markers - all examples given by Hyland (2007) for _analogy_ have the exemplification marker "like", which we found noisy for automatic extraction. Li and Nenkova (2016) examine the closely-related _instantiation_ discourse relation, where one text span explains in further detail the events described in another text span.

Long-form question answering:Our work studies exemplified mainly within the task of long-form question answering (LFQA), which involves generating paragraph-length answers to open-ended questions. Previous work has approached this problem using retrieval-augmented generation Fan et al. (2019); Lewis et al. (2020), while Nakano et al. (2021) set up an interactive language model that learns LFQA through human interaction. Krishna et al. (2021) demonstrate that lexical overlap metrics such as ROUGE are not meaningful for this task. With a series of human annotations, Xu et al. (2022) study the discourse structures of long-form answers and identify exemplification as one of the functional roles commonly present in different types of long-form answers.

Neural retrieval models:Our EgRet retriever builds on recently-developed neural models that retrieve evidence documents for open-retrieval question answering Karpukhin et al. (2020); Guu et al. (2020) and fact-checking Samarinas et al. (2021). These models demonstrate superior performance compared to non-neural methods like BM25 Robertson et al. (1995); that said recent sparse/dense hybrid retrievers Luan et al. (2021) could be interesting to explore on our exemplificication task in the future.

## 6 Conclusion

In this work, we present the first computational study of exemplification in long-form question answering. We perform a detailed annotation over the use of exemplification across various domains and observe different distributions over complex exemplification types and units. While existing LFQA systems are conditional language models that do not give special treatment to exemplification, we propose to _retrieve_ examples based on their context instead of generating them. We develop EgRet, a simple dual encoder trained with a contrastive learning objective, that outperforms a diverse set of baselines on this task of example retrieval, which we can meaningfully evaluate using simple ranking metrics instead of unsuitable metrics like ROUGE. We hope that our work spurs researchers to consider separately modeling and evaluating the fine-grained linguistic and discourse phenomena found in LFQA data.

\begin{table}
\begin{tabular}{p{34.1pt} p{184.9pt} p{184.9pt}} \hline \hline
* Economic and Military and for Pakistan, Egypt, and Jordan. The goal is to have a new people in the midest who call usiles. Essential, we buy their cooperation. That cooperation is sometimes useful. & For example, when we killed Osama Bin Laden, we sent troops into Pakistan. Normally, countries don’t violate rooms from other countries. The Pakistani doll campina little, but they didn do anything about it. & **EGRET** performed an on-topic hypothetical example, which contradicts with the context. \\ \hline You don’t usually work on the same files because everything is split up between the departments. In haven’t used USD yet but I have encouraged tered the following workflow in different studios (using Maya). & For example: a character that has been hired by one or more, but not the same time) rigger goes to the animators. Every amitator works with the same character rig of the work is done with other students around the world who are made up of multiple departments.[...] \\ \hline \hline \end{tabular}
\end{table}
Table 8: Error analysis of retrieved and generated examples.
 



## Ethical Considerations

We make use of pretrained language models to both generate and retrieve text in this work. Representations from pretrained language models are known to cause ethical concerns, such as perpetuating racial or gender bias (Field et al., 2021; Gala et al., 2020). We advise using caution and adopting a post-processing strategy to filter potentially offensive text produced by pretrained language models before releasing text content to users. Additionally, we note that most existing LFQA datasets (including the ELI5 dataset used in this work) and benchmarks are collected from English text sources. We hope future works can explore the use of exemplification in other languages.

## Acknowledgements

We are grateful to the UMass NLP group for many helpful discussions. We thank Yapei Chang, Kalpesh Krishna, Katherine Thai for sharing their retriever models with us, as well as Tu Vu and Andrew Drozdov for sharing their compute resources. We would also like to thank the anonymous reviewers for their insightful feedback. SW and MI were supported by awards IIS-1955567 and IIS-2046248 from the National Science Foundation.

## References

* Ael (2006) Annelie Adel. 2006. Metadiscourse in l1 and l2 english.
* Bajaj et al. (2018) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset.
* Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In _Proceedings of the International Conference of Machine Learning_.
* Clouse (2013) Barbara Fine Clouse. 2013. _The student writer_. McGraw-Hill.
* Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELIS: Long form question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
* Field et al. (2021) Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A survey of race, racism, and anti-racism in NLP. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1905-1925, Online. Association for Computational Linguistics.
* Gala et al. (2020) Dhruvil Gala, Mohammad Omar Khursheed, Hannah Lerner, Brendan O'Connor, and Mohit Iyyer. 2020. Analyzing gender bias within narrative tropes. In _Workshop on NLP and CSS at EMNLP_.
* Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noao Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. _arXiv preprint arXiv:2101.00027_.
* Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. _ArXiv_, abs/2002.08909.
* Hyland (2007) Ken Hyland. 2007. Applying a gloss: exemplifying and reformulating in academic discourse. _Applied Linguistics_, 28:266-285.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In _North American Association for Computational Linguistics_.
* Kurohashi and Nagao (1994) Sadao Kurohashi and Makoto Nagao. 1994. Automatic detection of discourse structure by checking surface information in sentences. In _COLING 1994 Volume 2: The 15th International Conference on Computational Linguistics_.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association of Computational Linguistics_

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yin, Tim Rocktaschel, Sebastian Riedel, and Douve Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _ArXiv_, abs/2005.11401.
* Li and Nenkova (2016) Junyi Jessy Li and Ani Nenkova. 2016. The instantiation discourse relation: A corpus analysis of its properties and improved detection. In _NAACL_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, Dense, and Attentional Representations for Text Retrieval. _Transactions of the Association for Computational Linguistics_, 9:329-345.
* Meyer (1992) Charles F. Meyer. 1992. Apposition in contemporary english.
* Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browsers-assisted question-answering with human feedback. _CoRR_, abs/2112.09332.
* Oliveira and Brown (2016) Alandeom W. Oliveira and Adam Oliver Brown. 2016. Exemplification in science instruction: Teaching and learning through examples. _Journal of Research in Science Teaching_, 53:737-767.
* Petroni et al. (2020) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2523-2544, Online. Association for Computational Linguistics.
* Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In _International Conference on Learning Representations_.
* Reimers and Gurevych (2021) Nils Reimers and Iryna Gurevych. 2021. The curse of dense low-dimensional information retrieval for large index sizes. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 605-611, Online. Association for Computational Linguistics.
* Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gateford, et al. 1995. Okapi at trec-3. _Nist Special Publication Sp_, 109:109.
* Samarinas et al. (2021) Chris Samarinas, Wynne Hsu, and Mong Li Lee. 2021. Improving evidence retrieval for automated explainable fact-checking. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_, pages 84-91, Online. Association for Computational Linguistics.
* Thai et al. (2022) Katherine Thai, Yapei Chang, Kalpesh Krishna, and Mohit Iyyer. 2022. Relic: Retrieving evidence for literary claims. In _Association of Computational Linguistics_.
* Triki (2021) Nesrine Triki. 2021. Exemplification in research articles: Structural, semantic and metadiscursive properties across disciplines. _Journal of English for Academic Purposes_, 54:101039.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _ArXiv_, abs/1807.03748.
* Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _ArXiv_, abs/1706.03762.
* Xu et al. (2022) Fangyuan Xu, Junyi Jessy Li, and Gunsol Choi. 2022. How do we answer complex questions: Discourse structure of long-form answers. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_. Long paper.

 

Evaluation Interface on Mechanic Turk

Given a question, a partial answer (context) and three candidate examples, a worker is asked to rate all three candidate examples, according to their fit with the question and the given context.

[title=Prices://www.prices.com]
**Please read the questions, for the partitioned access, and data each embedding example according to its fit as an EM/EM/EUR for the answer**

**Please try to give enough confidence samples a different setting. F use cannot decide between two candidates examples, you may get from equal orific, however, please do NOT guess at some conditions like new time?**

**Question:** Why don 100% identify not more naive environment?

Answer: Thirdly in the present of intent he is not: The reason that 100% identify would not mean better understands it can be lost but the best performing run referring to only suggestion the context of user who are at most. **Bounded:**

[title=Rules://code.google]
**Rule:** the relative candidate examples (Brown, 

Variations in Recall@k

Table 4 shows that ColBERT outperforms DPR in various recall@k measures when \(k\) is relatively small (less than 100) and yet underperforms DPR in the average rank. Similar phenomenon occurs with EgRet-Books3-only and EgRet-ELI5-only too. We further computed the recall@k for all these models when \(k\) is very large (close to \(10,000\)). Fig. 3 and Fig. 4 show that despite their lower recall@k when \(k\) is relatively small, DPR and EgRet-ELI5-only yield higher recall@k when \(k\) is relatively large, compared to ColBERT and EgRet-Books3-only respectively. With better performance at the long tail (when \(k\) is relatively large), DPR and EgRet-ELI5-only also produce lower average ranks in Table 4.

Figure 4: EgRet-Books3-only gives higher recall@k compared to EgRet-ELI5-only when \(k\) is relatively small but lower recall@k when \(k\) is larger

Figure 3: ColBERT gives higher recall@k compared to DPR when \(k\) is relatively small but lower recall@k when \(k\) is larger





\begin{table}
\begin{tabular}{c c l p{142.3pt}} \hline \hline
* C, E, G
- with an added sixth
- above the root ) or as a first inversion A minor seventh chord ( the A minor seventh chord contains the notes A, C, E and G, but in this example, the C note, the third of the A minor chord, is in the bass )._

\begin{tabular}{p{79.8pt} p{142.3pt} p{142.3pt}} \hline \hline \multicolumn{1}{c}{**For example:**_My uncle's business is primarily plastics. They get cast offs, seconds, etc plastic from all kinds of US manufacturers. They then sort, filter and break down the plastic to the most basic starting point (often really small non died beads) and ship it to China. [...]_

\begin{tabular}{p{79.8pt} p{142.3pt}} \hline \hline \multicolumn{1}{c}{**For example**_!_ _i currently intern at a medical malpractice firm and if we were forced to do criminal defense I would actually be the most qualified one there to do so- at a firm where the youngest attorney still has 15 years of experience_} \\ \hline Books3 & Real & For example, people were told to imagine they would “Go forward 3 m, turn clockwise 90”, then go forward 3 m."_ \\ \hline Books3 & Hypothetical & When we cook together, I have to stay alert because she is always throwing a lemon at me—sometimes double down on acid and mix lemon juice with a little bit of vinegar to get the sunny sweet-sour note of the citrus along with earthly, apple, or wine notes of a vinegar for greater complexity \\ \hline \hline \end{tabular}
\end{table}
Table A1: Different types of annotated examples in the three datasets. The anchor and _example_ are highlighted.