Elaborative Simplification as Implicit Questions Under Discussion
Authors: Yating Wu
William Berkeley Sheffield
Kyle Mahowald
Junyi Jessy Li
Venue: EMNLP 2023 Main (2023)
Keywords: text simplification
elaborative simplification
questions under discussion

Abstract: Automated text simplification, a technique useful for making text more accessible to people such as children and emergent bilinguals, is often thought of as a monolingual translation task from complex sentences to simplified sentences using encoder-decoder models. This view fails to account for elaborative simplification, where new information is added into the simplified text. This paper proposes to view elaborative simplification through the lens of the Question Under Discussion (QUD) framework, providing a robust way to investigate what writers elaborate upon, how they elaborate, and how elaborations fit into the discourse context by viewing elaborations as explicit answers to implicit questions. We introduce ELABQUD, consisting of 1.3K elaborations accompanied with implicit QUDs, to study these phenomena. We show that explicitly modeling QUD (via question generation) not only provides essential understanding of elaborative simplification and how the elaborations connect with the rest of the discourse, but also substantially improves the quality of elaboration generation.

Elaborative Simplification as Implicit Questions Under Discussion
Yating Wa William Sheffield Kyle Mahowald Junyi Jessy Li
Ellctricca and Computer Engineering, *Linguistics The University of Texas at Austin
Lyating.wy, sheffieldw, mahowald, jessy]@utexas.. edu
Abstract
Simplified, Those factories are gone now. New companies have come thar need skilled workers with more education. New Haven youth want those jobs, but they do not have the education the skills. Implicit QUD: Why don't inn't people aequire the necessary shills?
Automated text simplification, technique use- ful for making text more accessible to people such children and emergent bilinguals, is often thought of as a monolingual translation task from complex to simplified text. This view fails to account for elaborative simplifica- tion, where new information is added into the simplified text. This paper proposes to view elaborative simplification through the lens of the Question Under Discussion (QUD) frame- work, providing robust way investigate what writers elaborate upon, how they elab- orale, and how elaborations fit into the dis- course context by viewing claborations ex- plicit answers to implicit questions. We in- troduce ELABUUD, consisting of 1.3K elab- orations accompanied with implicit QUDs, to study these phenomena. We show that explic- itly modeling QUD (via question generation) not only provides essential understanding of elaborative simplification and how the elabo- rations connect with the rest of the discourse, but also substantially improves the quality of elaboration generation.
Many do not have the money to get the training they need.
That is where New Haven Promisc comes in, It will make difference by paying for college. New Haven Promise is no one-way street.
Original. Factories have closed and their low-skill manufacturing jobs are long gone. The new companies town require workers with college degree or
advanced training, [...] The New Haven Promise is part of bigger plan to improve the city's economy.
example elaborative simplification, taken from Srikanth and Li (2021). Both simplified and original snippets are shown; elaboration added to simplified version is shaded blue. "[...]" in the original text refers to content deleted in the simplified version. This work focuses on already identified elab- orations in the simplified text, and introduces implicit questions under discussion ("implicit QUD", yellow box) to characterize and help generate the elaborations.
Figure I:
1 Introduction
This work instead focuses on elaborative sim- plification (Srikanth and Li, 2021), i.e., explain- ing or elaborating difficult concepts or content during the simplification process, illustrated in Figure 1. Although elaborations would add to the amount of content reader needs to pro- cess, psycholinguistic studies have established the benefit of elaborative modifications for L2 reading comprehension (Parker and Chaudron, 1987; Yano et al., 1994). However, deriving elaborative sim- plification is challenging: existing simplification models-bbcausee they are trained end-to-end translation models-do not actively generate elabo- rations and, when they do, tend to hallucinate (De- varaj et al., 2021; Srikanth and Li, 2021). Thus to make progress, we argue that explicit analysis and supervision is necessary. There has been little
Text simplification systems aim to lower the bar- rier of reading for a wider, more inclusive audi- ence, for instance, children (De Belder and Moens, 2010), emergent bilinguals (Taylor et al., 2022), and individuals with language impairments (Car- roll et al., 1998; Rello et al., 2013). While there has been abundant research in automatic text sim- plification (Siddharthan. 2014), recent data-driven efforts have focused on re-writing a sentence or passage into simpler language while preserving its meaning, often as a monolingual translation task using encoder-decoder models (Anhh-Manchego et al., 2020; Sun et al., 2021; Devaraj et al., 2021) editing models (Dong al., 2019; Agrawal and Carpuat, 2022).
Yating and William contributed equally.

---

work understanding what people choose elabo- rate, how they elaborate, and how the elaboration fits into the discourse context. Understanding these dimensions is crucial for developing better systems by giving us a framework for analyzing elabora- tions.
ically generate the QUDs. We train these mod- els using two QUD corpora, then fine-tune on ELABQUD: one setting where the model is ex- posed to the elaboration (Ko et al., 2022), and one where the model is not (Ko et al., 2020) follow- ing the expectation-driver model of QUD (Kehler and Rohde, 2017). The latter setting mimics the realistic scenario where the answer (i.e., the tual elaboration that aim generate) is not known prior to asking the questions. We show that expectation-drive questions, although often plau- sible and valid, tend to deviate more often from the exact direction of the annotated QUDs.
We propose a simple but powerful way of think- ing about elaborations: as answers to implicit ques- tions. Consider Figure I: the editor inserted "Many do not have the money to gel the training they need" explanation for the preceding sentence "they do not have the education the skills". This elab- oration did not exist in the original (unsimplified) document, and it can be thought as answering the implicit question "Why don't 't people acquire the necessary skills?"
Next, we plug in the generated questions prompts for GPT-3 model (Brown et al., 2020) to derive elaborations in a zero-shot manner. We show that compared with no prompt or generic prompts, QUD-driven elaborations are of sub- stantially higher quality and are typically more elaboration-like.
This approach has a long history in the Ques- tion Under Discussion (QUD) linguistics frame- work (Von Stutterheim and Klein, 1989; Van Kup- pevelt, 1995; Roberts, 2012; Benz and Jasinskaja, 2017; Ko et al., 2023); the QUD framework views each sentence the answer to implicit plicit question from prior context. Thus, our model for elaborative simplification is that, while simplify- ing text, editors implicitly ask questions, especially when difficult concepts are encountered. Elabora- tive simplifications are (explicit) answers to these (implicit) questions.
We release ELABQUD and code at https:// gi thub. fww///hffff//// elabQUD (copyright issues discussed Appendix C).
Background and Related Work
Earlier work related to elaborative simplification mostly focused on a specific type of elaboration, namely retrieving defi- nitions in lexical simplification (Damay et al., 2006; Kandula et al., 2010; Eom et al., 2012). More re- cently, Srikanth and Li (2021) gathered general dataset for elaborative simplification, all of which were derived from the Newsela dataset (Xu et al., 2015), a corpus of professionally simplified news articles. The elaborations were obtained by first finding sentences in the simplified version of a doc- ument that failed to align to the original version. These candidates were then manually filtered via crowdsourcing to check whether they appeared in context window in the original version.
Elaborative Simplification
With this view, we formulate elaborative simpli- fication as a two-step process: question generation and question answering. The question generation step models "what is elaborated?" by means of recovering the implicit QUDs, which will guide the question answering model for the generation of the actual elaboration.
To support this, we present ELARQUD, novel corpus of implicit QUDs that are answered by the 1299 elaborations collected by Srikanth and Li (2021). In addition, ELABQUD also contains a finer-grained layer of annotation specifying which concepts the elaboration was about in the earlier context of the same document, which we call the targets of claboration. We find authors elaborate both about entities and events, and that elaborated concepts tend to be composed of less frequent words. We also analyze the types of questions to determine how authors elaborate, and find that elaborations often involve causal reasoning and ex- planations of entities.
Srikanth and Li (2021) found that only some of the inserted elaborations were definitions; many were contextually dependent explanations and clar- ifications (e.g.. Figure 1). In a few cases, editors would choose to add additional facts related to an full
event. This rules out definition retrieval solution the elaboration generation task. Addi- tionally, Srikanth and Li (2021) showed that vanilla use of an auto-regressive language model could generate ersatz "elaborations" that deviate from the document context, hallucinate, and/or do not
Using ELABQUD, we first train and evaluate question generation models attempting to automat-

---

actually explain the content.
Read elaboration (highlighted) and context
I The Giron brothers left Honduras in February 2013. [...] 4 It took the group 11 days to travel from Honduras the United States, They traveled all the way north through Mexico until they reached the Rio Grande River.
Questions Under Discussion The QUD frame-
work is way to reason through discourse by view- ing it as continuously posing and answering ques- tions (Von Stutterheim and Klein, 1989; Van Kup- pevelt, 1995; Roberts, 2012; Benz and Jasinskaja, 2017). In dialogues, participants actively resolve the current QUDs; however in monologues, the QUDs are implicit. Thus in this work we recover the implicit QUD that was triggered in context prior to the elaboration that answers it. Recent work has begun explore the annotation (De Kuthy al.,
The river divides the United States from Mexico. At the border, the guide put the Girons and other teens small rafi. He paddled them across the river to Texas. Once they were in the United States, their guide left the teens by themselves.
Formulate question
Why important for then reach the Rio Grande?
What concept, and in which sentence, does the claboration explain?
2018; Hesse al., 2020; Westera et al., 2020; Ko al., 2020, 2022) and automatic generation (Ko et al., 2023) of QUD structures. Our data collection process aligns with Ko et al. (2022)'s annotation paradigm for QUD recovery, wherein each sen- tence of a news document is considered an answer to QUD from prior context. In our case, each claboration is the answer to QUD that comes up when the need for more explanation arises.
Sentence 5, "the Rio Grande River"
Is this elaboration an organizational sentence??
Yes
No
Figure 2: Annotation procedure ELABQUD.
This section presents ELABQUD and its anno- tation. ELABQUD contains annotated implicit QUDs for all 1,299 elaborations Srikanth and Li (2021), along with their anchors and targets.
Despite decades of rich linguistic research on QUD, large-scale, task-oriented application of this framework is still in its infancy, with very recent ef- forts studying question generation (Ko et al., 2020) and answering (Ko et al., 2022), conditional gener- ation (Narayan al., 2023), and decontextualiza- tion (Newman et al., 2023). The goal of this work is to lay a foundation connecting elaborations with QUD: given a marked elaboration, using QUDs to characterize what and how the elaboration should be generated. Although this work does not address when an elaboration should be added (which we leave for future work), the QUD framework pro- vides natural, interactive, and personalized way to think about elaborations: the QUD will be explic- itly provided by the reader when they think more explanation is needed.
3.1 Annotation task
Our annotation process is depicted in Figure 2. We adapt Ko et al. (2022)'s 's annotation paradigm for less cognitive load since we focus on one elabora- tion at time, and we introduce task-specific mod- ifications. Specifically, for a given elaboration E. annotators were provided with context window of five sentences preceding E, E itself, and the three sentences succeeding E. We show five prior sen- tences as in Srikanth and Li (2021), who found that this is usually sufficient and effective to establish common ground. The three succeeding sentences were shown to provide a more rounded picture of the document, although this information is not nec- essary for the annotations. For case of reading, the elaborations were highlighted in yellow.
3 Implicit QUDs: what questions do elaborations answer?
Next, the annotators were asked to create ques- tions which were (a) plausibly generated by con- sidering only the context, and (b) for which the elaboration provides an answer. To better simulate the real elaboration simplification process where E is unknown when the question is asked, we ask annotators avoid including content specific to E
For a simplified document with sentences D {S1.... 1, St+1.... where sentence i is an elaboration (i.e., F = 5;), we aim to recover the implicit question under discussion such that E answers Q. We further define the target T of the elaboration, i.e., E elaborates or explains T. The sentence that contains T' is called the anchor sen- tence of Q, and can be taken to mean that Q arose from that anchor (Ko et al., 2022).
At the edge of documents, annotators were provided as many lines as possible.

---

in the questions.
2000
We then ask the annotators to identify the target T that E elaborates. After the first round of anno- tations both by the authors and by crowdsourced workers, we found that, in most cases, both the anchor sentence and T were in the sentence imme- diately preceding the elaboration (i.e., T E S when E S.). and that with multiple analyses S usually provided the most straightforward T. Thus, we also highlighted S in the interface. However, when asking annotators to provide T, we did not prime them further to S;-1, and allowed them to highlight T any subsequence in the prior context that they deem plausible.
Freq.
1500
1000
500
-2
Distance from Anchor to Elaboration
Figure 3: Distribution of distance from anchor sentence to elaboration.
3.2 Analysis
Are questions similar for the same elaboration? We report BERTscore (Zhang et al., 2019) be- tween each pair of questions. We include both raw and rescaled? values. Annotator questions have BERT F1 of 0.922 (rescaled 0.538). Com- pared to randomly-paired questions from the same article (F1 0.879; rescaled 0.281), these values in- dicate high similarity between questions from dif- ferent annotators for the same elaboration when compared to random question pairings.
Finally, we noticed that some sentences are or- ganizational: they are added to provide discourse cues that describe the way the next few sentences are organized, e.g., the elaboration text E in the ex- ample below. We included an additional question to mark these.
For the anchor sentence, measure agreement based on the distance from it the elaboration, meaning a distance of 3 indicates the anchor sen- tence occurs lines before the elaboration, while a distance 11 indicates the anchor sentence occurs in the line after the elaboration. The distribution of distances is provided in Figure 3; most anchor sentences immediately precede E. We observe a Fleiss' kappa (Fleiss, 1971; Randolph, 2005) of 0.6083 (percentage agreement 69.9%), indicating substantial agreement (Artstein and Poesio, 2008). Additionally, the selected targets overlap 62.4% of the time, reflecting that annotators agree on what is being elaborated most of the time.
Investigators say Kellogg tried copy the watermark,
E: Here's how they say he did it. First printed the front side money on one piece paper. Next, [...]
The primary annotation task had two stages. The first stage involves three expert anno- tators at our institution who each annotated the same 30 elaborations. From these, we identified a representative set of six elaborations for which all annotators agreed on the target T' and asked se- mantically equivalent questions to form worker qualification dataset. Their feedback was also used enhance instructions and guide minor improve- ments to the annotation interface.
Annotators
What is elaborated? Although the average tar- get is 4.54 tokens long, there is considerable vari- ation (standard deviation of 3.06). Nouns are the most frequent part speech in the targets (7452), specifically plural nouns (1589) and proper nouns (1449) out of a total number of 13153 tokens. These are often the targets of definitions, or some- thing along those lines. For instance, the first ex- ample in Table I has entity target that explains more about the entity without being explicit def- inition. Moreover, we surmise significant subset of elaborations focus on entities because 31.4% of all targets contain proper nouns.
The full dataset was then collected via crowd- sourcing using Amazon Mechanical Turk. Annota- tors that had previously worked with institution on other complex document comprehension tasks were asked annotate the six qualification elab- orations as a qualification task. Responses were manually inspected, and those that matched the ex- pert target annotations and gave highly similar or reasonable alternative questions were qualified. In total, 8 workers were approved. They were paid at rate of over $10 per hour. Each elaboration notated by annotators (with subset of 280 anno- tated by annotators); in total, collected 2,878 questions across the 1,299 elaborations in Srikanth and Li (2021). The collected questions had an av- erage length of 8.80 tokens dst 3.25).
"Rescaling is provided by the original authors to improve model interpretability as the original scores are often close "potentially hecause of the learned geometry of contextual embeddings" (Zhang et al., 2019).

---

Question Type Definition
Example from ELABQUD
Concept (34%); Asking for definition of an concept.
Anderson became interested in people like Landa when she noticed something strange about call center near her house. [Q: What do call centers do?) E: Workers call centers help people The government is split into two parties that ofhen have different political beliefs. IQ: What is an example of one of these porties?] E: One party the Democrats.
the phone.
Example (16.2%) Asking for example(s) or instance(s) of event or concept.
Consequence (13.9%); Asking for the conse- quencees or results of an event.
The tightropes that Wallenda walks across go between buildings, hundreds of feet above the ground. [Q: What if he falls?] E: There are no nets to catch him if he falls.
Cause (12%) %):; E: Asking for the cause or reason for an event or a concept.
But not many countries support Obama's 's plan to fire missiles at Syria. [Q: Why they heing unsupportive.] E: Some are worried ahout getting into another war in the area without knowing the facts.
Procedural (8.1 %): Asking for the procedures, tools, methods by which certain outcome achieved.
The drone safely flew above the Atlantic Occan and landed aircraft carrier called the George H.W. IQ: How did the drone navigate way aireraft carrier?] E: It given special directions from satellites above the earth.
Table I: Top question types, their definitions from Cao and Wang (2021), and examples.
While many targets comprise noun phrases, 48.99% of targets include a verb, indicating that writers elaborate on events well entities. Take, for instance, the organization example (1) stated earlier. In this example, the target copy the water- mark contains a verb and the elaboration focuses on the event copying the watermark rather than Kellogg or the watermark itself.
elaborations with expert-annotated discourse rela- tions on small set of 40 examples. We observe intuitive correspondences between discourse rela- tions and question types, detailed in Appendix A.
Question generation
With the QUD framework, elaborative simplifica- tion is two-step process: (1) given context C' = S S2, S2..., ..... S prior to
We also found that authors usually elaborate on less frequent words. We measured this using log frequency per million words from the SUBTLEX- US (Bryshaert et al. (2012), 2015 release) cor- pus. The average log frequency values (per million words) for targets is 1.72, significantly lower than the document average of 2.46 (by an independent- samples t-test, -34.5, p .00001).
the elaboration E St, generate a question Q to recover the implicit QUD by modeling P(q[C). (2) Given C and Q, generate elaboration E by modeling P(e|C; Q).
This section experiments with question genera- tion (QG) models for step (1). We explore three different settings varying how explicitly the model sees the elaboration target T and the anchor sen- tence, and establishing upper bound where the model is exposed to the gold "answer" E.
What types of questions are asked? To exam- ine the types of questions, we classify the questions collected using the taxonomy and model from Cao and Wang (2021). In Table 1, we show the top 5 question types in ELABQUD along with exam- ples. The implicit QUDs reveal that in most cases, the elaboration is explaining a concept (34%), pro- viding explicit causal reasoning by describing the cause (12%) consequences (13.9%) of event,
4.1 Models
Oracle setup: QG model sees E. Knowing the answer would inform QG model what questions to ask. Although our target model will not see the answer (as it generating a question used in-turn to generate the answer/elaboration), we can use such a QG model as a silver upper-bound on QUD gen- eration. Here we repurpose the DCQA dataset (Ko et al., 2022) for question generation. DCQA con- sists of 22K questions from ~600 news articles; these questions are implicit QUDs elicited from annotators where they considered each sentence of these articles as an answer to a question. Each question associated with an anchor sentence that triggers the question (the anchor sentence contains
providing an example (16.2%), or describing a com- plex process (8.1%). Other question types (c.g.. verifying the truthfulness of concept, compari- son among multiple events, or asking about the extent of an event) are rare, indicating that the com- municative goal of an elaboration in the Newsela dataset is to provide an explanation when reasoning is deemed difficult for children.
We additionally present an analysis connecting

---

the target T but DCQA does not annotate T) and an answer sentence. In our case, we include all sentences prior to E, along with E, to see how they help compose questions about E.
BERTS
BLEU-4
DCQA-base DCQA-ft
0.915 0.494
0.323
0.911 / 0.474
0.313
INQ-GoldT-base
0.901 / /0.414
0.253
We first fine-tune GPT2-medium (Radford et al., 2019) on DCQA with the imput consisting of prior context C, the anchor sentence, the answer sen- tence E. and annotated question Q with special delimiters separating each part. We call this model DCQA-base. We then fine-tune DCQA-base on ELABQUD, which we call DCQA-ft. We refer readers to Table 7 (Appendix) for listed view of model inputs to all systems.
INQ-GoldT-ft
0.908 / 0.453
0.295
INQ-PredT
0.902 / 0.421
0.260
Table 2: BERTScore (F) rescaled and BLEU-4 for generated questions.
The target prediction model was first trained on INQUISITIVE then fine-tuned on ELABQUD. 'I the question generation model, we replace the gold target in INQ-GoldT-ft with the predicted target (for both training and testing), with the rest of the setup identical to INQ-GoldT-ft.
Practical system: QG model does not see E. Realistically, since E is what we eventually want to generate, the QG model cannot not be exposed to it. This paradigm fits with the expectation-drivan ap- proach QUD (Kehler and Rohde, 2017), where the questions are more curiosity-dris driven and are asked without seeing upcoming context.
Settings We use the same train/validattion/esst splits as in Srikanth and Li (2021). All model input formats and hyperparameters are tabulated in the Appendix, Table 7.
Thus we train our QG model using the INQUIS- ITIVE dataset (Ko et al., 2020), the largest ques- tion generation dataset annotated in line with expectation-driven approach. INQUISITIVE con- sists of ~99 questions elicited as the reader sees the first 5 sentences of news article one by one (without access to the document as a whole). IN- QUISITIVE also includes target annotation in the anchor sentence where the question was raised; this allows us to experiment with models that explicitly predicts the target T.
4.2 Results
Automatic evaluation We first evaluate gen- crated questions with two automatic measures, BERTScore (Zhang et al., 2019) and BLEU (Pap- ineni et al., 2002), comparing the generated ques- tions with human annotated questions.
For SeRTS we include both the unsealed version and the rescaled version. The results are shown in Table 2. It is clear that our DCQA- based oracle models, exposed to the elaboration F,, performs better than INQUISITIVE-ben based mod- els. Fine-tuning with ELABQUD does not help with the oracle setup but improves substantially for INQUISITIVE-based models. INQ-PredT,, which predicts the target span, shows a drop in perfor- mance in line with the observation Ko et al. (2020), though still better than taking INQ-GoldT- base out-of-the-box.
Specifically, our model INQ-GoldT-base is from Ko et al. (2020), a GPT-2 medium model fine- tuned on INQUISITIVE. The input to this model includes all sentences prior to the anchor sentence, the anchor sentence itself including the gold target span 7' marked, and the annotated question Q with special delimiters separating each part.² 3 We then fine-tune this model on ELABQUD, which we call INQ-GoldT-ft.
Human evaluation We further perform human evaluation across three systems, taken from the stronger versions of each group: DCQA-base, INQ- GoldT-ft, and INedP.e We evaluate questions with a framework adapted from the QUD human evaluation schema of Ko et al. (2023); annotators judge questions along two criteria:
Our second INQUISITIVE model, INQ-PredT, involves a pipeline approach that first predicts T.
We following the same setting as Ko et al. (2020): we train a distill-bers wel---bertttaasse t-uncased model with a modified SQuAD-QU DA format.
We do not predict the anchor sentence; at test time, the annotated anchor sentence is used. Anchor prediction is noisy (Ko et al.. 2022). Since the overwhelming majority of the anchor sentence is the sentence preceding E' (Figure 3). we believe this has a limited effect on our conclusions while leading to hetter controlled experiments. We leave anchor prediction for future work.
(1) Is the question reasonable to ask given the current context? That is, is this a valid/reasonable
"Following the evaluation setup of (Ko et al., 2020): the span prediction model has a exact match of 48.05% and precision of 83.6% on our test set.

---

Reasonable?
Answered?
BERTScore
BLEU
Human
Yes
88
89
Context-only
0.886 0.322
0.200
No
12
11
Generic
0.877 0.270
0.166
DCQA
Yes
78
67
Human question
0.381 0.896/ 0.361
0.244
No
22
33
DCQA- base
0.894/0..0
0.248
DCQA-ft
0.891 / 0.353
0.226
INQ-GoldT-ft
Yes
42
18
No
58
82
INQ-GoldT-
0.880 / 0.288
0.165
INQ-GoldT-ft
0.880 / 0.288
0.178
INQ-Pred
Yes
INQ-PredT
0.879/0.200
0.172
No
88
Table 4: BERTScore (F rescaled F) and BLEU-4 for GPT-3 generated elaborations given different prompts.
Table 3: Human evaluation on generated questions; % questions marked yes/no for each criterion.
QUD having read so far?
5.1 Setup
(2) Is this question answered by the elaboration? For both criteria, annotators mark "Yes" (allows minor spelling and grammar issues for (1)) or "No".
We use GPT-3 (Brown et al., 2020) for this task due to its vast text generation and open open-domain question-answente answering capability (Liang et al., 2022). Specifically,
Two undergraduate annotators evaluated a ran- dom sample of 50 questions generated by these three models along with the human annotated ques- tions, with a total of 200 questions.
text text-davinci-005 (tempera- ture O. max # tokens 128) with the following condi- tions, giving context consisting up to sentences prior to the elaboration (this context window fol- lows the same setup as Srikanth and Li 2021):
They agree 70.0% of the time for criterion 1 and 79.5% of the time for criterion 2. Shown in Table 3, annotators found human questions of the highest quality along both criteria, followed by DCQA-base, then INQ-GoldT-ft, and finally INQ- PredT. This is in-line with the automatic evaluation results. Interestingly, annotators report that both INQUISITIVE models perform worse on criterion 2 than 1, indicating that some of these questions may be valid QUDs but do not match the direction of the human elaboration. Consider the following elaboration in context:
(1) Context only: the prompt consists of only
the context;
(2) Generic: concatenate the context with
" Pleass explain the last sentence in simple terms:" as a generic instruction;
(3) QUD: we formulate the input as Con- text: <context>, Question: <question>, Answer: We use questions both annotated questions from ELABQUD and ones from our question generation systems.
(2) Should kids play tackle foothall? Foothall is rough game. E: Players get bounced around.
5.2 Results
Automatic evaluation We first report automatic evaluation results in Table 4, using BERTScore and BLEU with the gold elaboration as reference. The best performance is achieved with QUD setup. positive indication that hypothesis holds: QUDs inform elaboration generation. However, this is more true for good QUD's generated by annotators, or by the DCQA models that were ex- posed to the elaboration itself. This result echoes what we found in Section 4.2: the INQUISITIVE models, although they often generate good QUDs, may not necessarily generate the QUD for the elab- oration that the professional editor had in mind (as both BERTScore and BLEU compares the hypoth- esis with reference). These cases lead worse automatic measures compared to context-only set- tings, indicating a challenging future direction for QUD generation. Interestingly, we also note that
A QUD like Why is football rough game? makes the most sense for the actual claboration "Players get bounced around", but a question such as the one generated by INQ-GoldT-ft, What happens to play- ers who get hurt playing football?, is not answered even though it is valid QUD.
5 Zero-shot elaboration generation
Finally, we experiment with the utility of questions on elaboration generation, i.e., task (2) in Section 4: given C' and Q, generate elaboration E by model- ing P(e] C, Q). Our hypothesis is that good QUD should be able to guide strong language model towards generating better elaboration than with- out such guidance, the sense that the elaboration should be more on-topic, and more frequently an explanation rather than simply continuing a story.

---

"That's when learned was never going write a real novel because could not think way." The Nobel Pirze 112 years old. Munro the 13th female literature winner. Munro grew up in Wingham, a traditional Canadian lown west of Toronto. She lended to write about her experience there and her life after social revolution the 1960s. Elaboration: was a time when people began to break free of traditional roles.
Context:
Context only: "I'm very good person ask ahout the of the novel," she said.
The last sentence is saying that Munro usually writes about her life in the small town she grew up in, and her life after the social revolution of the 1960s.
Generic:
Human question What happened daring the social revolution of the 1960s?:
The social revolution of the 1960s was a time of great change and upheaval.
DCQA-ben bass What type of social revolution was she talking about?:
The social revolution of the 1960s was a time of great change in many Western societies.
INQ-GoldT-ft What happened after the social revolation of the 1960s?:
Munro continued to write about experiences Wingham and her life after the social revolution of the 1960s.
INQ-PredT Who caused the changes in the community?: is not clear who caused the changes in the community that Munro writes about.
Table An example of GPT-3 generated elaborations.
using generic instruction does not yield better results than instead providing no instruction and only the context itself.
Elaboration-like
Coherence
#1
#2
#1
#2
Gold
27
31
28
31
Context-only Generic
16
14
Manual evaluation We additionally perform hu- man evaluation on the generated elaborations across these different prompts. In this setup, mimic how elaborations would happen organically in simplified document: reader would not have access to the QUD but only to the generated elab- oration, directly continuing prior discourse. hu- man evaluation would also reveal whether models generate elaborations that do not follow the exact direction from the document but are nonetheless good and plausible elaborations, aspect that is not captured by the automatic measures.
3
2
3
Human question
19
21
14
DCQA-base INQ-GoldT-ft INQ-PredT
16
22
21
7
10
Table 6: Human evaluation of generated elaborations. of times each system output ranked or #2 based on how elaboration-like and coherent the generation is, independently.
"the social revolution of the 1960s". Encourag- ingly, elaborations generated by human questions (and DCQA models) are ranked 1st most frequently (after the gold elaborations) in both criteria; this establishes the utility of good QUDs. For the INQ- style models, we see clearer degradation in co- herence despite them scoring well on Elaboration- like. We find that off-topic question, like the one produced by INQ-PredT in Table 5, can easily throw off GPT-3. Generally, the generic-prome and context-only elaborations are not similar to the human elaborations unless it is a description or definition would obviously come next. As such, the claborations generated without QUDs cannot replicate more sophisticated human elaborations. where those generated with QUDs can.
Specifically, we provide two linguistics student annotators with a randomly sampled subset of 50 instances from the test set. The annotators were shown up to sentences of prior context, then elab- orations from GPT-3 as well as the original human claboration from the document. These elaborations are randomly ordered. The annotators were asked select and rank the top 2 elaborations indepen- dently along two criteria: (1) coherent continuation of discourse; (2) elaboration-like or explanation- like, rather than providing new information for story continuation (Srikanth and Li, 2021).
Table 6 shows that informative and on-topic claborations, and so ranked as highly elaboration-like Take the context- only generation in Table 5; while it matches in style and is very fluent with the text (a very reasonable next line and quote from Munro), it is completely off-topic from the true elaboration, which describes
produce more
Conclusion
This paper tackles the task of generating clabo- rations during text simplifcation. We adopt the

---

view of the Questions Under Discussion (QUD) discourse framework, and model elaboration an- swers to implicit QUDs. We present ELABQUD, a dataset of annotated QUDs for elaborative simplifi- cation. We experiment with a question generation + claboration generation pipeline. Results show that good QUDs provide valuable cues for zero- shot elaboration generation.
(TACC)² UT Austin for many of the results within this paper. We also acknowledge Kathryn Kazanas and Keziah Kaylyn Reina for their help with the manual evaluation of generated questions and elaborations.
References
Sweta Agrawal and Marine Carpuat. 2022. An imita- tion learning curriculum for text editing with non- autoregressive models. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 7550-7563.
7 Limitations
This paper focuses on how to generate claborations, rather than when to do so. Namely, assume that we know which sentences constitute elabora- tions using Srikanth and Li (2021)'s s dataset. We leave the when question to future work, noting that sentence-level elaboration is infrequent among the articles analyzed by Srikanth and Li (2021). At the same time, what constitutes difficult content is subjective reader-specific, Future work can explore using QUD for elaborative simplification in an interactive manner. Additionally, the space of possible QUDs given context large, posing chal- lenges to INQUISITIVE-based systems for future work.
Fernando Manaheanceego, Carolina Searton, and Lu- cia Specia. 2020. Data-driven sentence simplifica- tion: Survey and benchmark. Computational Lin- guistics, 46(1):135-187.
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computa- tional linguistics, 34(4)5555-596
Katherine Atwell, Junyi Jessy Li, and Malihe Alikhani. 2021. Where are we discourse relation recogni- tion? In Proceedings of the 22nd Annual Meeting of the Special Interest Group Discourse and Dia- logue, pages 314-325.
Another challenge with generating claborations is inherit to elaborations themselves: because they contain information not present in the original text, they are hallucinations. It will be important to analyze the difference between helpful elaborations and undesirable hallucinations, and we leave this to future work.
Anton Benz and Katja Jasinskaja. 2017. Questions un- der discussion: From sentence to discourse. Dis- course Processes, 54(3):177-186.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
We also note that we focused here on English text of particular genre (news), and that results may not generalize to other languages or other gen- res of text.
Finally, acknowledge that the landscape of LLMs are rapidly changing every few months, and we have not experimented with the strongest mod- els (i.e., GPT-4). However, the space of possible elaborations prevents unconstrained generation; the utility of QUD is exactly to point to what is elab- orated. As shown with our results, both question generation and claboration generation benefit from stronger language models, hence we optimistic about what stronger LLMs will bring to elaborative simplification with QUDs.
Mare Brysbaert, Boris New, and Emmanuel Keuleers. 2012. Adding part-of-speech information the subtlex-us word frequencies. Behavior research methods, 44(4):991-997.
Shuyang Cao and Lu Wang. 2021. Controllable open- ended question generation with a new question type ontology. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the Ithh International Joint Conference on Nat- ural Language Processing ( (Volume 1: Long Papers). pages 6424-6439,
Acknowledgements
This research was partially supported by National Science Foundation (NSF) grant IIS- 214547. We acknowledge the Texas Advanced Computing Cen-
hittps://wwe tacc. utw.as.edu

---

John Carroll, Guido Minnen, Yvonne Canning, Siob- han Devlin, and John Tait. 1998. Practical simpli- fication of english newspaper text assist aphasic readers. In Proceedings of the AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology, pages 7-10.
Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation clas- sification: We need to talk about evaluation. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 5404- 5414.
Jerwin Jan S Damay, Gerard Jaime D Lojico, Kim- berly Amanda L Lu, D Tarantan, and E Ong. 2006. SIMTEXT: Text simplification of medical literature. In Proceedings of the 3rd National Natural Lan- guage Processing Symposiun- Tools and Resources, pages 34-38.
Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Dur- rett, and Junyi Jessy Li. 2020. Inquisitive question generation for high level text comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 6555.
Language
Jan De Belder and Marie-Francine Moens. 2010. Text simplification for children. In Prroceedings of the SIGIR workshop on accessible search systems, pages 19-26.
Wei-Jen Ko, Cutter Dalton, Mark Simmons. Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2022. Dis- course comprehension: A question answering frame- work to represent sentence connections. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 11752- 11764.
Kordula De Kuthy, Nils Reiter, and Arndt Riester. 2018. Qud-based annotation of discourse structure and in- formation structure: Tool and evaluation. In Pro- ceedings of the Eleventh International Conference on Language Resources and Evaluation.
Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srini- vas, Greg Durrett, and Junyi Jessy Li. 2023. Dis- course analysis via questions and answers: Parsing dependency structures of questions under discussion. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11181-11195.
Ashwin Devaraj. lain Marshall, Byron C Wallace, and Junyi Jessy Li. 2021. Paragraph-level simplification of medical texts. In Proceedings of the 202/ Con-
ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4972 4984.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan. Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R'e. Diana Acosta-Navas. Drew A. Hudson, E. 7elikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha. Niladri S. Chatterji. O. Khat- tab, Peter Henderson, Qian Huang, Ryan Chi. Sang Michael Xie, Shibani Santurkar, Surya Gan- guli, Tatsunori Hashimoto. Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. ArXiv, abs/2211.09110.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. Editnts: An neural programmer-interpeee interpreter model for sentence simplifi- cation through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3393-3402.
Soojeong Fom, Markus Dickinson, and Rebecca Sachs. 2012. Sense-specific lexical information for reading assistance. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 316-325.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Shashi Narayan, Joshua Maynez, Reinald Kim Am- playo, Kuzman Ganchev, Annic Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023. Conditional generation with a question-answering answering blueprint. Transactions of the Association for Computational Linguistics, 11:974- 996.
Christoph
Hesse, Anton Benz, Maurice Langner, Fe- lix Theodor, and Ralf Klabunde. 2020. Annotating quds for generating pragmatically rich texts. In Pro- ceedings of the Workshop on Discourse Theories for Texi Planning, pages 10-16.
Sasikiran Kandula, Dorothy Curtis, and Qing Zeng- Treitler. 2010. A semantic and syntactic text sim- plification tool for health content. In AMIA annual symposium proceedings, pages 366-370.
Benjamin Newman, Luca Soldaini, Raymond Fok, Ar- man Cohan, and Kyle Lo. 2023. A controllable QA- based framework for decontextulivattion In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing.
Andrew Kehler and Hannah Rohde. 2017. Evaluat- ing expectation-driven vusstionnunder--iicussiin model of discourse interpretation. Discourse Pro- cesses, 54(3)-219-238
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the

---

40th anmual meeting of the Association for Compu- tational Linguistics. pages 311-318.
Bonnie Webber, Rashmi Prasad, Alan Lee, and Ar- avind Joshi. 2019. The pemn discourse treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania, 35:108.
Kathryn Parker and Craig Chaudron. 1987. The effects of linguistic simplifications and elaborative modifi- cations on 12 comprehension. Technical report, Uni- versity of Hawaii working papers in ESL.
Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. TED-Q: TED talks and the questions they evoke. In Proceedings of The 12th Language Re- sources and Evaluation Conference, pages 1118- 1127.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC"08).
Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification search: New data can help. Transactions of the Asso- ciation for Computational Linguistics, 3:283-297.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Technical Report.
Yasukata Yano, Michael H Long, and Steven Ross. 1994. The effects of simplified and elaborated texts on foreign language reading comprehension. Lan- guage learning, 44(2):89-211,
Justus J. Randolph. 2005.
Free-maree tree-marginal multirater kappa (multirater k[ffre]]): An alternative to fleiss' fixed-marginal multirater kappa.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. International Conference on Learning Representations.
Luz Rello, Ricardo Baeza- Yates. Laura Dempere- Marco, and Horacio Saggion. 2013. Frequent words improve readability and short words improve un- derstandability for people with dyslexia. In IFIP Conference on Human-Compute Interaction, pages 203-219.
Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Seman- rics and pragmatics, 5:6-1.
Advaith Siddharthan. 2014. survey of research on text simplification. ITL-IIntenational Journal of Ap- plied Linguistics, 165(2):259-298.
Neha Srikanth and Junyi Jessy Li. 2021. Elabora- tive simplification: Content addition and explana- tion generation in text simplification. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 5 5123-5137.
Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-level text simplification: Dataset, crite- ria and baseline. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 7997-8013.
Zachary W Taylor, Maximus H Chu, and Junyi Jessy Li. 2022. Text simplification of college admis- sions instructions:
professionally simplified and verified corpus. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 6505 6515.
Discourse structure, top- Jan Van Kuppevelt. 1995.
icality and questioning. 31(1):109-147.
Journal of linguistics,
Christiane Von Stutterheim and Wolfgang Klein. 1989. Referential movement in descriptive and narrative discourse. North-Hollane Linguistic Series: Lin- guistic Variations, volume 54, pages 39-76.

---

A Analysis of discourse relations
Dypophon
While QUDs provide fine-grained information about the goal of each elaboration, we comple- ment this view by examining the discourse rela- tions between an elaboration and its prior context (S We use the relation taxonomy from the Penn Discourse Treebank (Prasad et al., 2008; Webber et al., 2019), a structural-neurg frame- work that lays out the discourse relations between two text spans (i.e., arguments) including temporal, comparison, cause, etc.
Natoi
Ex Sabainnio
Is dildd
Coancssion
Co
C Coodiio
TSyrch
16
Figure 4: Relation distribution (%) PDTB-3 and
sample of 40 agreed elaborations in ELABQUD.
Since most of the elaborations inter- sentential implicit discourse relations that are still challenging for models to identify automat- ically (Atwell et al., 2021), we randomly sampled 51 elaborations for two expert linguists to annotate using the PDTB-3 (Webber et al., 2019) level-2 taxonomy. The two experts agreed for 40 of those, which we use in this analysis.
Procedural Q: Expansion Manner, Expansion. Expansion.Conjunciin Conjunction
Overall, we observe relatively high correlation between the type of questions and the discourse re- lations connecting an elaboration and its preceding context; both informative in the type of content present in elaboration.
Figure 4 shows the distribution of Rpre. with PDTB-3 distributions for reference. Compared to PDTB-3, whose distribution came from news text, we observe many more Expansion.Mannn relations associated with elaborations that ex- plain the manner in which a situation in the pre- elaboration sentence was done. As expected, Con- tingency. Cause frequently appears. Our manual examination indicates that authors often stated the result in the complex explicitly and left cause implicit; when simplifying, this implicit cause was deemed too confusing for younger readers and so was added as the elaboration. Expan- sion.Conjune Conjunction is often linked with definitions. In many cases, an EntRel (entity relationship only) NoRel (no relation) involve organizational sen- tences (c.f. Section 3.1 example (1)) that opens succeeding discourse. We noticed many more Hy- pophora relations compared PDTB-3; these are questions posed by the editors simplifying the doc- ument that guides children for what comes next.
B Model setup and hyperparameters
We tabulate all model setup and hyperparameters in Table 7.
C
Copyrights
This work depends on the Newsela text simpli- fication dataset (Xu et al., 2015). This dataset is free-to-use for academic researchers at https: /nwwwela. cwm/ data. The authors have obtained permission from Newsela to use this data.
D Compute
sion.
For all models in this work, we used compute nodes each consisting of 3x NVIDIA A100 GPUs. All experiments finished in under hours.
We also report the most frequent discourse re- lations associated with each of the top 5 most fre- quent question type:
Concept Q: EntRel, Expansion.Conpee Conjunction
Example Expansion.Coonjuntioo. Conjunction, Contingeney. Consequence Q: EntRel, Expansion.Conjuuee Conjunction Cause Q: Contingency. Cause, EntRel
A Hlateoo--the-nt classifier (Kim et al., 2020) did poorly on correctly classifying the relations with 42.5% accuracy on the 40 relations; thus, we do not include analyses from automatically recognized relations.

---

Model
Input Format
Hyperparameters
DCQA-base
pannectt-dcqu.__unccoo [claboration], [question] [context-doqa), [anchor], [elaboration], [question] Icontext-ing), [anchor w/ gold target). [question] [context-ing], [anchor w/ gold target], question] [context-ing]. (anchor w/ predicted target), [question]
learning rate=5e-5, epochs=5.batch size=8
DCQA-ft
learning_rate=20s epochs=5, batch_atchiae size=2
INQ GoldT-base
learning ratiiin-__te--55_,_.0 epocls==,.aacch_sizes
INQ-Gne NoO-GoldT-ft
lcarning_rate=2es epochs=5, batch_size=2 learning_ parninngrate=2e-5,, epochs=5, batch_size=2
INQ-PredT
Target prediction
[context]. [anchor sentence). [gold span]
learning rate=5e-5, epochs=3, batch size=16
Table 7:
Model settings. INQUISITIVE models (including target prediction model) are reproduced from the same setup Ko al. (2020) before fine-tuning ELABQUD. Models fine-tuned ELABQUD is done with the same input format, where the hyperparameters denote training setup of the fine-tuning stage only. For DCQA models, context-doga denotes all sentences prior to the elaboration (where the anchor sentence is enclosed with delimiter). For INQUISITIVE models, context-ing denotes all sentences prior the anchor; the anchor includes the gold or predicted target denoted enclosed with a delimiter. For the target span prediction model, the SQuAD QA setup is followed in Ko et al. (2020)'s span prediction model: SQuAD question - context, SQuAD context anchor sentence, SQuAD answer + gold span. Questions are decoded with the HuggingFace default greedy decoding. All hyperparameters tuned validation set.


Reviews:
[{"review_id": "kIOgj0C9y6", "reviewer": "Reviewer_iTg6", "summary": "", "questions": "- Will the dataset be available freely or will it also require granted access to Newsela?\n- Did you have a look at the readability of the elaborations that were generated? For example in Table 5, the last example shows a cleft sentence, which is something we would want to avoid in a simple text. Is it common? Were regularities observed regarding the output of the models?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback! \n\n_Re: factuality of elaboration content:_ \nOur focus is to show that our QUD-based methods improve generated elaborations (Table 6), and we leave much of the (important!) nuance, such as hallucination from LLMs, to future work; we will make this explicit in our conclusions and limitations sections. \n\n_Re: readability of elaborations:_\nWe did look at readability of the generated elaborations: this falls under the “elaboration-like” criterion of the manual analysis in Table 6. We note some generalizations for elaboration generation in section 5.2 Manual Evaluation, but did not find any frequently occurring structures such as clefts. \n\n_Re: data sharing:_\nWe will publicly share all annotations contributed in ELABQUD; note that the Newsela dataset, while free for academic and research use, does require permission (Appendix C).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Simplification through content addition is neglected in the ATS literature, while it has been shown to play an important part in actual text simplification for humans.\n- The idea of exploring the QUD framework for elaborations in ATS as it is done here is a nice idea, which the authors soundly executed.\n- The paper is dense, as everything is thoroughly explained and detailed, and at the same time it remains easy to follow.", "reasons_to_reject": "I do not have strong reasons to reject, but there is one non trivial point I feel is lacking.\n\n- As it is rightly pointed out in the introduction, hallucinations are a well-known risk with elaborations. It is also a frequent behaviour of LLMs such as GPT3 that is used here. I was surprised to not see this issue discussed at all in the body of the paper. I think this is an important lack, especially as the authors report that 34% of the elaborations are explanations of concepts, which may require information that can not be deduced from the text itself. While I would have expected/liked so see experiments and analysis/discussion around this issue, I can understand it may be out of scope for this paper and too demanding for a camera-ready version. It should at least be mentioned as a future work and/or in the limitations section.", "questions_to_authors": "- Will the dataset be available freely or will it also require granted access to Newsela?\n- Did you have a look at the readability of the elaborations that were generated? For example in Table 5, the last example shows a cleft sentence, which is something we would want to avoid in a simple text. Is it common? Were regularities observed regarding the output of the models?", "paper_topic_and_main_contributions": "This paper presents experiments in automatic text simplification (ATS), and more specifically on an understudied aspect of it: the addition of information that improves clarity for the reader. This process is called elaboration. The authors explore how to generate elaboration with a framework called QUD (for question under discussion). In this framework, an elaboration is considered as an answer to an implicit question. They base their work on a previously released corpus of elaborations. The authors report experiments for which they produced an annotated corpus, named ElabQUD. The corpus is one contribution of the paper. Another contribution is an analysis on the types of elaborations that are found in Newsela. The last contribution consists in experiments on question generation and elaboration generation, with several different protocols. The results show that the proposed approach is promising for generating elaborations for ATS.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The paper shows a good knowledge of the domain, there is not important missing reference.", "Typos_Grammar_Style_and_Presentation_Improvements": "l. 195 simplifieddocument, missing space\n"}, "extra_text": {}}, {"review_id": "eZ8eyITwZG", "reviewer": "Reviewer_qEyA", "summary": "", "questions": "l.298-306: You report percentage agreement of 69.9%. Could you please elaborate a bit about the type of disagreement that you have noticed?\n\nl.310: If I got it correctly, your dataset includes 1,299 elaborations, thus 1,299 targets. As they are composed of an average of 4.54 tokens, I do not understand how you reach the number of 13,153 tokens.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback!. \n\n_Re: “topic and contributions could be seen as too focused”_:\nWe believe elaborative simplification to be very helpful in automatic text simplification, especially when simplifying for those who have limited knowledge of the domain and so are most in need of new information. Thus, we believe exploring elaborative simplification is useful and warrants detailed theoretical treatment (e.g., by using the QUD approach we take). \n\n_Regarding your questions_: often there are multiple plausible targets for an elaboration; this is the main source of disagreement in the location of the target (section 3.2). Hence, we kept all targets provided by the annotators, so there are actually 2878 targets (= number of questions in the dataset (end of section 3.1); this is why there are 13153 tokens. Finally, thank you also for catching these grammatical and spelling errors! We will be sure to fix them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Original topic, that although it is quite focus, could have wider repercussions over time.\n- Convincing methodology, at least as regards the resource building (for which inter-rator agreements are reported) and the evaluations (that combines automatic and manual evaluations).\n- There are two contributions of the paper: a resource and one scientific clarification about the impact of QUD when generating elaborations. ", "reasons_to_reject": "- The topic and contributions could be seen as too focused.", "questions_to_authors": "l.298-306: You report percentage agreement of 69.9%. Could you please elaborate a bit about the type of disagreement that you have noticed?\n\nl.310: If I got it correctly, your dataset includes 1,299 elaborations, thus 1,299 targets. As they are composed of an average of 4.54 tokens, I do not understand how you reach the number of 13,153 tokens.", "paper_topic_and_main_contributions": "This article tackles a particularly difficult challenge in automatic text simplification (ATS), namely dealing with the implicit. It approaches by focusing on elaboration, i.e. simplifying by adding content, whereas traditional methods tend to act by deleting or reducing textual elements. Elaborative simplification makes sense in cases such as the need to explain a technical word (in specialized domains) or to explicit some implicit information in the text that a given reader might not be able to retrieve by itself. More precisely, the paper adopts the framework of Question Under Discussion (QUD), which postulates that each sentence in a text could be seen as the answer to an implicit or explicit question from prior context. The authors then introduce ELABQUD, an extension to the corpus collected by Srikanth and Li (2021). The initial corpus comprises 1299 elaborations, to which the authors have added 2878 possible questions leading to these elaborations. They also annotated the targets of each elaboration. This is the first contribution of the paper.\n\nBased on this corpus, the authors investigated two research questions: (1) is it possible to automatically generate the question behind an elaboration (QUD)? and (2) does knowing this QUD help to generate elaboration in texts to be simplified? They first show that QUD can be generated in a promising way, although there is still much room for improvement. Then, they demonstrate that having access to the QUD significantly helps LLMs to generate valuable and coherent elaborations. \n\nIn my opinion, this paper deals with a very specific issue, which does not have a direct impact on the performance of ATS models. Nevertheless, it is also one of the rare first steps towards taking the implicit into account in applications linked to reading difficulty. What's more, the article is well written and the experiments were carried out rigorously. In particular, the authors combined an automatic evaluation with a manual evaluation on each occasion, which enabled them to remain connected to the data. The pattern of results also seems very clear and consistent, which adds to the solidity of the article.    ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "l.093: corpus of multiply annotated → multiply is probably not the correct term here\nl.100: We find authors elaborate→We find that authors elaborate \nl.193: as section 3 describes the ELABQUD corpus, I did not find the title of the section very coherent with the content. It seems that experiments will follow, whereas it is a resource. \nl.195: For a simplifieddocument → For a simplified document \nl.513: givng → giving"}}, {"review_id": "Ye6BS8wAOA", "reviewer": "Reviewer_uFn6", "summary": "", "questions": "I am accustomed to the circularity issue of QUD in NLG-related settings, so I immediately wondered why you actually trained a model for which the elaboration was not masked in the input. I mean, especially for generation, this is  important. One of the problems arising from this is the applicability of the model in a real text simplification scenario. Deciding when to generate an elaboration bases on the previous context, when the decision is made, you can only produce the QUD from previous context. Its therefore self-explanatory that the model without masked elaboration will perform best, but the performance that counts is the inquisitive type, which is reported to be worse than just feeding the context into GPT-3. \nThe question is: why do you conclude that its the QUD that improves the generation quality? Does it have to be a QUD? wouldn't be a representation that is not encoded in natural language be euqally usable and directly integratable in the generation (seq2seq) pipeline, as long as it encodes the info given in the QUD?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! \n\n_Re: focus and topic structure in QUD theory and  “the design of the setup from a QUD-theoretical perspective, namely what the questions under discussion are allowed to ask for”:_\n\nThank you for bringing this up, we have not engaged with this in our paper but will add a discussion. We agree that it could be worthwhile to bring in some of the large literature on topic, focus and non-at-issue in the QUD literature. Our particular approach was fairly unconstrained in what QUD questions could ask, focusing on annotators providing natural intuition for their questions. We were clear on their task of providing questions which (1)  the elaboration answered and (2) were based on the prior context, as well as identifying a “target” (i.e. a sub-line selection of the prior text) that the elaboration is about (section 3.1 Annotation task, paragraphs 2 and 3, respectively). We found this satisfactory based on manual review of annotators (as described in section 3.1 Annotators), and ran a question-type classifier to understand what kinds of questions are being asked (section 3.2 What types of questions are  asked?, Table 1). While we acknowledge that not all theoretical perspectives would endorse this, we will clarify exactly what we see as the terms of QUD and what latent theoretical assumptions there are.\n\n What could potentially be worth looking into is the connection between the “target” of those QUDs highlighted by the annotators and the information structure of the anchor sentence, but we have not analyzed whether the targets are more about the foci or the topics. We leave this to future work.\n\n_Re: why train a model where the elaboration is not masked:_\n\nWe view this as some sort of silver “upper bound”. We certainly agree with the shortcomings of DCQA seeing the elaboration (section 4.1 Practical System); however, it still provides insight into the utility of QUDs in elaboration generation, as our manual analysis shows (Table 6). The DCQA model follows a “recovery” approach of QUD, and is complementary to the expectation-driven (Kehler and Rohde) or “potential questions” (Onea) approach as in Inquisitive questions. The question we were seeking to answer with DCQA models is: If there was a model that had access to what our human annotators had, what kind of performance are we looking at? So it is more of a feasibility test with the tools we have, and pointing out the gaps between this and Inquisitive-style questions, and giving us and the community a sense of the height of the hill to climb.\n\n_Re: “why do you conclude that it's the QUD that improves the generation quality? Does it have to be a QUD? wouldn't be a representation that is not encoded in natural language be equally usable and directly integratable in the generation (seq2seq) pipeline, as long as it encodes the info given in the QUD?”:_\n\nOur analysis shows that QUDs do help elaboration generation even when the elaboration is masked, as your manual evaluation of generated elaborations (Table 6) ranks INQ-GoldT generations as more elaboration-like than DCQA’s. Additionally, this analysis ranks all QUD-based approaches as more elaboration-like than the Context-Only approach. We agree that the information can be an encoded representation rather than an explicit QUD, however we believe that a question-answering format is more natural to today’s NLG systems when the amount of training data is not a lot. We believe there is much improvement to be made, in generating and evaluating both questions and elaborations, and we leave this to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper sheds light on a topic that has not yet been thoroughly explored in NLP / NLG; namely QUD.\nAlthough the paper is a bit euphemistic about the impact of QUD on generation improvement, I think the most\nimportant findings of this paper are the things that do not correctly work. The authors give another prove about \nthe circularity issue of QUD in NLG-related tasks: The source text is needed to produce a QUD that truly captures\nthe anticipated informational piece and therefore serves the purpose of generating a good continuation. As the paper\nshows, the quality is strongly reduced when the model that produces the QUD gets input where the source sentence\nfor the QUD is masked, which results in QUDs that do not fit human intuition about the informational requirement that\ntriggered the elaboration. That also happens for data-to-text generation, where QUDs actually lead off-topic, loss\nof coherence by the LLM itself added on top. The finding the authors report are in alignment with my personal \nexperience of applying QUD theory in NLG, and I think these results are important and should being published.  \nAs the authors themselves term it, \"These cases lead to worse automatic measures compared to context-only set-\ntings, indicating a challenging future direction for QUD generation.\"", "reasons_to_reject": "There are some conceptual issues with the design of the setup from a QUD-theoretical perspective, namely what the questions under discussion are allowed to ask for. I personally do not mind those, but the theory-focused QUD community will.\nThere is a large discussion about topic, focus and non-at-issue in the QUD community, which I personally also dislike,\nbut the authors should mention what they interprete to be focus and topic structure in case of elaboration (or whether they even\ntreat this information as non-at-issue). Often enough, annotation guidelines (e.g. Riester et al.) destroy themselves when reaching\natomic information entities due to restrictions what the QUD may ask for. In case of atomic database queries, the distinction between\nfocus and topic starts to blur, resulting in paradox situations where QUDs may not ask for the only informational entity in the sentence they shall ask for in an NLG setting. This is a cumbersome discussion and of least importance from the perspective of NLG engineers (like myself), but the purely theoretical QUD  proponents will want to know details.", "questions_to_authors": "I am accustomed to the circularity issue of QUD in NLG-related settings, so I immediately wondered why you actually trained a model for which the elaboration was not masked in the input. I mean, especially for generation, this is  important. One of the problems arising from this is the applicability of the model in a real text simplification scenario. Deciding when to generate an elaboration bases on the previous context, when the decision is made, you can only produce the QUD from previous context. Its therefore self-explanatory that the model without masked elaboration will perform best, but the performance that counts is the inquisitive type, which is reported to be worse than just feeding the context into GPT-3. \nThe question is: why do you conclude that its the QUD that improves the generation quality? Does it have to be a QUD? wouldn't be a representation that is not encoded in natural language be euqally usable and directly integratable in the generation (seq2seq) pipeline, as long as it encodes the info given in the QUD?", "paper_topic_and_main_contributions": "Sequence to sequence models of simplification leave out of consideration the cases of elaborative simplification, where further information is added to the simplified text. Therefore, the authors make use of QUD theory to model elaborations as explicit answers to implicit questions.\n\nThe authors have created a corpus ElabQUD of elaborative simplifications that they enriched with respective QUDs.\n\nThis corpus they use to fine-tune a question generation model that they trained on two different QUD copora. These produced questions are then prompted to a GPT-3 model in order to produce real elaborations., which they claim to be of substantially higher quality.\n\nResults of BLEU and BERTScore show that QUDs improve generation quality of elaborations after inputting the GPT-2 generated questions as prompts into GPT-3 for zero-shot learning. BUT this is true mainly for human-produced questions and the QUDs from models where the elaboration in the input was NOT masked!\nINQUSITIVE models for which the elaboration was masked during training, did not produce QUDs that were useful in regard to the information requirements to which the elaboration is the answer. This leads to worse scores compared even to the context-only case.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "To reason though discourse p.2 line 158\nsimplifieddocument p.3 line 195\nplease reformulate in line 523/524"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reasons to Accept:
(1) The paper addresses a neglected aspect of abstractive text summarization (ATS) by focusing on content addition, which is essential for actual text simplification.
(2) The exploration of the Question Under Discussion (QUD) framework for generating elaborations in ATS is an innovative and well-executed idea.
(3) The paper is well-structured and thoroughly explains its concepts, making it easy to follow.

Reasons to Reject:
(1) There are no strong reasons to reject the paper, but it could benefit from discussing the risk of hallucinations in elaborations, especially given that some elaborations involve explanations of concepts. While this might be a topic for future work or limitations, addressing it would add depth to the paper.

The authors have provided reasonable responses regarding hallucinations and discussions on focus and topic structure.
Decision: Accept-Main