QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing
Authors: Yating Wu
Ritika Rajesh Mangla
Greg Durrett
Junyi Jessy Li
Venue: EMNLP 2023 Main (2023)
Keywords: discourse; questions under discussion; QUD; evaluation

Abstract: Questions Under Discussion (QUD) is a versatile linguistic framework in which discourse progresses as continuously asking questions and answering them. Automatic parsing of a discourse to produce a QUD structure thus entails a complex question generation task: given a document and an answer sentence, generate a question that satisfies linguistic constraints of QUD and can be grounded in an anchor sentence in prior context. These questions are known to be curiosity-driven and open-ended. This work introduces the first framework for the automatic evaluation of QUD parsing, instantiating the theoretical constraints of QUD in a concrete protocol. We present QUDeval, a dataset of fine-grained evaluation of 2,190 QUD questions generated from both fine-tuned systems and LLMs. Using QUDeval, we show that satisfying all constraints of QUD is still challenging for modern LLMs, and that existing evaluation metrics poorly approximate parser quality. Encouragingly, human-authored QUDs are scored highly by our human evaluators, suggesting that there is headroom for further progress on language modeling to improve both QUD parsing and QUD evaluation.

QUDEVAL: The Evaluation of Questions Under Discussion Discourse Parsing
Yating Wu Ritika Mangla² Greg Durrett Junyi Jessy Li Electrical and Computer Engineering, CCompterr Science, Linguisttics
The University of Texas at Austin ritikamangla, gdurrett, jessy)Jutexas. edu
(yating.ww wu,
Abstract
QUD Parsers
Questions Under Discussion (QUD) is ver- satile linguistic framework in which discourse progresses continuously asking questions and answering them. Automatic parsing of discourse to produce a QUD structure en- tails complex question generation task: given a document and an answer sentence, generate a question that satisfies linguistic constraints of QUD and can be grounded in an anchor sentence in prior context. These questions are known be curiosity-driven and open-ended. This work introduces the first framework for the automatic evaluation of QUD parsing, in- stantiating the theoretical constraints of QUD in concrete protocol. We present QUDE- VAL., a dataset of fine-grained evaluation of 2,190 QUD questions generated from both fine- tnndd systems and LLMs. Using QUDEVAL, we show that satisfying all constraints of QUD is still challenging for modern LLMs, and that existing evaluation metrics poorly approximate parser quality. Encouragingly, human-authored QUDs are scored highly by our human eval- uators, suggesting that there headroom further progress language modeling im- prove both QUD parsing and QUD evaluation.
al. (2023) GPT (this work)
doee...
Alpaca (this work)
did...
Documents
Predicted QUDs
This work: how do we evaluate QUDs with respect to theoretical criteria from the literature?
Human scoring predicted QUDs: Language quality, Answer compatibliity, Givenness, Relevance
Evaluating automatic metrics: Do commanly metrice agree with human annotation?
Reference-free Metrics
Reference-based
Givenness: Rule-based,
Metrics (vs. gold QUDs)
Hou (2021),
Relevance: Rule-based, GPT-Ols,
BERTS
What
QSTS
An overview of this work. Given QUD parser outputs, we (a) design protocol for evaluation: (b) col- lect human judgments using this protocol; (c) evaluate automatic evaluation metries. This work additionally contributes new LLM-based QUD parsers.
Figure I:
tractable for settings that require higher-level pro- cesses, c.g., planning in text generation (Narayan et al., 2023), contextualization in question answer- ing (Newman et al., 2023), and elaborative simpli- fication (Wu et al., 2023). However, rigorous eval- uation of the question generation aspect of QUD frameworks remains an open challenge. QUD parsing relatively new the NLP com- munity, with most of the prior work in discourse parsing focusing on structures like Rhetorical Struc- ture Theory (Mann and Thompson, 1988), Seg- mented Discourse Representation Theory (Asher et al., 2003), and the Penn Discourse Tree- bank (Prasad et al., 2008), all of which use fixed, hierarchical discourse relation taxonomy and are relatively straightforward evaluate using accu- racies or F measures. QUD parsing (illustrated in Figure 2), however, entails a question generation task that is contextually grounded (Ko et al., 2023).
1 Introduction
Text comprehension at the discourse level entails understanding higher level structures between sen- tences and paragraphs, beyond the meaning of in- dividual words. The linguistic framework of Ques- tions Under Discussion (QUD) (Van Kuppevelt, 1995; Roberts, 2012; Benz and Jasinskaja, 2017) views the progression of discourse as continuously posing (implicit) questions (i.e., QUDs) and an- swering them; thus each sentence in a monologue is answer, or part of answer,
QUD. In Fig- ure 2, the third sentence answers the implicit QUD, "What does Glenn say about the situation?" elicited from sentence 2. The advent of large language models (LLMs) makes viewing discourse in ques- tion generation and answering fashion increasingly

---

IU.S. exports of nuclear material cannot be edequstely traced from country country, according to a congressional report. *Scarcely a day goes by without a report of new black market deal,' said Sen. John Giann ... reacting to the report.
QUD (1,2) What do lawmakers think about this issue?
Answer (82): Scarcaly day goes without a report af new blcc market deal' said Glenn reaciig the report
Languege quality: QUD is gra grammetical and makes sense
y Answer compatibilty; the answer sentence answers the
Glvven the staggering amount of nucleer materials we have exported, could only be doadly
Givenness: QUD does not contain concepts beyand anchor and context
matter time before some contraband proves to be of U.S. origin.' *Aa chairman of Glenn commessioeed the report from General Accounting Office, which conducts investigations for legislators. "The report plutonium and highly enriched uranium have accumulated worldide...
y Relevance: reader could ask this after seeing anchor (QUD grounded in S1) I
QUD (2,3) What does Glenn say about situation? Lang. Answer compatible Givernees
grounded $2
hundreds of tons of
QUD (1,5)
What does the report say Is the reason for the export ban?
y Lang.
Arswer incompatible:
Hallucination: ban not S1
QUD partially grounded S1
reason for ban missing
4tt does not incluce figures U.S. nuclear exports says export licenses nuclear materials were granted in 1993.
QuD (1,6)
What is missing in the report?
y Lang.
Answer
Answer leokage:
QUD pertially
competibbe
"missing" is not given
grounded in $1
A QUD dependency parse and our evaluation labels; showing 4 of the 5 QUDs due to space constraints. Given the article snippet on the left, a QUD dependency parser generates the tree with edges as QUDs; QUD(2, means that sentence answers the corresponding question anchored in sentence 2. Besides generating the question, the position of the anchor sentence (head node) is also predicted. QUDEVAL includes 4 criteria for each edge (right) evaluating both the generated QUD and the predicted anchor, discussed in Section 3.
Figure 2:
This structure necessitates two key principles for evaluation. First, the questions are raised after a certain amount of common ground is established, parsing QUD involves identifying where in the context a question can be triggered (also called the anchor sentence). These anchors are produced by parsers and form part of the structure evalu- ate. Second, for a question to become a QUD, needs to satisfy several linguistic constraints (Ri- ester et al., 2018) and be plausible without knowing a priori how the discourse will progress.
by trained linguists across 51 news articles. These annotations evaluate three QUD dependency pars- ing models: Ko et al. (2023)'s pipeline system, as well as our newly designed prompts for QUD pars- ing with LLMs using OpenAI ChatGPT (GPT 3.5 Turbo), GPT-4, and Stanford Alpaca (Taori et al., 2023). Human annotations using the QUDEVAL framework reveal that modern LLMs,, while able to generate fluent questions, often fail at satisfying these constraints; the errors vary across systems, rendering such tnn-ggrined evaluation necessary. QUDEVAL also provides a valuable testbed for the assessment of automatic evaluation metrics for QUD parsers. We engage with rule-based and LLM- baselines, prior work relevant for spe- cific criteria, and reference-based metries standard in question generation evaluation. Results show that while these metrics align with QUDEVAL annotations to some extent, they give poor esti- mates of QUD generation quality (except for an- chor groundedness) in comparison to an estimated human upper bound. This points to both challenges and substantial headroom for the development of automatic metrics.
The above characteristics make QUD parsing much tricker to evaluate than other discourse parses; thus, prior work in QUD dependency pars- ing (Ko et al., 2023) performed only manual evalu- ation. Automatic evaluation for generation tasks known to be challenging (Celikyilmaz et al., 2020) and standard evaluation metrics for question gen- eration (Amidei et al.. 2018; Nema and Khapra, 2018; Gollapalli and Ng, 2022) tend to be superfi- cial linguistically. They do not evaluate the afore- mentioned desiderata of QUD.
Outlined in Figure 1, this work provides founda- tional building blocks for the automatic evaluation of QUD dependency parsers. Building on prior the- oretical work on QUD and its annotation (Riester et al., 2018; De Kuthy et al., 2018), as well as er- ror analysis on a large number of machine outputs from several different models, we design four key criteria assess QUD question generation quality, covering language quality, answer compatibility, question givenness, and anchor relevance.
To sum, QUDEVAL points to clear directions for future progress: (1) human evaluation reveals con- sistently higher quality for crowdsourced QUDs compared to generated ones, suggesting opportu- nities for better QUD parsers; (2) the development of mnnguustically-infomeen automatic metries target- ing QUD parsing quality is necessary, for which QUDEVAL can serve as a benchmark.
With these criteria, we present QUDEVAL, 2,190 generated question-ands anchoo pairs annotated
QUDEVAL is available at https://gg thub. cmmp/Lingchennsaeenn/Queaall

---

2 Background and Related Work
guage Generation (NLG) tasks in general (Celikyil- maz et al., 2020), automatic evaluation of question generation is known to be challenging and inconsis- tent (Amidei et al., 2018). We evaluate commonly- used metrics, as well as a recent metric from Gol- lapalli and Ng (2022), using our linguistically- grounded frameworks.
QUD Annotation and Parsing Despite its rich linguistic history (see Benz and Jasinskaja (2017) for overview), QUD's presence in NLP is in its infancy annotated datasets have only recently started emerging. These datasets follow different QUD paradigms, reflecting its diverse interpreta- tion: Westera et al. (2020) and Ko et al. (2020) pre- sented QUDs in an expectation-die driven (Kehler and Rohde, 2017) manner, where questions are elicited while reading (i.e., without seeing the rest of the article). Unanswerable questions are thus a natural artifact of this process. De Kuthy et al. (2018) an- notated QUD trees with full, hierarchical questions (i.e., the answer to a parent question entails the answer to a child question (Roberts, 2012)); how- ever, due to the challenging annotation process, the dataset only contains two sections of an interview transcript, too small to train automatic parsers.
QUD Evaluation Framework
QUDs are subjective: different readers often come up with distinct questions even with the same an- swer sentence (Ko et al., 2022). Thus our goal is to target common principles that QUDs need to sat- isfy. First, we instantiate theoretical constraints of QUD (Riester al., 2018; De Kuthy et al., 2018) in a concrete protocol. Second, consider com- mon errors in text generation systems, including question generation systems, and ensure they are captured in our taxonomics.
The parsing framework that this paper engages with is from Ko al. (2023), trained with the DCQA dataset from Ko et al. (2022). Ko al. (2023) views the QUD structure as a dependency tree, where each sentence in a document is con- nected to an anchor sentence in its prior context via a QUD (formalized in Section 3). This is a lightweight approach to QUD as the questions themselves are not guaranteed to be hierarchi- cal; however this simplification allows large-scale, crowdsourced data collection (Ko al., 2022). To date, this remains the only QUD parser available.
Setup and Definitions Each evaluation instance corresponds to an individual edge in a dependency tree where QUD connects two sentences: the head of an edge is an anchor sentence Sy. (the Ath sen- tence in document) where the question is elicited, and the child node is an answer sentence S. that answers the question (Figure 2). Thus we denote an edge as (k. where Q the question string. QUD dependency parsing entails considering each sentence in a document as Sa and, for that answer sentence, generating Q and predicting k the anchor index.
Evaluation of Question Generation Prior work has tackled contextual question generation, includ- ing conversational (Nakanishi et al., 2019; Gu et al., 2021; Do et al., 2022; Kim et al., 2022), open-ended (Ko et al., 2020; Cao and Wang, 2021;
We present four criteria that assess both tasks separately and jointly, with the full annotator- facing instructions listed in Appendix Figure 4. Note that it is both theoretically and operationally possible that more than one sentence satisfies the criteria of being an anchor. In practice, we see that this depends on varying levels of specificity of the question and the document context. Because of this, our evaluation framework independently evaluates each QUD and its predicted anchor.
Chakrabarty et al., 2022), and nested questions (Kr- ishna and lyyer, 2019). While some of these works perform human evaluation, they typically cover di- mensions such as fluency, relevance/plausibiliyy scope, and question-anse -answer consistency. In clari- fication question generation (Rao and Daumé III, 2019), dimensions such as usefulness and informa- tiveness are also considered. However, the above criteria are insufficient for linguistically meaning- ful evaluation of QUD generation. Ko et al. (2023) used human evaluation taxonomy for their QUD parser; this work reflects extended version of that earlier laxonomy, better aligned with thcoreti- cal principles.
Language Quality (Lang.) This criterion is de- signed to filter out obviously bad questions. These include: badly formed questions that are not commodatable, questions that are irrelevant to the article, and questions with content that obviously contradicts the article. Note that we direct annota- tors to skip the rest of the evaluation if Q fails to satisfy this criterion. Examples of such questions
Similar to automatic evaluation of Natural Lan-

---

are shown in Appendix Table 17.'
(1) No new concepts: all concepts in Q are discourse-old or mediated (examples Figure 2). (2) Answer leakage: Q contains new concepts not in question context Co - Si,..., Sg but in Sa- In Figure 2, QUD (1,6) states "What is missing in the report?"; yet the notion of "missing" is absent in S (S1) and only introduced in S.. (S6), i.e., based on the common ground established in St. it is quite stretch for reader to ask this question. Figure 3 shows another example.
Answer Compatibility (Comp.) This criterion states that Sa should answer Q. This is one of the key criteria used both in Riester et al. (2018)'s guidelines (called "congruence"), as well as in hu- man evaluations for QG systems, e.g., called "an- swer validity" in Krishna and lyyer (2019) and "consistency" in Gu et al. (2021). We addition- ally consider the important role QUD plays in in- formation structure pragmatics (Buring, 2008; Beaver and Clark, 2009), i.e., QUDs are used to tease apart the main part (called focus or at-issue content) from background information in a sen- tence (Riester et al., 2018). 8). Thus it is the focus of So that should answer Q (Ko et al., 2022). We in- troduce graded notion of Answer Compatibility:
(3) Hallucination: Q contains new concepts that are not answer-lcakage. An example shown in Appendix Table 19.
Note that since S S position is predicted. this criterion jointly evaluates Q and k.
Anchor Relevance (Relv.) Our last criterion evaluates the anchor prediction è given Q This di- verges significantly from Riester et al. (2018) since anchor prediction is not a task in human annota- tion. Yet. we align with their work stating that Q should be relevant (i.e., contains given material) to the context where it was elicited. Referencing observations from our experiments and Ko et al. (2023), we design three categories:
(1) Direct and explicit: full compatibility de- scribed above (examples in Figure 2).
(2) Unfocused: S, contains the answer, but the answer is not its focus. An example is shown in Appendix Table 18.
(3) Not answered: Sa does not answer Q. In Figure 2, sentence should be the answer for the generated QUD (1,5); however, sentence 5 does not state the reason for the export ban. Another example is shown in Figure 3.
(1) Fully grounded: k satisfies the desiderata above, which typically means that content in Q follows mostlyfrom S (examples in Figure 2). (2) Partially grounded: some content from Q is grounded in S For QUDs (1,5) and (1,6) in Figure 2, Sj. (S1) only contain some of the concepts accessible from Q.
Givenness (Givn.) Since QUDs should be reader questions invoked at Sg for upcoming, unseen dis- course given the common ground already estab- lished, Q should only contain concepts that are accessible by the reader from prior context; this is called "Q-Givenness" in Riester et al. (2018). While they loosely define givenness as "given (or, at least, highly salient) material", concretely specify this notion with information status (Mark- ert et al., 2012): Q should only contain concepts (entities, events, or states) that were mentioned in the question context Cj = S1, ... Sy (discourse- old), or concepts that have not been directly men- tioned but are generally known or inferrable from mentioned ones (mediated).
(3) Not grounded: content from Q is largely not question in Figure 3, the concept of "restrictions" is the main focus of the question, which is irrelevant to Sgg (S5) that provides more information on the contents of the report.
from Si. For the
The QUDEVAL Dataset
This section describes our finguss--anottate QUDEVAL dataset evaluating fine-tuned (Ko al., 2023) and LLM parsers, and their results.
Based observations of machine errors from Ko al. (2023), generated questions can often contain concepts that are from S, itself, which is a particular error that prevents QG model being used more widely in conversations (Nakanishi et al., 2019). We call this answer leakage. This criterion is divided into the following categories:
4.1 Parsers Evaluated
Ko et al. (2023) This parser is trained as a pipeline, with a Longformer model (Beltagy et al.. 2020) for anchor prediction and GPT-2 (Radford et al.. 2019) for question generation; both are fine- tuned on the DCQA dataset (Ko et al., 2022).
For wrong predictions of = a, the annotators also skip the QUD; this happens in about 3% of the L.M parsers and does not happen with Ko et al. (2023)'s parser.
LLMs We also experiment with ChatGPT (GPT 3.5 Turbo), as well as Stanford Alpaca-7B (Taori

---

Lang.
Comp.
Givn.
Relv.
Yes
No
Dir-Ans.
Unfocus.
Not-Ans.
No-New.
Ans-leak.
Hallu.
Full.
Part.
No-G.
Ko et al.
92.5
7.5
52.5
11.0
36.5
75.6
12.1
12.3
70.1
19.3
10.6
ChatGPT
96.1
3,9
82.4
8.8
8.8
64.9
31.2
3,9
56.7
31.6
11.6
Alpaca
93.9
6.1
43.2
16.9
39.9
61.2
30.7
8.1
46.3
25.5
28.2
GPT4
100.0
0.0
90.6
3,9
5.5
61.8
34.3
3,9
54.1
35.5
10.4
DCQA
98.0
2.0
71.4
16.4
12.2
85.7
10.9
3,4
80.3
17.7
2.0
Table 1: Distribution (in %) of human-amme annottted labels for 2,040 system-generatee QUDs and 150 crowdsourced QUDs in DCQA. For Comp., Givn. and Relv., these percentages are calculated over the total # of questions that passed the Language Quality criteria. The error distributions (other than Lang.) across models are significantly different from each other (Chi-Square, p < 0.05) except for ChatGPT vs. GPT-4 for Givn. and Relv.
We first prompt the model generate Q given the article and Sg Subsequently, we used Q. the article, and Sa to generate the Si. We experimented with var- ious zero-shot and few-shot variations; our best prompt containing four in-context examples is pro- vided in Table 8 in the Appendix. In particular, we explicitly prompted the model not to introduce phrases ideas in the question which are newly introduced in Sg-
al., 2023), as LLM-based parsers. 2
Lang.
Comp.
Givn.
Relv,
Krippendorff's
0.56
0.52
0.48
0.51
3/3 Agr %
0.91
0.58
0.64
0.57
Pairwise F1
0.78
0.61
0.60
0.60
Table 2: Krippendorff's a (nominal for Lang. & Gävn., ordinal for Comp. & Relv.); % of total agreement: pair- wise Fl scores macro-averaged across labels.
lines on 20 questions (2 news articles x 2 systems) our team has iterated on. We release our annota- tion interface, presented in Appendix A.
Finally, we collected QUDs with GPT-4 after its APT became available. This portion of the data (510 QUDs) was triply annotated as an addition to the dataset. However, due to the lack of API access at the time of experimentation, this portion of the data was not part of metric evaluation in Section 5.
Next, all three annotators annotated 200 distinct QUDs (10 articles, 10 answer sentences each, and QUDs for each answer sentence). This set is used to calculate inter-annotator agreement, reported in Table 2. Krippendorff's o (Krippendorff, 2011) values indicate moderate agreement (Artstein and Poesio, 2008) across all criteria. Table also shows the % of questions where all 3 annotators agreed on a label, and an aggregated pairwise F1 score that reflects how accurately one annotator captures another's ' labels. The F1 can be viewed as a human upper bound for automatic metrics (Section 5).
4.2 Human Data Collection
Data Sourcing We run the above parsers on all news articles from the validation and test sets from Ko et al. (2023), which came from the DCQA dataset (Ko et al., 2022). For each document, we sample 10 sentences as the set of answer sen- tences; for each answer sentence, 4 distinct (Q, Sg) pairs were generated, one from each system. This amounts to 2,040 machine -eeeerree QUDs in to- tal. Additionally, we annotated 150 crowdsourced questions in DCQA across 15 articles both as an assessment of prior annotated QUD questions and a validation of our taxonomy. Thus the total number of distinct QUDs annotated is 2,190.
The annotation team discussed a representative portion of the data on which they did not fully agree. In almost all cases, disagreements are gen- uine differences in interpretation of the article itself or borderline cases between labels; some of these examples are shown in Table 20 in the Appendix. In addition, there were 25 QUDs that involved at least one criterion without majority label; these questions were adjudicated after agreement was calculated. The adjudicated labels are included QUDEVAL.
Annotation Procedure and Agreement Our an- notation team consists of 3 students in linguistics department; these students had extensive prior ex- perience in data annotation before this task. The students were trained with our annotation guide-
Given (a) the high cognitive load of this task, (b) how well-trained the annotators are, and (c) the fact that we found no case of erroneous task interpretation for the triply annotated set above, the rest of the questions are annotated by one annotator.
'A temperature of is used throughout the paper for LLMs. We noticed that increasing the temperature for our setting resulted in the generated questions and anchors containing concepts which are outside of the context provided.

---

System
Duplicate
Avg. Len.
Context
"The report says hundreds of tons of plutonium highly enriched
have accumulated worldwide... fil does not include figures on U.S. nuclear exports but 71 export licenses for materials were granted in 1983.
Ko al.
(1%)
13.11
Alpaca
104 (20%)
11,45
ChatGPT
19 (4%)
16.88
GPT4
16(3%)
15.09
Anewer
"Nuclear exports weapons or weapons research are prohibited, as is transfer of nuclear materials to a third country.
Table 3: Count and % of identical questions, well as average number of tokens in generated questions, stratified by system. Each system generated 510 QUDs.
DCQA What restrictions exist on nuclear material exports? Anchor: S6
(human)
What alse included in the report?
Anchor: S6
Answer incompatible
4.3 QUD Parser Evaluation Results
ChatGPT What are the restrictions on nudler exports and transfer nuclear materials third country?
Table 1 shows the distribution of annotated labels for each system evaluated, and for the set of 150 DCQA questions. All models score greater than 92% in terms of Language Quality, a clear indi- cation of their question generation capability in general. However, for all other criteria, there is large percentage of errors. For Answer Compatibil- ity, GPT-4 clearly leads the other models. Ko et al. (2023)'s ' system generates questions that are more grounded in the context, and predicts anchors that are the most relevant; however, their system scored the worst in hallucinations. None of the LLMs out- performs Ko et al. (2023)'s fine-tuned Longformer model for anchor prediction.
Anchor: S5
Xhe seswer for the eno
Answer laakage
QUD is not grounded in S5
Alpaca
What are the restrittions on nuclear exports?
Anchor: S1 QUD is partially grounded in S1
Figure 3: Model-generated questions for answer sen- tence 7 in the same article as Figure 2.
parsers. As a first step, we evaluate a set of base- line reference-free and reference-basee metrics." Note that we do not evaluate the Language Quality criterion, since all modern LLMs capable of generating fluent questions (Table 1).
5.1 Reference-Free Metries
Notably, for all criteria other than Answer Com- patibility, the models underperform crowdsourced questions in DCQA by a large margin. For An- swer Compatibility, ChatGPT and GPT-4 scored higher in terms of direct answers, but we observe that this is directly linked to its tendency to leak answers in the first place. These results show that QUD parsing remains challenging task and that QUDEVAL's evaluation framework is able capture errors from distinct types of models. Qualitative analysis reveals that outputs from these models are different styles (Figure 3). Ko et al. (2023)'s model tends to generate more gen- eral questions, likely resulting in fewer errors in Givenness and Anchor Relevance. On the other hand, GPT models generate more verbose ques- tions that are too grounded in the answer sentence itself, resulting in profuse answer leakuge. Table shows the average lengths of questions for each system. Alpaca makes a distinct set of errors: generates identical questions across different an- chor and answer sentences up to 20% of the time, much more frequently than other models (Table 3).
Baselines tested here include rule-based metrics, prior work on information status (Hou, 2021), and classification/scrinng with GPT models. The lat- ter has shown promise for evaluation in machine translation (Koemi and Federmann, 2023) and sum- marization (Luo et al., 2023; Wang et al., 2023). We held out the two articles used for annotator training as the validation set (60 questions).
Metrie Preliminaries All metrics are classifiers taking the form f: (S., Sg. Q (g) + y, map- ping from a QUD (answer sentence, anchor sen- tence, predicted question, and context) to one of the labels (in the set )) for one of our criteria. Note that not all criteria necessarily use every piece of in- formation; anchor relevance does not use anything about the answer sentence, for example.
We define two types of GPT4-based metrics throughout this section. First, GPT-CIs uses Chat- GPT as either zero-shot (-zs) or few-shot (-fs) classifier with an appropriate prompt. This directly maps into the label set y. Second, GPT-Ser fol- lows Koemi and Federmann (2023) and Wang al. (2023) use GPT-4 to assign real-valued score between and 100 for the quantity of interest. We
5 Automatic Evaluation Metrics
"s stated in Section 4.1, human evaluation results of GPT- 4 generated QUDs were not included to test automatic metries due to the lack of API access at the time of experimentation.
QUDEVAL can serve as the first benchmark for automatic evaluation metrics for QUD dependency

---

Direct
Unfo- cused
Not
Macro F1
Direct Ans.
Unfo- cused
Not Ans.
Macro
Comp-
Comp.
Ans.
GPT-Ser
0.70
0.05
0.36
0.37
BLEU-1
0.51
0.14
0.36
0.32
GPT-Ans
0.84
0.37
0.60
BERTScore
0.51
0.14
0.43
0.36
Random
0.62
0.11
0.31
0.35
METEOR
0.63
0.07
0.36
0.36
ROUGE
0.61
0.03
0.37
0.34
Givn.
No new
Ans. leak.
Hallu- cinat.
Macro
QSTS
0.52
0.12
0.38
0.35
enept.
Fl
GPT-Ser
0.64
0.08
0.31
0.35
Rule-based
0.52
0.40
0.19
0.37
No new
Ans. leak.
Hallu- cinat.
Macro
Hou (2021)
0.77
0.09
0.10
0.32
Givn.
enept.
F1
GPT-Cls-zs
0.75
0.02
0.22
0.33
GPT-Cls-fs
0.61
0.02
0.19
0.27
BLEU-1
0.61
0.21
0.12
0.32
Random
0.66
0.20
0.08
0.32
BERTScore
0.4
0.29
0.13
0.28
METEOR
0.45
0.33
0.08
0.29
Fully grnd.
Some grnd.
Not
Macro
Rely.
ROUGE
0.73
0.13
0.10
0.32
grnd.
Fl
QSTS
0.62
0.30
0.07
0.33
Rule-based
0.31
0.35
0.37
0.34
GPT-Ser
0.65
0.35
0.1
0.37
GPT-Cls-zs
0.65
0.34
0.49
0.49
Fully grnd.
Some grnd.
Not
Macro
GPT-CIs-fs
0.60
0.26
0.46
0.44
Relv.
grnd.
GPT-Scr 0.73
0.41
0.57
0.57
BLEU1-sim 0.59
0.26
0.22
0.35
BLEU-1
0.55
0.19
0.21
0.32
Random 0.58
0.28
0.15
0.34
BERTScore
0.36
0.27
0.20
0.28
METEOR
0.42
0.27
0.24
0.32
Table 4: Reference-free metric assessment results (in FI). For Answer Compatibility, GPT-Ans does not dif- ferentiate between direct answer and unfocused answer.
ROUGE
0.63
0.19
0.18
0.34
QSTS
0.57
0.21
0.22
0.34
GPT-Ser
0.63
0..66
0.22
0.37
Table 5: Reference-based metric assessment results.
then use a mapping function to convert values in this interval to labels in y, tuned for macro-F1 on the validation set. Such a mapping from this inter- val is possible because many of our eπrоr categories have an ordinal semantics associated with them.
We check for content word overlap with the St. Q labeled as 'fully grounded' if all words in its max NP are present in Sj. 'not grounded' if none are present, and 'partially grounded' other- wise. (2) GPT-CIs: the prompts are in Appendix Ta- bles 14 (zero-shot) and 15 (few-shot). (3) GPT-Ser: mapping function and prompt are in Appendix B. (4) BLEUI-sim scores answer by computing BLEU-1 (Papineni et al., 2002) between Q and Sj mapping function detailed Appendix B.
Answer Compatibility (1) GPT-Ser assesses how well Sa answers Q. Mapping function and prompt detailed in Appendix B. (2) GPT-Ans: Wad- hwa et al. (2023) showed the efficacy of GPT-4 on QA setting similar to QUD. We prompt GPT-4 using their prompt to generate an answer, given the article, Q, Sg. We then prompt it again to find the closest sentence the article the generated answer (prompt in Appendix Table 10).
5.2 Reference-Based Metries
We further evaluate reference-based metrics stan- dard in question generation that capture the similar- ity between Q and Q: (1) BLEU-1 (Papineni et al., 2002),* 4 (2) BERTScore (rescaled) (Zhang et al., 2020); (3) METEOR (Lavie and Agarwal, 2007); (4) ROUGE-FI (Lin, 2004); (5) QSTS (Gollapalli and Ng. 2022), recent reference-basee metric for question generation evaluation that explicitly rep- resents the question class and named entities in given question pair, and combines them with de- pendency tree information and word embeddings to measure the semantic similarity between two questions; (6) GPT-Ser (prompt in Appendix Ta-
Givenness (1) Rule-based: We scan lemmatized content words Q for new words not present in Cgi they appear only in So, then we label Q as 'answer leakage', otherwise as 'hallucination'. (2) Hou (2021): we run the Matc-of-the-ar information status classification model from Hou (2021), using the top-level labels (new, old, mediated). Since both new and mediated concepts are allowed, we merge these classes. We similar rule I)) differentiate between answer leakage and hallucination, detailed in Appendix B. (3) GPT-CIs: the prompts can be found in Appendix Tables 12 (zero-shot) and 13 (few-shot).
*While standard to report BLEU 2, 3, 4, Nema Khapra (2018) found BLEU-1 correlates human annotations better than other settings.
Anchor Relevance (1) Rule-based: This method checks the focus (approximated by maximum NP) of Q overlaps with the predicted anchor S
'The original QSTS uses the harmonic mean to combine several sub-scores, which yields a score of zero when there

---

Comp. Unfocus.
Givn. Ans-leak.
Relv.
Dir-Ans.
Not-Ans.
No-New.
Hallu.
Full.
Some.
No-G.
Reference Free
Ko et al. ChatGPT
0.88
0.11
0.82
0.09
0.09
0.75
0.33
0.46
0.75
0.24
0.86
0.10
0.04
0.75
0,50
0.51
Alpaca DCQA
0.96
0.03
0.72
0.10
0.18
0.68
0.36
0.65
0.82
0.17
0.76
0.15
0.09
0.71
0.39
0.14
Reference Based
Ko et al. 0.04
0.92
0.29
0.49
0.29
0.24
0.46
ChatGPT 0.50
0.36
1.0
Alpaca 0.54
0.32
0.42
0.45
0.12
0.43
0.26
0.30
Table 6: Predicted scoring of various parsers and DCQA questions using the best metrics. Reference-free: GPT4-Ans (Comp.), GPT-CIs-zs (Givn.), GPT4-Scr (Relv.). Reference-bused: BERTScore (Comp.), GPT-4 (Givn.), GPT-4 (Relv.).
Metric
Corr.
Metric
Corr.
in Table 6. In general, the metrics do not ade- quately capture system-level ordering determined by human judges in Table 1. GPT-Scr is used for the Givenness and Anchor Relevance criterion; in both, it scores ChatGPT-gennrated QUDs much higher than it should, and even higher than human QUDs. This confirms the bias that GPT can favor itself when being used as an evaluation tool (Liu et al., 2023), although we did not include GPT-4 generated QUDs in the evaluation data per Do reference-basee metries capture similarity to annotated QUDs? One hypothesis is that stan- dard reference-based metrics are not actually cap- turing similarity with ground-truth QUDs. To check this, our annotators rated subset of 200 pairs of (Q, Q) for similarity (detailed in Ap- pendix C). This annotation allows us to examine label distributions stratified by how similar gener- ated QUDs are to the reference QUDs (Appendix D, Figure 9). While questions similar to the refer- ence QUDs tend to score better, our metrics do not clearly capture question similarity well (Table 7). Furthermore, even knowing similarity may not be enough evaluate all cases of QUD, particularly when there are many possible QUDs that models can generate.
BLEU-1
0.12
METEOR
0.21
BERTScore
0.32
QSTS
0.23
ROUGE 0.19
GPT4-Ser
0.57
Table 7: Spearman rank correlation coefficients of auto- matic metrics with annotated QUD similarity. The corre- lation values are significantly higher than (p 0.05).
ble 11). The reference-based metric values are mapped labels using the same mapping function mechanism described in Section 5.1.
5.3 Results and Analysis
Results are shown in Tables 4 (reference-free and (reference-based), respectively. For reference, we report random baseline that samples according to the distribution of labels in QUDEVAL.
Reference-free metrics score substantially better than reference-based ones, which are only slightly better than random. This questions the validity using reference-based metrics for QUD parser evaluation, which we analyze further. Notably, only GPT-Ser for the Relevance crite- rion is close to the human upper bound (Table 2). The performance minority classes (see Table I for class frequencies) especially low. GPT-Scr turned be often the best metric across differ- ent criteria; however, we point out caveats with its usage later analysis.
Do reference-based metries capture QUD qual- ity? Appendix Figures 7, show the score distri- butions given by the best reference-based metrics split by ground-truth annotator label. For both ME- TEOR and BERTScore, the score distributions are nearly identical across the categories, suggesting that it is tricky if not infeasible to tune a good threshold for such automatic metrics to map to our evaluation framework. This observation aligns with results from Table where the metries perform only slightly better than the random baseline. Since QUDs reflect discourse interpretations and hence are subjective, we believe this is strong evidence
Do these metrics rank systems correctly? It is conceivable that, despite these metrics being im- perfect detectors of errors, they might still give reliable aggregate judgments about systems. To visualize the impact of applying these metrics to score systems, we show the predicted system per- formance using the best metrics for each category
is no entity overlap between Q and Q. We found this too restrictive for QUDs that are less factoid and entity-centric; thus our variant uses the arithmetic mean instead.

---

that imperfect standard reference- methods should not be used for QUD evaluation.
Acknowledgements
Special thanks to Kathryn Kazanas, Keziah Reina and Karim Villaescusa F. for providing data anno- tation for this project. This research was partially supported by National Science Foundation (NSF) grant IIS-2145479, We acknowledge the Texas Ad- vanced Computing Center (TACC) UT Austin for many of the results within this paper.
5.4 Discussion and Future Directions
As we established previously, QUD has utility for downstream generation tasks (Narayan et al., 2023; Newman et al., 2023; Wu et al., 2023). These re- cent lines of work indicate that question generation and answering is becoming more and more promi- nent as a way of handling discourse representation, despite a lack of rigorous evaluation of the gener- ated questions themselves. This paper fills this gap by establishing benchmark to evaluate automatic metries, which are annotated by trained linguists. The standard metrics evaluated in this section show that far from having a reliable automatic metric for QUD evaluation; future work can iterate on more sophisticated methods derive metrics, e.g., via supervised learning. iterative prompting, or ofain-of-thought prompting.
References
Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Evaluation methodologies in automatic question gen- cration 2013-2018. In Proceedings of the 1/th Inter- national Conference Natural Language Genera- tion, pages 307-317.
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguisties. Computa- rional linguisticss, 34(4):555-596.
Nicholas Asher, Nicholas Michael Asher, and Alex Las- carides. 2003, Logics of conversation. Cambridge University Press.
6 Conclusion
We present a dataset, QUDEVAL, for evaluating contextual question generation in the context of QUD discourse parsing. QUDEVAL implements prior theoretical evaluation criteria for QUD, oper- ationalized four criteria for which expert linguis- ties annotators give high-quality judgments. This dataset sheds light on the divergent performance of existing systems, and enables future work on de- veloping stronger automated metrics for evaluating QUD discourse parsing.
David Beaver and Brady Clark. 2009. Sense and sensitivity: How focus determines meaning. John Wiley & Sons.
Iz Beltagy, Mathew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Anton Benz and Katja Jasinskaja. 2017. Questions under discussion: From sentence discourse. Dis- course Processes, 54(3):177-186.
Daniel Buring. 2008. What's new (and what's given) in the theory of focus? In Annual Meeting of the Berkeley Linguistics Sociery, volume 34, pages 403- 423.
Limitations
QUDEVAL evaluates each QUD edge indepen- dently thus does not take into account the relation- ship between questions. While full, hierarchical QUD trees constrain the answer of QUD to entail the answers of its descendants (Roberts, 2012), QUD dependency tree does not inherit this prop- erty (Ko et al., 2023). Thus we leave for future work to explore potential global constraints.
Shuyang Cao and Lu Wang. 2021. Controllable open- ended question generation with new question type ontology. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing ( (Volume I: Long Papers), pages 6424-6433 6439.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006. 14799.
QUDEVAL is collected by trained linguistics students. We are yet to explore reliable ways to scale this to a crowdsourcing platform with the hopes of speeding up data collection. QUDEVAL is also limited to English newswire data; future work should explore other languages and genres.
Tuhin Chakrabarty, Justin Lewis, and Smaranda Mure- san. 2022. CONSISTENT: Open-ended question generation from news articles. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 6954-6968, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
The metrics we evaluated are baseline metrics. As pointed out in Section 5.4, this leaves headroom for future metric development.
'nttps://tacc.utewaa

---

Kordula De Kuthy, Nils Reiter, and Amdt Riester. 2018. QUD-based annotation of discourse structure and information structure: Tool and evaluation. In Pro- ceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Klaus Krippendorff. 2011. Computing Krippendorff's alpha- reaha-reiabbility.
Kalpesh Krishna and Mohit lyyer. 2019. Generating question question-answer -nswer hierarchies. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 2321-2334.
Xuan Long Do, Bowei Zou, Liangming Pan, Nancy Chen, Shafiq Joty, and Aiti Aw. 2022. Cohs-cqg: Context and history selection for conversational ques- tion generation. In Proceedings of the 29th Inter- national Conference on Compulational Linguisties, pages 580-591.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceed- ings of the Second Workshop on Statistical Machine Translation, pages 228-231.
Sujatha Das Gollapalli and See-Kiong Ng. 2022. QSTS: question-sensitie text similarity measure for ques- tion generation. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages -346.
Chin New Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jing Gu, Mostafa Mirshekari, Zhou Yu, and Aaron Sisto. 2021. ChainCQG: Flow-Aware Conversational Ques- tion Generation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint arXiv:2303.166:3
2061-2070.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. ChatGPT as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint 2rXiv:2303.1622
Yufang Hou. 2021. End-to-end neural information sta- tus classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1377-1388.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward functional the- ory of text organization. Text- sekz--tterdiscppinarry Jour- nal for the Study of Discourse, ((3):243-281.
Andrew Kehler and Hannah Rohde. 2017. Evaluat- ing an expectation-drive question-unet under- model of discourse interpretation. Discourse Pro- cesses, 54(3):219-233
Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine mne-graindd information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 795-804.
Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. 2022. Generating information-seeking conversations from unlabeled documents. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2362- 2378.
Mao Nakanishi, Tetsunori Kobayashi, and Yoshihiko Hayashi. 2019. Towards answer-unaware conversa- tional question generation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answer- ing, pages 63-71.
Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. 2020. Inquisitive question gener- ation for high level text comprehension. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Languuge P Processing, pages 6654-5555.
Shashi Narayan, Joshua Maynez, Reinald Kim Am- playo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lap- ata. 2023. Conditional generation with question- answering blueprint. Transactions of the Association for Computational Linguistics, 1
Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2022. Dis- course comprehension: A question answering frame- work to represent sentence connections. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1152-11764.
Preksha Nema and Mitesh M Khapra. 2018. Towards better metric for evaluating question generation systems. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 3950-3959.
Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srini- vas, Greg Durrett, and Junyi Jessy Li. 2023. Dis- course analysis via questions and answers: Parsing dependency structures of questions under discussion. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11181-11195.
Benjamin Newman, Luca Soldaini, Raymond Fok, Ar- man Cohan, and Kyle Lo. 2023. A controllable QA-based framework for deconttetualization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Tom Koemi and Christian Federmann. 2023. Large language models are of-e--o-theeare evaluators of translation quality. pages 193-203.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318.
Tianyi Zhang, Varsha Kishore, Felix Wu. Kilian Wein- berger, and Yoav Artzi. 2020. BERTScore: Evalu- ating Text Generation with BERT. In International Conference on Learning Representations.
A Annotation Interface
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind K Joshi, and Bon- nie L Webber. 2008. The Penn Discourse TreeBank 2.0. In Language Resources and Evaluation Confer- ence.
Figure 4 shows the full annotator instruction. In Figure 5, we present our system UT for QUD evalu- ation. The answer sentence, anchor sentence, and prior context are highlighted. Questions to be eval- uated are bolded, with other information displayed in gray.
Alec Radford, Jeffrey Wu, Rewon Child. David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Technical Report.
Each criterion has selectable options, and there is also text box for annotators to provide additional comments on specific errors. When the evaluation for QUD is completed, the corresponding block is marked in light blue to track progress. If question does not make sense, all criteria marked "skipped" and the block also highlighted in light blue.
Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification ques- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational L. Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 143-155.
Arndt Riester, Lisa Brunetti, and Kordula De Kuthy. 2018. Annotation guidelines for questions under discussion and information structure. Information structure in lesser-described languages. Studies in prosody and syntax, pages 403-443.
After submitting their answers, annotators can view the feedback in the table shown in Figure 6 to check their annotations and they can also copy the survey code directly.
Craige Roberts. 2012. Information structure: Towards integrated formal theory of pragmatics. Semantics and pragmatics, 5:6-1.
B Implementation Details for Reference-free Metries
(1) GPT-Scr (Answer Compatibility): We prompted GPT-4 to generate score between I to 100 for how well Sa answers Q. Our mapping uses tuned thresholds on the validation set: score of over 80 mapped to 'Direct and explicit', be- tween 60 and 80 was mapped to 'Unfocused' and score lower than 80 was mapped 'Not answered'. Our best prompt is present in Table 9.
Rohan Taori. Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction follooiig LLaMA model. https: //gi thub. cow/tatsu- http:/sttffoodddaaaaaaap
Jan Van Kuppevelt. 1995. Discourse structure, topicality and questioning. Journal of linguistics, 31(1):109- 147.
Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. 2023. Using natural language explana- tions rescale human judgments. arXiv preprint arXiv:2305.14770 147700
(2) Hou (2021) (Givenness): We found that running the full mention detection process is pro- hibitively slow, thus we extracted the maximum NP and their heads to speed up mention detection.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxi- ang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. ChatGPT good NLG evaluator? preliminary study. arXiv preprint arXiv:2303. 04048.
We differentiate answer leakage vs. hallucina- tions in 'new' mentions with the same mechanism as our rulc- metric: we check if content words (lemmatized; also excluding question (wh-) words, pronouns and names) in Q are present in Cs and S.. If there are content words absent in C3 but found in S., Q is labeled as 'answer leakage". If there are content words absent in both, they are deemed as "hallucination'. Otherwise, we label Q as 'no new concepts'.
Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. TED-Q: TED talks and the questions they evoke. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1118-11 -127.
Yating Wu, William Sheffield, Kyle Mahowald, and Junyi Jessy Li. 2023. Elaborative simplification as implicit questions under discussion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
(3) GPT-Ser (Anchor Relevance): The prompt is shown in Table 16. A score of 80 and above

---

is mapped to 'fully grounded', 80-20 to 'partially grounded', and below 20 'ungrounded'.
5), all instances were categorized as "No new concepts".
(4) BLEUI-sim: If the BLEU-1 score between the anchor and the question is greater than 0.05, we consider the QUD fully grounded in its predicted anchor. If the score falls between 0.01 and 0.05, we classify it as partially grounded. Otherwise, it is considered not grounded.
These observations indicate that the human- annotated in-context similarity scores somewhat correlate with both criteria experimented on here.
Question Similarity Annotation
The annotators were given pairs of (Q, Q) and Sa. well as the full article context (66, 66, and 67 questions generated from Ko et al. (2023), Chat- GPT and Alpaca respectively). Each question pair consisted of a numan-generater question and -ener--geeerated question for the same sentence within an article.
Similar to the setup in DCQA, scores between and 5 were given based on the extent of similarity, 5 being the score given to identical or paraphrased questions. Each question is annotated by annota- tors and their average score considered for analysis. The inter-annotator correlation is 0.728 and Krippendorff's a for the annotators 0.687.
Trends on Question Similarity and QUD Quality
Figures 9a and 9b, visualize the categoriza- tion of Answer Compatibility and Givenness for the QUD parsers across question similarity intervals. We can observe that for Answer Compat- ibility. instances with high similarity scores tend to mostly have S, that explicitly answers Q. As the similarity score decreases, greater percent- age of instances are either Unfocused or Not swered. For ChatGPT however, across all similar- ity score intervals, the focus of 5a answers Q most frequently, owing to the verbose nature of its gen- erated questions which leak the answer concepts.
A similar trend can be observed for Givenness, where the percentage of answer leakage and hallu- cination increases with the decrease in similarity score. However, across the 3 parsers, the most similar ( Q) pairs show answer leakage. Since Givenness is also an evaluation Sg. plausible cause for such observation is bad l prediction. To further investigate this, analyze the instances with the highest similarity scores (between 4 and 5) with the parser's Sg same as gold Si. generated for the same Sa. was observed that barring ChatGPT (which showed answer leakage for similarity score

---

Instructions
Go Back Annotation
This task investigates the capability of generate good reading comprehension questions. Imagine you are reading the following article where the first few sentences go like this:
spooked currency traders but prompted quiet little party among bond investors. Prices of long-term Treasury bonds moved inversely to the stock market as investors sought safety amid growing evidence the economy is weakening.
1 The stock market's
point, you may want answered in sentence 7:
question: How
the prices long-term Treesury increase? This question
7 At Its strongest, the Treasury's benchmark 30-year bond
than point, than $10 for each $1,000 face amount.
Indeed, we can view sertences in a document as answers to questions that come from the readers, it's just that these questions are not explicilly stated. The goal of the Al model is recover these questions and ask them explioifly Having these questions will helpful for readers who have trouble understanding the document.
Clearly, the question above good it anchored the second sentence, overly generic, and sentence an answer that fits right Al-generated questions are as good; consider
many points did Treasury's 's benchmark 30-year increase? This quesion is still answered by sentence 7, but is odd la imagine a reader coming up with this question from sentence 2. because the noton of "points" (and maybe even "30-year") has not been introduced and only very expert readers would be familiar with this! questions like, What happened
the hand, trying to understand the text.
This question way too generic be useful for puzzled reader
define few answer sentence. Sentences before the anchor form the question context. Sentences between the anchor and answer are faded as they should not provide extra information for the question.
call sentence the anchor sentence, ie., where the questions could reasonably arise from, and sentence 7 the
We you to judge quality of questions
fallowing criteria:
1. Is this question make sense?
Yes
Examples really bad queslions don't sense are:
Bad language (grammaar errors, incomplete)
Irrelevant the article gog.. peace talks in article vs. dlimate change in question)
Straight contradiction or much hallucination
If the answer to this is "NO", then the rest of the annotation should be skipped.
Does the "answer sentence" actually Explicit direct answer: 'answer sentance " main content answers this question. Note that the answer is provided multpple sentences and the 'answer sentence" is onl one of them, that is ok.
this question? There three postibillies here:
b. Unfocused answer: the "answer sentence" isn't really about answering the question, but some parts of the answer sentence answer the quesfion answer could be inferred. E.g., Sentence isn't about evidence weakening economy. hence an 'unfocused' answer to the question "How is the economy?" anchored in sentence 1.
Not-an-answer: the 'answer sentence" is not answering the question.
hard come up with? (By "new concepts" mean concepts that cannot be easily inferred by world knowledge from existing ones). are several possibillies here as well:
Does question
concepts
reader
This question does not contain new concapts.
Answer laakage: The question contains concapts that
AND in the anchor sentence the question context. 'How many points did Treasury's benchmark 30-year bonds increase?" would question that leaks new concepts in the anewer sentence: "30-year bonds would not be a concapt that moet people would automatically jump to.
the
Hallucination: The question contains new concepts. This indudes:
Concepts not article.
The question contains new concepts that are not in the anchor sentence or the question contexd, but can be found later in the document.
this question grounded well the anchor sentence we ask you to judge from a scale:
The question is fully grounded in the anchor sentence
question grounded c. The question is not all grounded the anchor sentence As an example, given the anchor UU.S exports of nuclear material cannol be adequately traced from country to country, according to congressional report", the question "What does the report the reason for the export ban" is only partially grounded, because although "the report" was mentioned, "exporl was not.
Some parts
anchor sentence
Extra Issues
Redundant question: question already anewered
Question too generic to be informatveican be asked anywhere in the document
Anything else
Once all criteria for one suestion are answered. the question will be marked lightblue. This way can know which question you haven't finished yet.
Figure 4: Annotation instructions for QUDEVAL.

---

Essay Context Article #1492
Gaetbee) How soudy forree?
Dhn slab wel mmm whet onyy hevm lina lurumn
ode mecogioa velue amar
caterdle womgate wew.ee Palt adkee
aear slpesricc Begg altaggas
Crberi
send avall beseet secar Biscot
C-1.10 Ves
Then seang calic mork bed vapcin eoform encpponnt
ohhrrd
Devar eueden comaan mulicle sumizoot
C/2.fo
are bords le bul Pa dean vartier
Doas arover secarol dly anaver quarion??
30000 pours
Sorvatcoane ser with people wery receenty fer ther seiven our rquats
aatae sallo
social
Start conolaa tutt canee eaily irfined vorid coovllgg Ago eartin
the col Bull oan 1o Avsiicaa bociee bave case
qussion co aonoppa.
Ancoer teakags
voting unading
ban desp at and
Oleer hauas
quniton indicse baliv Campleer c' bruae
canden annwered
U.S. Covatio intoviaiiiann
Figure 5: Annotation system
MTURK CODE: NTTRR_CODE:
BDOFHYF3GW
Do Book to Annotation
Clibbard
Questions
laaus
laaurs(tppin))
Multi-quaation
Nona
Fatso
Question
None
Fotee
Queetion
Nonw
Fatee
Question
None
Faboo
Questio 10
No
eAppped
siipped
#Mpped
skiped
False
Qusstion 11
Norw
Falee
Question 15
No
sAppped
skipped
sMpped
dkpped
Falso
Queetion 15
None
Falen
Figure 6: Annotation system feedback.

---

Step Question Generation
Examples for this question generation are:
Context: The stock market's woes spooked currency traders but prompted a quiet little party among bond investors. Prices of long-term Treasury bonds moved inversely the stock market as investors sought safety amid growing evidence the economy is weakening, But the shaky economic outlook and the volatile stock market forced the dollar lower against major currencies. The bond market got an early boost from the opening- sell-off in stocks. That rout was triggered by UAI. Corp.'s 's amouncement late Monday that the proposed management- buy-out had collapsed. The 80-point decline in the Dow Jones Industrial Average during the morning trading session touched off a flight to safety that saw investors shifting assets from stocks to Treasury bonds.
Target Answer: At its strongest, the Treasury's 's benchmark 30-year bond rose more than a point, or more than $10 for each $1,000 face amount.
Question: How much did the prices of long-term Treasury bonds increase? ...L3 more in-context examples]
By reading the context, generate a question that indicates how the Target Answer elaborates on earlier sentences. The Target Answer given should be the answer to the generated question. The question should reflect the main purpose of the Target Answer. It should not use information first introduced in the Target Answer and shouldn't It copy-paste phrases newly introduced in the Target Answer.
Step 2. Anchor Selection Context: ..ame as above]
Question: ...[same as above]
Anchor Sentence: Prices of long-term Treasury bonds moved inversely to the stock market as investors sought safety amid growing evidence the economy is weakening ...[3 more in-context examples]
By reading the Context, pick sentence from the Context such that the above Question arises from it. An Anchor Sentence is a sentence from the Context that the Question is most related to. The words and concepts from the Anchor Sentence used to generate the Question. The Target Answer cannot be the Anchor Sentence.
Table 8: Few-shot prompt for QUD generation (with example article and answer sentence)
article: PORT LAUDERDALE, Fla. Researchers are looking to the sun to give hunted and overfished sharks a new ray of hope. Using a special polar-powered tag, marine scientists now can study a shark's 's movements for up to two years by way of data beamed to satellites. Previously, researchers relied on tags that ran on batteries and sometimes died before all the information could be transmitted. The new tags are like a smartphone for marine animals,' said Marco Flagg, CEO of
Desert Star, Marina, Calif., company that offers solar devices." Just like smartphones, the tags have many sensors and communication capability. The The Guy Harvey Research Institute, based in Dania Beach, Fla., is looking to use solar tags to track certain species of fierce fish, including tigers, makos, hammerheads, oceanic white tip and sand sharks. The goal is to better understand their migratory patterns and ultimately keep their population healthy. Sharks are critical to the overall balance of occan ccosystems, hut commercial fishermen catch them by the millions for their fins, cartilage and meat," 'W''v ve learned a lot from tagging sharks, not least of which is that they are highly migratory,' said Antonio Fins, executive director of the Guy Harvey Ocean Foundation, which supports the institute." "They are not American sharks or Bahamian sharks or Mexican sharks They don't know borders or nationalities.
question: Why are rescarchers studying sharks and using solar-powered tags to track their movements? answer: Sharks are critical to the overall balance of ocean ecosystems, but commercial fishermen catch them by the millions for their fins, cartilage and meat.
score:
For the above article, give a score between 1 to 100 for how well the answer actually answers the question.
Table 9: GPT-Scr prompt for Answer Compatibility (with example article, question and answer)
article: FORT LAUDERDALE, Fla. Researchers are looking to the sun to give hunted and overfished sharks a new ray hope. Using a special solar-powered tag. marine scientists now can study a shark's 's movements for up to two years by way of data heamed to satellites. Previously, researchers relied on tags that ran on batteries and sometimes died before all the information could transmitted. The new tags are like a smartphone for marine unimals,' said Marco Flagg, CEO Desert Star, Marina, Calif,, company that offers solar devices. Just like smartphones, the tags have many sensors and communication capability. The Guy Harvey Research Institute, based in Dania Beach, Fla., is looking to use solar tags to track certain species of fierce fish, including tigers, makos, hammerheads, occanic white tip and sand sharks. The goal is to better understand their migratory patterns and ultimately keep their population overall balance of ocean ecosystems, but commercial fishermen catch them by the millions for their fins, cartilage and meat,' 'W'vv ve learned a lot from tagging sharks, not least of which is that they are highly migratory.' said Antonio Fins, executive director of the Guy Harvey Ocean Foundation, which supports the institute. "They are not American sharks or Bahamian sharks or Mexican sharks. They don't 't know horders or nationalities.
Sealthy,Shark, are critical to the
Which sentence in the article is closest to the sentence: 'Researchers are studying the movements of sharks using a special power- tag that can transmit data to satellites for up to two years." A:
Table 10: GPT-Ans prompt for Answer Compatibiliyy (with example article and GPT-4 generated answer)

---

The Justice Department in process of trying to gain control over a law that federal Judge David Sentelle recently called a 'monster' Needless say, was talking about RICO. With its recently revised guidelines RICO, Justice makes it clear that law currently holds too many incentives for abuse by prosecutors. The text of the 'new policy' guidelines from the Criminal Division are reprinted nearby. They strongly suggest that Justice's 's prosecutions of Drexel Burnham Lambert, Michael Milken and Princeton/Newoot violated notions of fundamental bainess.Justice attempting to avoid a replay of these tactics.This amounts to an extraordinary repudiation of the tenure New York mayoral candidate and former U.S. Attorne Rudolph Giuliani, who was more inclined to gathering scalps than understanding markets. The new guidelines limit the pretrial forfeitures of assets RICOed defendants and their investors, clients, bankers and others. This follows earlier new guidelines from the Tax Division prohihiting Prineeto/NNeworr-iig tax cases from masquerading as RICO cases.
Context:
Reference Question: What is the rationale for limiting the pretrial forfeitures?
Candidate Question: In what way are forfeitures limited now?
Score:
Given the Context, score the above Candidate Question for similarity with respect to the Reference Question on a continuous scale from 1 to 5, where score of means 'no similarity' and a score of 5 means "similar intent and phrasing"
Table 11: Prompt for assessing similarity between ( Q) (with example context, reference and candidate question)
Context: CHARLESTON, W. Downtown businesses and restaurants began to reopen after water was declared safe to drink in portions of West Virginia's is capital, but life has yet to return to normal for most of the 300,000 people who haven't it been able to use running water in the five days since a chemical spill. 2 It could still be days before everyone in the Charleston metropolitan area is cleared to use water, though officials say the water in certain designated areas was safe to drink and wash with as long as people flushed out their systems. 3 They cautioned that the water may still have a slight licorice-type odor, raising the anxieties of some who believed it was still contaminated. 4 'I wouldn't 't drink it for a while. I'm skeptical about it' said Wanda Blake, a cashier in the electronics section of a Charleston Kmart who fears she was exposed to the tainted water before she got word of the spill.
Question: How widespread were the effects of the spill?
Answer: Thursday's 's spill affected 100,000 customers in a nine-county area, or about 300,000 people in all. Does the question contain new concepts that a reader would he hard to come up with? (By "new concepts", mean concepts that cannot be easily inferred by world knowledge from existing ones). There are several possibilities here as well: This question does not contain new concepts. Answer leakage: The question contains new concepts that are in the answer sentence AND not in the context. Hallucination: The question contains new concepts. This includes: Concepts not in the article. The question contains new concepts that are not in the context, but can be found later in the document.
Given the Context, Question, and Answer, select one of the following options on the basis of your under- standing of the instructions.
I: No new concepts
2: Answer leakage
3: Hallucination
Table 12: GPT-CIs-2s prompt for Givenness (with example context, question and answer).
Here are a few examples for all cases:
Example 1: Context: 1 U.S. exports of nuclear material cannot be adequately traced from country to country, according to a congressional report.
Question: What does the report say is the reason for the export ban?
Answer Sentence: The report says hundreds of tons of plutonium and highly enriched uranium have accumulated worldwide, mostly from nuclcar power generation.
Selected option:
[3: Hallucination]
...C5 more in-context examples; each option has two examples]
Does the question contain new concepts that reader would be hard to come up with? (By "new concepts". I mean concepts that cannot be easily inferred by world knowledge from existing ones). There are several possibilities here well: This question does not contain new concepts, Answer leakage: The question contains new concepts that are in the answer sentence AND not in the context. Hallucination: The question contains new concepts. This includes: Concepts not in the article. The question contains new concepts that are not in the context, hut can be found later in the document.
Given the Context, Question, and Answer, select one of the following options on the hasis of your understanding of the instructions.
I: No new concepts
2: Answer leakage
3: Hallucination
Table 13: GPT-CIs-fs prompt for Givenness

---

Question:
How widespread were the effects of the spill?
Anchor Sentence: 'I know I've ingested it'. By Tuesday momiing, officials had given the green light to about 35 pereent of West Virginia American Water's ' customers. Does the question contain
up with? (By "new concepts", mean concepts that cannot be easily inferred by world knowledge from existing ones). There several possibilities here well: This question does contain concepts, Answer leakage: The question contains sentence AND not in the context. Hallucination: The question contains concepts. This includes: Concepts not the article. The question contains concepts that not the context, but be found later in the document.
enncepts that reader would be hard
concepts that the
Is the question well-grounded in the anchor sentence? Please evaluate using the following scale: The question is fully grounded the anchor sentence. Some parts the question are grounded in anchor sentence. The question is grounded all in the anchor sentence.
Based on the question same entity as the anchor, we consider question to be grounded.
anchor, please choose one of above options. If the question refers to the
Table 14: GPT-Cls-zs prompt for Anchor Relevance (with example question and anchor sentence)
Here are a examples all cases: Example
Question: What lawmakers think about this issue?
Anchor Sentence: U.S. exports nuclear material cannot be adequately traced from country to country, according to a congressional report.
Result: [1: question is fully grounded the anchor sentence.]
...C5 more in-context examples; each option has two examples]
the question well-grounded the anchor sentence? Please evaluate using the following scale:
The question is fully grounded in the anchor sentence. 2: Some parts of the question are grounded in the anchor sentence. The question is not grounded at all in the anchor sentence. Based on the question and the anchor, please choose one of the above options. If the question refers to the same entity as the anchor, we consider the question to be grounded.
Table 15: GPT-CIs-fs prompt for Anchor Relevance
Question:
What foreign policy experts say about the issuc?
Anchor Sentence: U.S. exports of nuclear material cannot be adequately traced from country to country, according to a congressional report.
Based on the question and the anchor, give score between 1 to 100 for how confident you are about the question grounded in anchor sentence. the question refers to the same entity as the anchor, we consider the question to be grounded.
Table 16: GPT-Scr prompt for Anchor Relevance (with example question and answer sentence)

---

Question
What is the main objective of Clinton in forging a wedge between Milosevic and the Serbs
What will happen owners who cannot distinguish their owners' voices?
Table 17: Examples questions that failed the Language Quality criterion.
Question
Answer
What happened after he took the students to Poland??
But he wasn't prepared for the uproar that followed.
What was the reason for the scheduled resumption of peace talks in Ingushetia?
The scheduled resumption of talks in the town of Sleptsovsk came two days after agreement cease-fire, calling for both sides to stop using heavy tillery Tuesday.
limited
Table 18: Examples where the answer sentence an "unfocused" label for Answer Compatibility. In the first example, the answer does not directly talk about an event. In the second example, the answer is not the focus of the sentence. This because the cease fire alone doesn't seem to the reason for the resumption of peace talks
Anchor (S1): SEATTLE There's little lyrical language to be found in the most recent international report on climate change.
Question: What is the essence of the IPCC report?
Answer: So when bad cold kept him in the house one weekend, Johnson decided to distill the report to its essence via a centuries-old Japanese art form: haiku.
Anchor (S1): Fundamental freedoms are lagging behind rapid economic growth in Vietnam, according to new U.N. report.
Question What is the purpose of the U.N. Human Rights Ombudsman's visit?
Answer The group visited Vietnam for one week October last year.
Table 19: Examples where Q is labeled "hallucination" for Givenness. Both anchor sentences are identified to be the first sentence. In the first example, IPCC is not in the common ground (S1) nor in S., and inferring that the international report was written by IPCC is quite a leap; the second sentence. "U.N. Human Rights Ombudsman's " is also unknown to the reader.
Answer Compatibility
Annotationl:
not answered, Annotation2: not answered, Annotation3: unfocused
Anchor Sentence: But the tcchnological sleuthing it took a group of Carnegie Mcllon University students and alumni to recover and preserve some digital images apparently created and stored by Andy Warhol on old-school floppy computer disks nearly 30 years ago is a tale worth telling.
Question: What was unique about these digital images? Answer Sentence: Those three images of an altered Botticelli's 'Venus.' a Warhol self-portrait, and a Campbell's 'sss soup can of 28 that were found on the disks were enough to excite Warhol fanatics around the world over the possibility that something anything new by the King of Pop Art had been revealed.
Anchor Relevance
Annotationl: fully grounded, Annotation2: partially grounded, Annotation3; partially grounded
Anchor Sentence: Every spring, from Florida to New Jersey, crabs that look more like fossils than a postcard for passion make their way ashore by the thousands when the moon is bright to lay millions of eggs that provide critical food for migrating shorehirds.
Question: What huppened to their numbers? Answer Sentence: But in the 1990s, their numbers began falling.
Table 20: Examples where annotators disagree. In the first example, the Answer Sentence mainly focuses on the excitement around Warhol fanatics on the revelation of the three digital images and what they were. Although it mentions that this maybe new work, suggesting might be unique (quite subjective), it doesn't seem to be the main focus of the Answer Sentence, justifying the 'unfocused' annotation. For the second example, no concept in the question is discourse-new new and wondering about what happened to the numbers natural to some readers, justifying the fullly grounded' annotation of one of the readers.

---

Direct and explicit
Unfocused
Not answered
ME TEOR
Count
Count
Count
200
-1.0
-0.5
-1.0
-05
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Figure 7: Distribution of automatic metrics for Answer Compatibility.
new concepts
Answer leakage
Hallucination
400
Count
Count
100
Count
20
200
50
-1.0
-0.5
0.0
0.5
1.0
-1.0
-05
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Figure 8: Distribution of automatic metrics for Givenness.

---

similaity
cotegory
Aron tanage
30
20
10
10-
(a) Answer Compatibility
(b) Civenness
Figure 9: Categorization of Answer Compatibility, Givenness across similarity score intervals


Reviews:
[{"review_id": "LisicODPX2", "reviewer": "Reviewer_FkHr", "summary": "", "questions": "- The paper is very dense. It would have been nice to actually talk about the discourse parsing task in more detail, show\nan example of a dependency tree (either in the main text or the appendix). \n\n- Some more detail on Ko et al.'s parser would also have been useful. \n\n- Regarding the modeling, have you considered to use previously generated QUDs as input to generate the question in the next time step? \nThis might prevent models from generating repeated questions/answers. \n\n- Please explain why you generated a few QUDs from several documents rather a complete set from a lesser number. \n\n- Table 1, are differences in parser performance statistically significant?\n\n- Table 4, please report what metric you are using to evaluate the automatic metrics. \n\n- Table 5, why are you using ROUGE? You do not care about recall. Also please explain what QSTS is. \n\n- Table 7, do you mean Spearman's rho? Are are these correlations statistically significant? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and the recognition of the merit of our work. Below we address “Questions for the authors”:\n\n_Re: example of a dependency tree and details of Ko et al.’s parser:_\\\nWe appreciate your suggestions and will definitely include more examples and a short summary of Ko et al’s parser in the appendix.\n\n_Re: using previously generated QUDs:_\\\nThis is an interesting suggestion! We focused more on evaluating parsers, so this was a bit outside the scope of what we were experimenting with here, but we agree that's a great direction for future work.\n\n_Re: Please explain why you generated a few QUDs from several documents rather a complete set from a lesser number:_\\\nWe think that the utility of having a more diverse coverage in terms of topics across documents is higher (note that since the annotation is high in cognitive load, evaluating full, long documents would further complicate this task).\n\n_Re: Table 1, are differences in parser performance statistically significant?_\\\nWe ran the Wilcoxon rank sum test between all pairs of parsers. Other than language quality, differences across most parser comparisons in table 1 are statistically significant (p<0.05). Language quality differences are not significant as all systems were able to generate fluent and sensible questions; the Ko et al model does not differ significantly from ChatGPT or Alpaca in terms of unfocused answers, nor with ChatGPT in terms of anchor relevance. We’ll add this into the table. \n\n_Table 4:_ we are using per-class F1 and macro-averaged F1. Although we mentioned the macro F1 in the column name, we overlooked that the per-class F1 was not specified there. We will add this to the caption; thanks for flagging this.\n\n_ROUGE:_ We reported ROUGE-F1 in the context of *reference-based* evaluation only, since  it is often used for the evaluation of QG tasks. There, recall does matter if we are measuring the similarity between the generated output and the reference input.\n\n_QSTS_ (Gollapalli and Ng, 2022) stands for Question-Sensitivity Text Similarity. It explicitly represents the question class and named entities present in a given question pair and combines them with dependency tree information and word embeddings to measure the semantic similarity between 2 questions. QSTS is a state-of-the-art reference-based metric for question generation evaluation, hence we included it. We will add a short description to the appendix.\n\n_Table 7_ indeed has the Spearman Rank Correlation coefficients for the automatic metrics like METEOR, ROUGE against the human annotated QuD similarity with all the values being statistically significant. We will clarify significance in the caption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "QUD parsing is not well-known or established in the research community. The paper takes an important step \ntowards evaluating the output of QUD parsers, it proposes theoretically established criteria and shows that \nthe proposed annotation scheme leads to relatively high annotation agreement. The paper further \nproposes automatic metrics for the evaluation of QUD output, demonstrating there is room for improvement.\nIt further provides background on QUD, and develops parsing models based on GPT. ", "reasons_to_reject": "I cannot find any reason to reject, the paper is interesting, a serious effort towards the automatic evaluation of QUDs. ", "questions_to_authors": "- The paper is very dense. It would have been nice to actually talk about the discourse parsing task in more detail, show\nan example of a dependency tree (either in the main text or the appendix). \n\n- Some more detail on Ko et al.'s parser would also have been useful. \n\n- Regarding the modeling, have you considered to use previously generated QUDs as input to generate the question in the next time step? \nThis might prevent models from generating repeated questions/answers. \n\n- Please explain why you generated a few QUDs from several documents rather a complete set from a lesser number. \n\n- Table 1, are differences in parser performance statistically significant?\n\n- Table 4, please report what metric you are using to evaluate the automatic metrics. \n\n- Table 5, why are you using ROUGE? You do not care about recall. Also please explain what QSTS is. \n\n- Table 7, do you mean Spearman's rho? Are are these correlations statistically significant? ", "paper_topic_and_main_contributions": "This paper presents a dataset, QUDEVAL, for evaluating question generation in the context of\nQUD discourse parsing. QUDEVAL implements \nprior theoretical evaluation criteria for QUD, for which expert linguistics annotators give high-quality judgments.  Using the annotations, the authors evaluate the performance of existing QUD parsers and their own models. \nThey further come up with automatic metrics for the assessment of automatically generated QUDs. The annotations\nand evaluation protocol are made available (annotator instructions are provided in the appendix). ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "whRdCPq54y", "reviewer": "Reviewer_WebN", "summary": "", "questions": "- There is little explanation on how the reference-based metrics are mapped to integers. Can you explain this?\n\n- Fn2: What is the 'temperature', why is it set to 0.0 and not to some other value?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\nAs stated in the paper, QUD parsing is an emerging area of research. The lack of prior computational work is not an indication for a lack of work in the future. Considering the history of RST parsing, it took decades following Mann & Thompson’s theoretical work for it to become widely used in the NLP community, even after the first corpora & parser led by Daniel Marcu. We believe that it is important for an emerging area to have a sound evaluation paradigm grounded in linguistic theory, and this work aims to achieve that. The need for rigorous evaluation is further manifested by existing work that uses QUD for generation while lacking an intrinsic evaluation for question generation (e.g., Narayan et al ArXiv 2022 now TACL 2023, Newman et al. 2023, etc).\n\nAs for the fact that we cover a few different aspects: this is necessary, as linguistic theories point to the fact that all these criteria need to be satisfied _jointly_. Future work can explore combining them into a “single score”, but we emphasize the need for fine-grained evaluation that is one of the key insights in recent work that evaluates NLG system outputs. \n\n_Re: multiple reference:_\\\nWhile we agree that multiple references would be very nice to have, we want to point out that for most high-entropy generation tasks, reference-based evaluation is inherently flawed given the high number of valid outputs that are semantically different; Section 5 brings up this point. Our stance is that reference-free metrics with sound linguistic principles are the best way forward for such tasks.\n\n_Re: explanation of mapping for reference-based metrics:_\\\nThanks a lot for catching this! The mapping is the same mechanism as reference-free, stated in lines 452–457; we will fix this. \n\n_Why temperature is 0.0:_\\\nTemperature is a hyperparameter that controls the randomness of the generated output. We noticed that increasing the temperature for our setting resulted in the generated questions and anchors containing concepts which are outside the context provided. Hence it was set to 0. We will add this point as a footnote into the paper.\n\n_“Typos Grammar Style And Presentation Improvements”:_\n\n_Re: 449:_ thank you for catching this. We'll fix this.\n\n_Re: macro F1:_ thank you for the suggestion. We opted for a Macro F1, along with a per-class F1 for all categories, because the number of classes is small (3), and not all of the aspects we measure have an ordinal semantics associated with them. We can definitely add an MAE measure to those that have this property.\n\n_Re: line 578:_ We apologize for the confusion that the reference to Table 2 makes. “High reliability” doesn’t necessarily mean high agreement, which isn’t true for most of the NLP tasks (e.g., coreference resolution, entailment, etc). Our annotators are linguists well-trained for this task, and we have been through multiple rounds of discussion about potential disagreements. We are highly confident about the quality of the annotation.\n\n_References mentioned above:_ (both cited in the paper)\n- Narayan et al., Conditional Generation with a Question-Answering Blueprint, TACL 2023\n- Newman et al., A Controllable QA-based Framework for Decontextualization, ArXiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and the annotation protocol of the resource is very carefully described. Therefore, I presume that the annotation is of good quality. I appreciate this. \n\n- The human upper-bound seems to suggest that automatic metrics for QUD parsing (even the ones based on LLMs) have a long way to go to properly solve the task. If true, this paper can show a weakness of LLMs, which may valuable.", "reasons_to_reject": "Even though the paper tries motivating the task/resource, I have some doubts on whether an evaluation benchmark for evaluation metrics of QUD is/will be a relevant resource for the community. One reason is that QUD parsing doesn't seem very active. Another reason is that it is not really clear what conclusions we can draw from evaluating a metric on QUDeval. If a purpose of the benchmark is to enable QUD parsing researchers to better select a metrics for scoring QUD parsers, then from the presented results I think they will have a hard time doing this, since the metrics seem to score so differently in different aspects. Maybe it is also not the automatic metrics that are the culprit, but the main problem is that there aren't multiple references?", "questions_to_authors": "- There is little explanation on how the reference-based metrics are mapped to integers. Can you explain this?\n\n- Fn2: What is the 'temperature', why is it set to 0.0 and not to some other value?", "paper_topic_and_main_contributions": "The paper proposes QUDeval, a data set containing automatically generated (implicit-question, anchor) pairs with human quality annotation. The goal of the paper is to provide a framework for evaluation of evaluation metrics for QUD parsing. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 449: there seems to be an 'and' too much or some grammar issue going on.\n\n- Macro F1 doesn't seem like a good choice to evaluate the prediction of an ordinary scale. Obviously, the score 4 fits much better to 5 than to 1. With a softer calculation, the metrics may perform better, also many of them do not naturally assign a category and require some (possibly messy) thresholding (on which I find too little information).\n\n- 578: I don't think table 2 shows \"high reliability\" of agreement among human annotators."}}, {"review_id": "PpTEZh5GFf", "reviewer": "Reviewer_ttgK", "summary": "", "questions": "Does the order of the subtasks matter? Could an anchor sentence be identified first, followed by question generation?\n\nCan there be more than one anchor sentence for a given question and answer?\n\nCan the first sentence in the document be considered an answer sentence (see def in 206-208)?\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Re: motivation:_\\\nPlease also see the first point of our response to R2. We will incorporate this framing of the motivation into the paper when extra space is available.\n\n_Re: annotation:_\\\nWe address your concerns towards the end of this response. We believe that most of the concerns came from the lack of full document context rather than annotator error; we will address that by putting more context into the tables.\n\n_Re: Does the order of the subtasks matter? Could an anchor sentence be identified first, followed by question generation?_\\\nThat's a good question. Yes, it's possible. In fact, that's how Ko et al’s QUD parser operates (as indicated for R1, we will include a brief summary of their parser in the appendix). In GPT models and Alpaca, however, we found that the current order better guides those models, resulting in a superior QUD output.\n\n_Re: Can there be more than one anchor sentence for a given question and answer?_\\\nIt is both theoretically and operationally possible that more than one sentence satisfies the criteria of being an anchor. In practice, we see that this depends on varying levels of specificity of the question and the document context. Because of this, our evaluation framework takes into account this possibility by independently evaluating each QUD and its predicted anchor.\n\n_Re: Can the first sentence in the document be considered an answer sentence (see def in 206-208)?_\\\nIn our experiment, the first sentence is not considered an answer sentence. It is possible, however, to treat it as one if the document title is taken into account. As neither datasets nor parsers in prior work did so, we did not feel right to evaluate it as such. \n\n_Re: Line 300 - the term ‘’irrelevant” is too strong here. The question is clearly not irrelevant to the anchor sentence since both refer to nuclear materials:_\\\nWhat we meant here is only scoped towards the specific requirement for anchor groundedness, i.e., concepts in the question needs to be hearer-old. We will clarify this. While we agree that both the anchor sentence and question mention nuclear materials, the context of usage is very different. The anchor sentence mentions what’s present or missing in the report, namely that the report talks about the export license of nuclear material. The question is centered around the restrictions on nuclear exports and transfer of nuclear material. The concept of “restrictions” is the main focus of the question which is irrelevant to the anchor sentence that provides more information on the contents of the report. \n\n_Re: Table 19: the second example doesn’t seem like an example of “unfocused”. The answer sentence implies that the resumption of peace talks came as a result of the cease-fire. What is the focus if not this?_\\\nThe annotators did provide their reasoning for the unfocused label (before we submitted the paper): given the article context, the annotator didn’t think that the cease fire alone is the reason for the resumption of peace talks; rather they believed that it should be the reason for the cease fire.\n\n_Re: Table 20: there’s a clear difference between the two examples of hallucination. In the first example, the IPCC report is in fact mentioned in the first sentence - it’s just that the IPCC acronym is not used._\\\nWe disagree. The anchor sentence just says \"international report on climate change.\" It is quite a leap to infer that the international report was necessarily written by the IPCC (\"Intergovernmental Panel on Climate Change\") and not some other organization. In our view, most readers would not write this question unless they already had deep prior knowledge of this topic.\n\n_Re: The annotator disagreement examples given in the Appendix (Table 21) should be discussed. The second one (anchor relevance) looks like an error on the part of one annotator, although on lines 376-377 it says that no instance of task misinterpretation on the part of an annotator was found? How could the question be “fully grounded” in the anchor sentence in this example?_\\\nWe will happily discuss the disagreements. For this particular example, we disagree that this is not “fully grounded”: no concept in the question is discourse-new, and that wondering about what happened to the numbers (that were explicitly mentioned in the anchor) is natural to some readers (though there is a degree of subjectivity here).\n\n_Re: Line 578: do people really agree with “high reliability”? IAA is only moderate._\\\nWe apologize for the confusion that the reference to Table 2 makes. “High reliability” doesn’t necessarily mean high agreement, which isn’t true for most of the NLP tasks (e.g., coreference resolution, entailment, etc). Our annotators are linguists well-trained for this task, and we have been through multiple rounds of discussion about potential disagreements. We are highly confident about the quality of the annotation.\n\nWe also thank the reviewer for the clarification and presentation suggestions; we appreciate them and will address/fix them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A new and useful dataset\n* A new set of well justified evaluation criteria which will likely be used by other researchers working on this task\n* A thorough set of experiments with some findings that reveal avenues for future progress on this task\n* A clearly written paper\n\n", "reasons_to_reject": "* Motivation - the introduction doesn’t do a good enough job in explaining why this is a useful task. The role of QUD in the applications listed in 43-46 need to be spelled out more.\n\n* Annotation quality - the inter-annotator agreement isn’t particularly high and some of the examples in the appendix raised a few questions for me (see Annotation section below)\n\n* Relatively minor issues in presentation which nonetheless should have been caught before submission - see below\n\n", "questions_to_authors": "Does the order of the subtasks matter? Could an anchor sentence be identified first, followed by question generation?\n\nCan there be more than one anchor sentence for a given question and answer?\n\nCan the first sentence in the document be considered an answer sentence (see def in 206-208)?\n\n\n", "paper_topic_and_main_contributions": "This paper introduces an evaluation framework for the Questions Under Discussion (QUD) task. The task itself involves generating a question from an article that is answered by a sentence within the article, and determining the anchor sentence within the article which inspired the generated question. The first contribution of the paper is a set of evaluation criteria for this task. The criteria are inspired by an error analysis of existing QUD systems and by evaluation criteria used in previous studies. The second contribution of the paper is the application of these criteria to the output of 3 QUD systems (one fine-tuned, and two based on LLM prompting) and to the dev set of an existing QUD dataset (DCQA) whose questions have been manually generated by crowdsourcing. The system outputs and the human-generated questions are annotated by a single trained annotator according to the evaluation criteria - this data will be released. A third contribution is the analysis of the system and human performance on the task - the main findings here are that the ChatGPT system produces the questions most compatible with the answer sentences but is prone to answer leakage (where the question is grounded in information only available in the answer sentence), and that the system fine-tuned on the DCQA dataset is the most prone to hallucination but is also best at predicting the anchor sentence. The Alpaca system has a tendency to repeat the same questions, given the same article as input. The human-generated questions are evaluated highest overall, although they are not deemed as compatible with the answer sentence as the ChatGPT-generated questions. Finally, the last contribution is the use of the new dataset to evaluate automatic metrics (reference-free and reference-based). The results show that there is much room for improvement for these metrics, particularly the reference-based ones.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Clarification\n* 169-171: say why these criteria are insufficient\n* 381 - clarify that these 150 are human-generated\n* 406 - “Qualitative Analysis” based on what? Provide more detail.\n* Section 5.2 - clarify where the reference questions come from. The crowd-sourced DCQA questions?\n\nAnnotation\n* Line 300 - the term ‘’irrelevant” is too strong here. The question is clearly not irrelevant to the anchor sentence since both refer to nuclear materials.\n* Table 19: the second example doesn’t seem like an example of “unfocused”. The answer sentence implies that the resumption of peace talks came as a result of the cease-fire. What is the focus if not this?\n* Table 20: there’s a clear difference between the two examples of hallucination. In the first example, the IPCC report is in fact mentioned in the first sentence - it’s just that the IPCC acronym is not used.\n* The annotator disagreement examples given in the Appendix (Table 21) should be discussed. The second one (anchor relevance) looks like an error on the part of one annotator, although on lines 376-377 it says that no instance of task misinterpretation on the part of an annotator was found? How could the question be “fully grounded” in the anchor sentence in this example?\n* 578: do people really agree with “high reliability”? IAA is only moderate\n\nPresentation Details\n* Table 5: is the heading wrong in the last section? I assume these are the Anchor Relevance results?\n* There are a lot of informative examples in the appendix and it’s distracting to have to switch back and forth. Consider moving a few to the main paper for a nine-page camera-ready version.\n* Grammar error in the instructions to the annotators (on the criteria related to language quality)\n* Table 1: No col not needed for Lang criteria\n* Table 4: the upper bound from Table 2 should be included somewhere. Maybe in the caption?\n"}}]

Author rebuttal:

Meta-review: This paper presents detailed evaluation criteria, new evaluation data, and an evaluation of multiple outputs for the QUD discourse parsing task, an emerging area of computational discourse modeling. Reviewers generally agree that the quality of the paper is high, that the level of detail is a strong point, and that there are novel contributions here, while their views on the excitement and potential of this work moving forward are more mixed. All reviewers assign positive soundness scores, and two reviewers give a strong excitement score. Considering the relatively unexplored state of QUD research at this point, I would like to see more papers in this area get a chance to develop the framework further, and especially the contribution of new data could be substantial at this point. If accepted, I would ideally like to see this presented as a poster to encourage discussion and allow interested researchers to find out more about the data and how they can use it to develop these ideas further.
Decision: Accept-Main