Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models’ Understanding of Discourse Relations (2024)

Abstract: While large language models have significantly enhanced the effectiveness of discourse relation classifications, it remains unclear whether their comprehension is faithful and reliable. We provide DISQ, a new method for evaluating the faithfulness of understanding discourse based on question answering. We first employ incontext learning to annotate the reasoning for discourse comprehension, based on the connections among key events within the discourse. Following this, DISQ interrogates the model with a sequence of questions to assess its grasp of core event relations, its resilience to counterfactual queries, as well as its consistency to its previous responses. We then evaluate language models with different architectural designs using DISQ, finding: (1) DISQ presents a significant challenge for all models, with the top-performing GPT model attaining only 41% of the ideal performance in PDTB; (2) DISQ is robust to domain shifts and paraphrase variations; (3) Open-source models generally lag behind their closed-source GPT counterparts, with notable exceptions being those enhanced with chat and code/math features; (4) Our analysis validates the effectiveness of explicitly signalled discourse connectives, the role of contextual information, and the benefits of using historical QA data.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6277–6295 August 11-16, 2024 ©2024 Association for Computational Linguistics
We then evaluate language models with different architectural designs using DISQ, finding: (1) DISQ presents a significant challenge for all models, with the top-performing GPT model attaining only 41% of the ideal performance in PDTB; (2) DISQ is robust to domain shifts and paraphrase variations; (3) Open-source models generally lag behind their closed-source GPT counterparts, with notable exceptions being those enhanced with chat and code/math features; (4) Our analysis validates the effectiveness of explicitly signalled discourse connectives, the role of contextual information, and the benefits of using historical QA data.

heading: 1 Introduction
text: While language models can generate coherent and seemingly human-like text, their true grasp of discourse relations remains unclear. Traditionally, discourse relation prediction has been evaluated using accuracy scores from classification tasks. However, task accuracy may not reflect a reliable understanding, as a high score might not reflect sound reasoning or consistent comprehension of discourse semantics. Drawing inspiration from Socrates’ method of examining his students’ understanding through a series of questions, we introduce Discursive Socratic Questioning (DISQ), a new method
that assesses a model’s understanding of discourse relations by requiring systematic accuracy over multiple questions, rather than just a single accurate prediction (Figure 1).
While QA-based evaluation is well-researched (Fabbri et al., 2022b; Hu et al., 2023), DISQ addresses unique, discourse-centric challenges: What to Ask: While many discourse spans can form questions, not all provide salient insights into discourse understanding. While previous research like e-SNLI (Camburu et al., 2018) rely on labor-intensive human annotations to extract relevant signals for natural language inference, we advocate the use of in-context learning (ICL). This harnesses the power of large language models to annotate salient discourse signals efficiently. How to Ask: The manner in which questions are framed is essential for assessing three key attributes of a model’s faithfulness: (1) Responsiveness to Targets: We generate questions centered on key spans with ground-truth semantics, to which the model should
6277
respond affirmatively (e.g., affirming a cause in a contingency relation). (2) Robustness to Counterfactuals: We design questions based on counterfactual semantics, expecting the model to negate the query (e.g., dismissing contrast in a contingency relation). (3) Logical Consistency: We formulate converse questions with equivalent semantics, anticipating the model to deliver consistent responses (e.g., aligning answers to a result question with its corresponding reason). Berglund et al. (2024) find that LLMs struggle with the “reversal curse” (finding it hard to infer “B is A” from “A is B”). We introduce this facet as a discourse-centric test to the broader LLM research on models’ logical consistency.
We base our evaluation on the widely-recognized PDTB corpus (Prasad et al., 2008). Initially, we select 11 second-level discourse senses and employ in-context learning to select salient evidence for questioning. Subsequently, we invite human experts to validate our chosen evidence, underscoring the soundness of our questions.
We apply DISQ to a range of models encompassing various architectures and sizes. Notably, many models demonstrate zero-shot capabilities, even without training on discourse-specific data. This suggests that the prevailing training paradigms yield emergent ability to understand discourse semantics. We find that while larger, closed-source models excel in responsiveness, they also struggle with the “reversal curse”, indicating a probabilistic approach to discourse semantics without full logical consistency. We further demonstrate that DISQ’s measure is robust against domain shifts (TED-MDB corpus (Zeyrek et al., 2018)) and question paraphrasing. We highlight the benefits and limits of using linguistic features like discourse connectives, context, and historical QA to enhance comprehension faithfulness1.

heading: 2 Question Bank for DISQ
text: We detail “what to ask” in DISQ by identifying key events and use in-context learning to identify salient evidence. We then perform human verification to guarantee the quality of these questions.

heading: 2.1 Preliminaries
text: What counts as discourse understanding? Organized text makes sense as discourse elements
1The software and data of DISQ are publicly available at https://github.com/YisongMiao/DiSQ-Score.
link the text together. Such linking elements are referred to as cohesive devices (Halliday, 1976), including reference, ellipsis, and lexical cohesion. Formally, two textual spans s1 and s2 are linked by the relation r. We define (s1, s2, r) as an evidence triple to understand the discourse. Concretely,
Definition 1. (s1, s2, r) is an evidence triple to understand the discourse, where Arg1 and Arg2 are two given discourse arguments participating in a discourse relation R, and two contiguous spans s1 ∈ Arg1 and s2 ∈ Arg2 link the two arguments into a coherent discourse with semantic relation r.
We argue that a model understands discourse when it reliably identifies such triples. As shown in Table 1, the event triple (s13, s21, r) is the salient signal for the causal semantics. A model must identify them to understand the Contingency discourse relation (R).
Define a proxy for discourse understanding: We approach the notion of understanding by questioning. We interrogate the model with a set of questions concerning different semantic relations and text spans. If a model is said to understand, it must answer questions in a manner under three criteria: (1) Responsiveness to targeted questions (e.g., providing affirmative “True” answers, without abstaining); (2) Robustness against counterfactual queries (e.g., responding with “False” or abstaining to answer); (3) Consistency across consecutive responses (e.g. consistently saying “True” (or “False”) to converse questions).

heading: 2.2 Annotating Salient Signals Using ICL
text: As illustrated in Table 1, not all spans serve as salient signals for understanding the discourse. For instance, the span “I want to buy” (s11) lacks a causal connection with “It’s very frustrating” (s21).
It is important to filter unessential ones to make our evaluation reliable. Earlier research in explainable NLP, such as e-SNLI (Camburu et al., 2018), adopts a similar span-based reasoning formalization. In their corpus, humans are tasked with annotating key spans pivotal to understanding the NLI labels. However, this method is labor-intensive and lacks generalizability to other tasks.
To address this challenge, we introduce a new annotation method requiring minimal human intervention. (1) Candidate extraction: We first extract m spans from Arg1 and n spans from Arg2, resulting in m× n evidence triples (s1, s2, r) that may act as evidence for discourse understanding. The spans are similar to elementary discourse units (EDU) in the RST/SDRT theories. They are characterized by a self-contained subject–verb–object (s–v–o) structure, identified by semantic role labeling (SRL), can be regarded as events. However, only some of them are salient indicators of discourse relation. (2) Select salient pairs: Subsequently, we leverage In-context Learning (ICL) (Brown et al., 2020) to identify these pivotal evidence triples. The underlying premise is that by providing the model with exemplars of salient versus non-salient triples, it can distinguish them in new instances. Formally, we ask model to predict if r holds between events (s1, s2) in a discourse Arg1, Arg2, R with in-context learning. It makes its prediction on a new input X after being exposed to both a positive and a negative example (each example is structured as X → y).
Figure 2 presents the in-context learning (ICL) template in JSON, designed for step-by-step reasoning to predict event relation (ER). The ICL performs a binary classification on whether ER holds. The output y mirrors human reasoning and includes: “DR Summary”, condensing the discourse relation (DR) in model-specific terms; “Event1 comprehension”, linking Event1 to argument Arg2 (and Event2 to Arg1) and examining their discourse roles; “What if ER holds” and “What
if ER does not hold” exploring event relation (ER)’s influence on the discourse; and “Predicting ER” followed by the “Final prediction”, synthesizing the analysis to predict ER between events. We implement ICL using the LLaMA2-13B model, employing just one positive and one negative example for each discourse relation (Appendix A).

heading: 2.3 Dataset Statistics
text: Table 2 outlines the 11 Level-2 relations from PDTB-3.0, with modifications in the Contingency relation to merge smaller groups. It lists each discourse relation’s corresponding event relation and whether the question type is uni- or bi-directional, aiding in generating converse questions. For bidirectional relations, the converse mirrors the original (e.g., “A happens at the same time as B” and vice versa), while for uni-directional, the converse flips the sequence (e.g., “A happens before B” becomes “B happens after A”). We annotated all implicit discourse in PDTB test set (Sections 21 and 22 in the PDTB). For counterfactual analysis, we chose 5 irrelevant r not pertaining to a particular discourse R. Additionally, both targeted and counterfactual questions include converse inquiries, amounting to 22,380 questions in total. See Appendix C.1 for detailed statistics, including those for the TED-MDB dataset. DISQ operates at the finest-grained level in the PDTB taxonomy, using Level-2 or Level-3 distinctions as applicable. Level-2 results are reported for consistency, with detailed Level-3 results in Appendix C.6.
Even though our current implementation is limited to PDTB-style discourse analysis, extending DISQ to other discourse formalisms is possible
(Braud et al., 2023). As long as they provide clear EDUs and discourse relation annotations, they can be incorporated. For example, Rhetorical Structure Theory (RST) (Mann and Thompson, 1987) is a viable option since it provides these two elements. RST is particularly interesting because it includes relations not covered by PDTB (e.g., the “summary” relation in RST can be approached as a “Does A summarize B?” question).

heading: 2.4 Human Verification
text: To guarantee the reliability of DISQ evaluations, we verify the salient event pairs identified by ICL with humans. Importantly, our verification design contrasts with many evaluation metrics studies which align a system’s end output score with human’s judgements (Fabbri et al., 2021). Rather, we directly evaluate the questions’ correctness of the benchmark. Upon verification of the questions, subsequent steps are transparent, deterministic, and verifiable — all important features for any measure.
Verification Results. We invite human annotators to identify whether a relation r exists between events s1 and s2 in a discourse instance (Arg1, Arg2, R) – the same binary task that the ICL method tackles. They annotated 61 event relation instances across all Level-2 discourse relations, as detailed in Appendix B.4. Two NLP-specialized graduate students performed this task, each paid at the university’s standard rate of US$10 per hour for 2 hours work. To ensure clarity, instances requiring extensive domain-specific knowledge, such as finance, were excluded after random sampling. The annotation began following a basic discourse semantics tutorial.
Table 3 shows strong agreements between ICL method’s predictions and human annotators (~85%). Furthermore, despite the majority of data samples being positive cases, ICL demonstrates a decent Cohen’s Kappa score with human annotators. The ICL & human scores are even higher than the score between humans. A possible reason is that humans have a higher tendency to respond positively, increasing chance probability and decreas-
ing the Kappa score. Most importantly, the success rate – the proportion of positive cases confirmed by human annotation – exceeds 93%, validating the effectiveness of our ICL method in identifying salient event pairs for discourse understanding.

heading: 3 Discursive Socratic Questioning for Evaluation
text: Having confirmed the validity of salient events in discourse comprehension, we now consider our measure’s core as established. Now we outline our systematic approach to “how to ask”: generating questions, querying models with these questions, and subsequently computing the scores.

heading: 3.1 Question Generation
text: We generate three types of questions (Table 4): (1) Targeted Questions: Ground truth answers for Qt are always affirmative, as they tap into the salient signals. We employ a rule-based question generator (QG) to weave events into a cohesive query. As an example, for Contingency.Result, where r denotes “the reason for”, a typical question might be “Is s1 the reason for s2?” (2) Counterfactual (CF) Questions: These gauge model robustness, as their answers are negative, due to the event relation being altered into a counterfactual r′. For instance, “Is s1 contrasted against s2?” is unrelated to contingency discourse. (3) Converse Questions: These test a model’s response consistency to the logically-equivalent converse question. For example, “Is s2 the result of s1?” (←−r ) corresponds to the earlier question about reason (r). We anticipate consistent responses from LMs. For bi-directional questions, only the entity order is reversed (e.g., “Does A happen at the same time as B?” becomes “Does B happen at the same time as A?”). For uni-directional questions, both relation and entity order are inverted (e.g., “Is A the reason for B?” to “Is B the result of A?”), detailed in Appendix C.9.

heading: 3.2 Question Answering
text: Algorithm 1 DISQ interrogates a language model. 1: Input: Discourse d and its corresponding questionsQ. 2: H = {∅} ▷ The history is initialized. 3: Stage 1: Targeted and Counterfactual QA 4: for qi inQt andQc do 5: ai = LM(q = qi, c = d) ▷ The model performs
QA. The context c is the discourse d. 6: H ← (qi, ai) ▷ The history is updated. 7: end for 8: Stage 2: Converse QA 9: for (qi, ai) inH do 10: q̃ = Lookup(q, {Q̃c, Q̃t)} ▷ Look up the converse question in converse question sets. 11: ãi = LM(q = q̃i, c = d, (qi, ai) ∈ H) ▷ The model executes QA on the converse question, q̃i, optionally utilizing the previous response (qi, ai) as supplemental context. 12: H ← (q̃i, ãi) ▷ The history is updated. 13: end for 14: Output: H
Questioning is divided into two stages (Algorithm 1). In the first stage, DISQ interrogates the model with targeted questions Qt (expecting a positive answer) and counterfactual questions Qc (expecting negative). These questions focus on events s1 and s2, and reference the discourse context d, specifically Arg1 and Arg2. Note that we do not inform the model of the discourse relation; the model must infer the discourse relation to answer correctly. The answer to each question updates the history H. In the second stage, DISQ performs converse QA to test the model’s consistency. For each converse q̃i, we look for model’s response to the original question and can choose to reuse it to promote the consistency (we find that this choice affects performance significantly; see §4.4).
It is worth noting that DISQ operates at the finest-grained level possible in the PDTB taxonomy. If a relation cannot be further divided (e.g., Comparison.Contrast), we consider it as a Level-2 relation. However, if a Level-3 distinction is possible (e.g., Comparison.Concession.Arg1-as-denier or Arg2-as-denier), DISQ operates at Level-3. For consistency, we report the results for Level-2, while the Level-3 results are presented in Appendix C.6.

heading: 3.3 DISQ Score
text: We gauge a model’s overall proficiency by its DISQ Score 2, which combines three scores, as coefficients of a product: sdisq = st × scf × scon.
2Named after our method DISQ, this term is also used to denote our measurement score when unambiguous.
DISQ then comprises of (1) Target Score (st); (2) Counterfactual Score (scf ) — assessing the accuracy of the model’s answers to targeted and counterfactual questions; and (3) Consistency Score (scs) — evaluating the model’s consistent responses to a question and its converse. Concretely, for N questions asked:
st = 1
N
N∑
i=1
1[ai = True], qi ∈ {Qt, Q̃t} (1)
scf = 1
N
N∑
i=1
1[ai = False], qi ∈ {Qc, Q̃c} (2)
scon = 1
N
N∑
i=1
1[ai = ãi], qi ∈ Q, q̃i ∈ Q̃ (3)
DISQ uses a product since we favor balanced individuals scores, which may not occur with a sum aggregate — c.f., (0.6, 0.6, 0.6) vs. (0.9, 0.9, 0).

heading: 4 Evaluations
text: We guide our evaluation with following research questions (RQs): RQ1: How do models perform on DISQ’s three scores overall? RQ2: Are DISQ’s scores consistent in different datasets and variations in question phrasing? RQ3: What impact do different discourse relations have on model performance? RQ4: What linguistic structures can help models improve their performance on DISQ?
Datasets: Besides PDTB, we also use English sections of the TED-Multilingual Discourse Bank (TED-MDB) dataset, with PDTB-style annotations from TED talks. After preprocessing it in a manner similar to our treatment of the PDTB, we obtain a question bank consisting of 448 discourse instances and 8,376 questions (Appendix C.1).
Models: While any language model can be assessed, our evaluation targets two of the current strongest large language models (LLMs): (1) Closed-Sourced Models: GPT-4 and GPT-3.5turbo. To manage costs, we limit evaluations to 20% of our test samples (Appendix C.5). Despite this constraint, we noted stable performance throughout our experiment and relation distributions similar to those of the entire dataset. (2) Open-Sourced Models: LLaMA-2 (Touvron et al., 2023) and its variations, Vicuna (Chiang et al., 2023) and Wizard (Xu et al., 2023). Vicuna is a
distilled model that refines LLaMA-2 using user interactions with GPT models; and Wizard is specifically designed for complex code and math instruction adherence, which we felt might help with discourse explanation. We use 13B sized models for all of these but also investigate a smaller 7B LlaMA-2 model for scaling effects.
Implementation Details: We use a zero-shot approach in our evaluations to mirror real-world conditions. Recognizing that smaller models may falter with different instruction templates, we experimented with various templates and report the optimal performances. This method allows us to focus on evaluating the models’ ability to answer questions about discourse relation, minimizing the influence of their instruction comprehension skills. Respond to a true-or-false question derived from a two-sentence discourse, comprising Sentence 1 (Sent1) and Sentence 2 (Sent2), linked by a relationship type like causal, temporal, expansion, contrasting, etc. The question targets two events within this discourse, and your task is to evaluate if these events exhibit the specified relationship. Answer with ’True’ or ’False’ based on your analysis.
Sent1: “When I want to buy, they run from you – they keep changing their prices.” Sent2: “It’s very frustrating.”
Question: Is “It’s very frustrating. (event 2)” the result of “hey keep changing their prices (event 1)”? True or False? Answer:
Following Zhao et al. (2021), we prompt models with concise instructions, as shown in the example above, followed by the given discourse context and
questions. We determine predictions based on the probability of the first token (see Appendix C.4).

heading: 4.1 Overall Performance (RQ1)
text: Figure 3 displays the overall performance of various models in a zero-shot setting, leading to the following observations: (A) GPT Models Show Room for Improvement (1st row): None of the models achieve an ideal score (all three scores at 1.0), indicating room for growth. GPT-3.5 underperforms GPT-4 significantly in both the PDTB and TED-MDB datasets (almost half in sdisq), particularly in the Targeted Score, which shows its limitation in understanding discourse. (B) Significant Improvements with LLaMA Enhancements (2nd and 3rd row): Initial tests show Vanilla LLaMA-2 models (7B and 13B) perform below the random baseline. However, significant improvements are noted with their Chat variants, and further enhancements are observed with Vicuna-13B after tuning on user interaction, and upon further tuning for Code and Math based on Wizard models. Vicuna-13B notably surpasses GPT-3.5 in both datasets, suggesting open-source models can rival GPT-3.5 in discourse understanding. Our discovery of the chat variant’s benefit is corroborated by a recent study (Sravanthi et al., 2024), which finds that LLaMA’s chat variants perform better in pragmatic understanding tasks (Appendix C.11). (C) Consistent Performance Across Datasets: The scores for both datasets align well, visualized as blue and green shapes in the radar charts in Figure 3, demonstrating DISQ is consistent in differing domains.
4.2 Consistency of DISQ Scores (RQ2)
While evaluating DISQ intrinsically through human assessment of question correctness, we also present extrinsic evidence of DISQ Scores’ robustness across domain and paraphrase variations. The first plot in Figure 4 shows DISQ Scores for various open-source models (abbreviated by initial letters) across two datasets, with model rankings demonstrating strong consistency, evidenced by a Kendall’s Tau correlation of 0.857.
The second plot, focusing on the PDTB dataset, illustrates models’ resilience to question paraphrasing, involving synonym replacement and syntactic changes, and results in two paraphrase sets. This analysis shows the DISQ Score remains stable under these variations, with a mean Spearman correlation of 93.6 across three pairs. For the TED dataset, consistency is also observed (Appendix C.8).

heading: 4.3 DISQ Scores by Discourse Relations (RQ3)
text: Table 5 provides performance per discourse relation. These results yield several intriguing insights: (1) Persistent Challenge of Minority Classes for LLMs: Historically, minority classes have posed difficulties for supervised methods (Kim et al., 2020), and this trend continues with LLMs. Classes like Comp.Concession, Exp.Equivalence, and Temp.Synchronous remain challenging, evidenced by DISQ score at or below 0.3 for most models, suggesting that merely increasing model and data samples do not yield comprehensive discourse understanding. (2) Open-sourced Models Rival GPT: The granular analysis of scores reveals that open-source models are capable of matching, and in some instances, surpassing the performance of the esteemed GPT models. For instance, within Contingency relations, models like Vicuna-13B (6A, 6B in Table 5) and WizardCode (8A, 8B) as well as Wizard-Math (9A, 9B) excel, even outperforming GPT models. It underscores the potential of LLaMA-based specialized training as a promising method to enhance discourse comprehension. (3) Task Difficulty Asymmetry: An intriguing pattern is Contingency.Reason consistently outscoring Contingency.Result across all models, despite both addressing causality. Similar trends are noted in other fine-grained relations like Temp.Async.Precedence and Temp.Async.Succession (Appendix C.6), indicating a potential asymmetry in semantic processing by language models.

heading: 4.4 Linguistic Features (RQ4)
text: We are driven to find what linguistic features can improve the faithfulness of discourse understanding, especially for open-source models. We examine the following features:
Feature 1: Is the Presence of Discourse Connectives Beneficial? Discourse connectives transform implicit discourse into explicit forms, enhancing comprehension (Kurfalı and Östling, 2021). Does this also extend to LLMs? In short, yes.
We selected LLaMA2-13B-Chat, Vicuna-13B and Wizard-Code as representative open-source models. We then inserted PDTB connectives at the start of Arg2, and conducted DISQ. While these experiments are replicated with the one optimal task template from previous experiments, we also trial with several random seeds and find that the results remain consistent. Table 6 reveals that connectives benefit all models, increasing DISQ by 8% to 22%. For example, it boosts Vicuna’s overall performance from 0.325 to 0.396, closely approaching GPT-4’s score of 0.414 (which lack connectives). This suggests that correctly inferring connectives significantly enhances the accuracy of LMs’ discourse comprehension. Our results corroborate the findings in (Liu and Strube, 2023), which jointly predict discourse connectives and discourse relations, highlighting the benefits of exploiting discourse connectives.
Feature 2: Does Context Enhance Comprehension? Next, we assess the influence of surrounding context on discourse comprehension. Does it enhance LLM’s faithfulness? In short, yes.
In the PDTB corpus, texts are segmented into paragraphs. Accordingly, we provide the mod-
els with the local paragraph surrounding the discourse arguments. Table 7 demonstrates that models exhibit overall performance enhancements when contextual information is integrated. For instance, LLaMA’s overall performance improves from 0.253 to 0.311. We find the improved performance is primarily due to a significant rise in Targeted Score with minimal changes in Counterfactual Score, as detailed in Appendix‘C.7. This indicates that models particularly benefit from extra context when positively responding to targeted questions.
Feature 3: Is QA History Beneficial for Consistency? Open-source LMs typically achieve an 80% Consistency Score without accessing their own QA history. We explore whether models exhibit greater consistency when referencing their previous QA interactions. In this process, while posing converse questions, we include the history of corresponding targeted and counterfactual questions, along with the model’s responses, in the input. The ideal outcome is for the model to make consistent predictions.
Table 8 presents mixed outcomes: Vicuna-13B and Wizard-Code exhibit significant improvements, whereas LLaMA2-13B-Chat experiences a reduction in consistency. Further analysis into LLaMA’s Consistency Scores by question type reveals lower scores for uni-directional questions (e.g., “happen before”) and higher for bi-directional (e.g., “happen at the same time as”). For uni-directional questions, the converse questions are different from the original (e.g., “happen before” becomes “happen after”). However, for bi-directional questions, the form remains unchanged (Appendix C.9). This pattern suggests that LLaMA might focus mainly on literal keywords, lacking in deeper reasoning abilities, while Wizard-Code’s code-based training appears to have bolstered its logical reasoning (detailed in Appendix C.10).

heading: 5 Related Work
text: Evaluation Methods in NLP: Recent approaches for evaluating and interpreting LMs in-
clude: (1) The probing paradigm takes out the representation of LMs and train a model to predict whether one linguistic property is captured by the representation (Tenney et al., 2019; Wallace et al., 2019; Li et al., 2021). (2) Behavior analysis and post-hoc interpretation produce fine-grained interpretation of model’s output. The common practice is to perturb the text to reveal the decision boundary or unwanted bias of the model (Belinkov et al., 2020; Ribeiro et al., 2016; Poliak et al., 2018; Rudinger et al., 2018). But the creation of the perturbation usually requires manual efforts. (3) QA-based Evaluation offers a transparent and granular approach (Hu et al., 2023; Fabbri et al., 2022a), yet its application in evaluating discourse faithfulness remains unexplored. There are several efforts re-formalizing discourse parsing as QA, including QADiscourse (Pyatkin et al., 2020), QA for reference/ellipsis resolution (Hou, 2020; Aralikatte et al., 2021), and Question Under Discussion (QUD) framework (Ko et al., 2022; Wu et al., 2023). However, their focus is on parsing rather than utilizing QA for evaluating faithfulness.
Discourse Modeling and Evaluation: (1) Discourse Modeling: Language Models (LMs) serve as the core for custom neural networks to predict discourse relations (Liu et al., 2021; Jiang et al., 2021; Zhou et al., 2022; Xiang et al., 2022; Chan et al., 2023; Wang et al., 2023). These approaches show improvements over traditional feature-based methods (Pitler et al., 2009; Rutherford and Xue, 2014) but lack in interpretability. Additionally, LMs are applied in coherence modeling (Joty et al., 2018; Jwalapuram et al., 2022) and hierarchical discourse parsing (Huber and Carenini, 2022; Ko et al., 2023), yet they often overlook robustness evaluation, a key contribution of DISQ. (2) Discourse Evaluations: Recent benchmarks have moved beyond traditional treebanks like PDTB (Webber et al., 2019). DiscoEval by Chen et al. (2019) assesses sentence embeddings across various discourse tasks. Wu et al. (2023) evaluate QUD parsers, and Chan et al. (2024) analyze ChatGPT’s capabilities in diverse discourse tasks. However, none of these studies focus on the faithfulness aspect of LMs.

heading: 6 Conclusion and Future Work
text: In this paper, we contribute Discursive Socratic Questioning (DISQ), the first systematic evaluation for faithful discourse comprehension. To ensure
the reliability of DISQ’s assessment, we employ both intrinsic verification via human annotation and extrinsic evaluation to demonstrate its resilience against domain shifts and paraphrase variations. Our extensive experiments reveal that even leading models like GPT-4 have their shortcomings in DISQ, and that open-source models — despite trailing GPT-4 — can close this gap with fine-tuning on chat and code/math data. To advance LLMs’ understanding, we suggest incorporating linguistic features such as discourse connectives, contextual information, and historical QA data. In the future, we aim to extend our analysis to longer-range discourse, incorporate additional discourse annotation frameworks beyond PDTB, and distilling knowledge from larger models to benefit smaller models.

heading: Acknowledgements
text: We thank several group members from the Web Information Retrieval / Natural Language Processing Group (WING) at NUS for their research discussions and proofreading of our drafts, especially Xiao Xu, Vicor Li, Taha Aksu, Tongyao Zhu, Guanzhen Li, Zekai Li, and Yanxia Qin.
We also thank our anonymous reviewers for their time spent on our review and their detailed and insightful feedback, which greatly helped us refine our work.

heading: Limitations and Ethical Considerations
text: Our human verification annotation tasks were approved by our institutional review board (IRB). IRB reviewed our experimental design and research procedures to ensure that the research involves no more than minimal risks to the research participants. In particular, we ensure that none of the phrases involve sensitive topics or would not elicit strong negative responses. We also ensure that research participants’ privacy and the confidentiality of their research data will be protected.
When performing DISQ, we note that output answers may be offensive in certain contexts, because the model can respond True/False to any question. This is a common concern for all LMs to overcome, not specific to DISQ. But according to our pilot study, we have not found any cases of such offensive Q&A pairs.
DISQ also has particular limitations. (1) We only use the behavior of the model given a set of questions as a proxy for understanding. It is not a causal analysis. We may causally study the role
of individual neuron or subnetwork for discourse function in the future, similar to a recent study about individual neuron’s role for factual knowledge (Meng et al., 2022; Liu et al., 2024). (2) We have only studied standard English corpora. It is meaningful to apply DISQ to LMs’ understanding of discourse on other English corpora with language variations and to corpora in other languages. (3) Our study primarily utilizes PDTB-style annotations, yet adapting DISQ to other discourse frameworks is also feasible. To the best of our knowledge, PDTB and TED-MDB are the only two compatible corpora in English, since other datasets like GUM, adhere to the RST framework, and a suitable Twitter-based PDTB corpus is not openly accessible. Consequently, we chose TED-MDB as our supplementary dataset due to its compatibility. Our study examined written text in the genres of news articles and public speeches, so may not generalise beyond these domain. However, we believe it is possible to extend DISQ’s analysis to a broader range of genres in future work.

heading: A Details for Question Bank Preparation
text: In Section 2.2, we briefly introduced our InContext Learning (ICL) approach; here, we offer a more detailed explanation. Figure 5 elucidates the ICL template. The ‘DR summary’ section encapsulates the discourse using the model’s specific terminology. In ‘Event1 comprehension’, Event1 is linked with Arg2, exploring its role in discerning the discourse relation. A similar analysis is conducted for Event2 in ‘Event2 comprehension’. This stage prompts LMs to begin reasoning, as demonstrated in Example 2, where the model identifies the actual object of denial, such as ‘shipping the card’. The sections ‘What if ER holds’ and ‘What if ER does not hold’ present hypothetical scenarios of ER’s presence or absence, exploring their implications for the given DR. The model is encouraged to offer explanations (like “It suggests to the audience that IBM’s actions are inconsistent and perhaps not well-planned”). ‘Predicting ER’ synthesizes the preceding rationale to predict an ER between Event1 and Event2, leading to the ‘Final prediction’ that provides the definitive conclusion. While our focus is on using ICL to identify salient signals in discourse understanding, we have not fully explored the potential of prompt engineering. Concretely, for 11 Level-2 discourse relations, we create 22 examples in total.

heading: B Annotation Details for Human
text: Verification

heading: B.1 Annotator Recruitment
text: Following Institutional Review Board (IRB) approval, we enlisted two graduate students specializing in Natural Language Processing (NLP) to conduct our annotations. These individuals possess strong English proficiency and academic expertise, equipping them with the necessary skills to comprehend our discourse task effectively. They have consented to the use of their anonymized data, with
the assurance that their identities will remain confidential.

heading: B.2 Annotator Training
text: Discourse and events
Positive case: { "Discourse relation": "Expansion.Level-of-detail.Arg2-as-detail, which means Arg2 provides more detail about Arg1", "Arg1": "An international group approved a formal ban on ivory trade despite objections from southern African governments, which threatened to find alternative channels for selling elephant tusks", "Arg2": "The move by the Convention on Trade in Endangered Species, meeting in Switzerland, places the elephant on the endangered-species list", "Event1": "An international group approved a formal ban on ivory trade", "Event2": "The move places the elephant on the endangered - species list", "Event relation": "Event2 provides more detail about Event1" }
(a) Training slides screenshot: Positive sample (negative sample omitted).
An author led the training for the two annotators, starting with a tutorial on discourse semantics by referring to PDTB annotation guideline (Webber et al., 2019). This session then equipped annotators with the skills to discern positive and negative Event Relation (ER) cases (Figure 6). Our task instructions were straightforward, focusing on binary classification.

heading: B.3 Annotation Interface
text: Our annotation interface: (Figure 7) adopts a minimalist style, straightforwardly prompting the annotators to determine whether an Event Relation (ER) is present in the given scenario.
Annotation Execution: On average, the annotators spent approximately one minute to answer each question. The 61 questions were divided into two sessions, with each session lasting about 40 minutes.

heading: B.4 Samples for annotation
text: We selected these samples randomly, but excluded any that necessitate extensive domain knowledge.
We summarize the distribution of the number of questions (i.e., event pairs for annotators to determine) in Table 10. Samples for one discourse relation originate from one or two instances in PDTB. The presence of more pairs in some samples is
attributable to the extended context in those cases.

heading: C Experiment Details

heading: C.1 Detailed Dataset Statistics
text: In Table 11, we provide detailed statistics of our dataset, which encompasses 11 Level-2 discourse relations. For PDTB, we use the new sense taxonomy from PDTB 3.0 (Webber et al., 2019), but we only retain instances with provenance in PDTB 2.0 (Prasad et al., 2008). This approach is taken because most studies focus on PDTB 2.0, and we want our scores to provide a more relevant reference. Most of the new instances from PDTB 3.0 are intra-sentence short arguments, which can be adapted using our method. It would be interesting to compare their performance with existing instances, which are mostly inter-sentence or interclause. The TED-MDB corpus only has 448 instance, which is smaller than PDTB. Due to its small size, we remove the discourse connectives in explicit discourse instances to augment the data for implicit discourse instances. It contributes 8,376 questions, aiding our examination of the crossdomain robustness of DISQ scores.
There are around 2% of corner cases where ICL methods fail to deliver any salient event pair prediction (as a positive prediction) in a discourse instance. Therefore, as an approximation, we consider all event pairs as valid to represent such instances. We find that the final DISQ score changes only slightly when these corner cases are ablated, and the rankings of models remain unchanged.

heading: C.2 Model Details
text: We list the models being evaluated in Table 12, using APIs and weights hosted on Huggingface. We also use AllenNLP (Gardner et al., 2018) for semantic role labeling toolkit.

heading: C.3 Computing Resource and AI Tools
text: We use one NVIDIA A40 GPU to perform our experiment. For in-context learning for ER prediction, it takes around 30 seconds for each instance due to long reasoning to be decoded. It takes less than one day to finish all predictions. For the evaluating models against DISQ, since it only needs to decode a short answer, it takes around 0.1 seconds for one instance. It takes around 2-3 hours to finish evaluation of one model against DISQ.
We employ GitHub Copilot as a coding assistant, primarily to complete specific lines of code once
the core functions are established. Additionally, we use GPT for grammar checking, but all the writing is conducted independently by us.

heading: C.4 Task template
text: We adopt the approach outlined by Zhao et al. (2021), employing a straightforward instruction template in Table 13. Initially, a succinct instruction is provided, followed by the context information for the model. Recognizing that the model may not be well-versed in discourse semantics, we use the term “sentence” in place of “argument” for clarity. Subsequent to the question, we include an “Answer: ” prompt, guiding the model to respond with either “True” or “False” tokens. During evaluation, we consolidate the probabilities for the “True” token (covering variations like “True”, “true”, “TRUE”, etc.), and similarly for the “False” token.
We tested three template variations and report each model’s best outcomes: (1) removing the “True or False” phrase, (2) inserting a line break at the end, and (3) placing a line break between the “question” and “answer”.

heading: C.5 Samples for GPT Experiments
text: Table 14 displays the distribution of discourse relations in both the PDTB and TED-MDB datasets for GPT evaluation. We selected the first 200 samples from PDTB and randomly chose 100 samples from TED-MDB to ensure their relation distributions align closely with each dataset. This selection process was designed to match the overall distribution without needing random sampling for PDTB.
The analysis reveals that PDTB features a higher prevalence of causal discourse, whereas TEDMDB exhibits a greater number of expansions, reflecting the distinctive nature of TED Talks. This difference highlights the unique characteristics of each dataset.

heading: C.6 Level-3 Discourse Relations
text: It is important to note that DISQ functions at the most detailed level within the PDTB taxonomy. When a relation cannot be further subdivided (e.g., Comparison.Contrast), we treat it as a Level-2 relation. However, if a Level-3 distinction is available (e.g., Comparison.Concession.Arg1-as-denier or Arg2-as-denier), DISQ operates at Level-3.
Table 15 presents the DISQ scores for all Level-3 discourse relations in PDTB, omitting rare classes not present in our test set (e.g.,

heading: Discourse relation PDTB TED-MDB
text: Exp.Subst.Arg1-as-instance and Exp.Subst.Arg1as-subst). We observe a notable performance gap between converse relation pairs. For instance, Contingency.Reason consistently performs worse than Contingency.Result across all models. Similar disparities are found in other converse relation pairs, suggesting a potential intrinsic asymmetry in Large Language Models’ (LLMs) processing of semantic relationships.

heading: C.7 Contextual Results
text: To explore the effect of surrounding context on the comprehension of discourse arguments, we decompose the DISQ Score into Targeted and Counterfactual categories, as detailed in Table 16. Our analysis reveals that the overall enhancement in the DISQ Score is predominantly due to the elevation in Targeted Score. For the majority of relations, we observe a pronounced increase in Targeted Score, contrasted with a decrease or slight rise in Counterfactual Score. This indicates that context primarily benefits affirmatively answering Targeted questions.

heading: C.8 Paraphrasing Performance on TED-MDB
text: In the main paper, we focused solely on paraphrasing within the PDTB dataset. We now extend our reporting to include model performance on the TED dataset regarding paraphrasing variability.
Figure 8 demonstrates a high degree of correlation among the three sets, with a mean Spearman correlation of 94.4 across the three pairs.
To build upon our original questions, which detail the event relations in Table 11, we introduce two sets of paraphrases:
Paraphrase set 1: ‘is the consequence of’, ‘is the cause of’, ‘does occurs simultaneously as’, ‘does occurs before’, ‘does occurs after’ ‘is opposed to’, ‘is negated by’, ‘negates’ ‘serves as a substitute for’, ‘is being provided an substitute by’ ‘provide additional information about’, ‘is being provided additional information by’, ‘is equal to’, ‘are contributed to the same circumstance’ ‘is an instance of’, ‘is being instantiated by’
Paraphrase set 2: ‘is due to’, ‘leads to’, ‘takes place simultaneously as’, ‘does takes place before’, ‘does takes place after’, ‘is contrary to’, ‘is refuted by’, ‘refutes’, ‘acts as a replacement for’, ‘is replaced by’ ‘present more specifics on’, ‘is presented with more specifics by’, ‘is on par with’, ‘are contributed to the same scenario’, ‘serves as an example of’, ‘is exemplified by’.

heading: C.9 Converse questions
text: A key aspect of DISQ involves the inclusion of converse questions. Table 17 outlines the discourse relations, original questions, and their converse counterparts. In bi-directional questions, we reverse only the order of entities (e.g., from “Does A happen at the same time as B?” to “Does B happen at the same time as A?”). For uni-directional questions, we invert both the relation and the order of entities (e.g., changing “Is A the reason for B?” to “Is B the result of A?”).

heading: C.10 Details for Experiments Using Historical
text: QA
Table 18 details consistency scores for each question type, revealing that bi-directional questions generally achieve higher Consistency Scores. For instance, “happen at the same time as” scores an impressive 98.6, while uni-directional questions, such as “happen before”, score merely 26.5. Figure 9 offers a clear visual comparison, showing an average consistency score of 79.2 for bi-directional relations versus 60.6 for uni-directional ones.
This trend indicates that LLaMA-13B-Chat may predominantly rely on literal keyword matching, possibly at the expense of deeper reasoning capabilities in question answering. Conversely, the
code-based training of Wizard-Code seems to enhance its logical reasoning, leading to better overall performance.

heading: C.11 Pragmatic Understanding Tasks Corroborates Our Findings
text: Evaluation on DISQ reveals that LLaMA models benefit from their chat-enhanced variants. This finding is corroborated by a recent study (Sravanthi et al., 2024), which evaluates language models in pragmatics understanding tasks. The study finds that chat variants perform better in the zero-shot setting. These pragmatic understanding tasks include Direct/Indirect Response Classification, Implicature NLI, and Reference via Metonymy, among others, which share a similar formalization to cointerpreting several linguistic units and inferring implicatures, as we do in discourse understanding.

heading: Future Work
text: We may causally study the role
of individual neuron or subnetwork for discourse function in the future, similar to a recent study about individual neuron’s role for factual knowledge (Meng et al. , 2022; Liu et al. , 2024). (2) We have only studied standard English corpora. It is meaningful to apply DISQ to LMs’ understanding of discourse on other English corpora with language variations and to corpora in other languages. (3) Our study primarily utilizes PDTB-style annotations, yet adapting DISQ to other discourse frameworks is also feasible. To the best of our knowledge, PDTB and TED-MDB are the only two compatible corpora in English, since other datasets like GUM, adhere to the RST framework, and a suitable Twitter-based PDTB corpus is not openly accessible. Consequently, we chose TED-MDB as our supplementary dataset due to its compatibility. Our study examined written text in the genres of news articles and public speeches, so may not generalise beyond these domain. However, we believe it is possible to extend DISQ’s analysis to a broader range of genres in future work