Structured Discourse Representation for Factual Consistency Verification
Authors: Kun Zhang, Oana Balalau, Ioana Manolescu
Accepted at: acl25

Structured Discourse Representation for Factual Consistency Verification

Kun Zhang1,2 and Oana Balalau1,2 and Ioana Manolescu1,2
1Inria, 2Institut Polytechnique de Paris

Abstract
Analysing the differences in how events are represented across texts, or verifying whether the language model generations hallucinate, requires the ability to systematically compare their content. To support such comparison, structured representation that captures fine-grained information plays a vital role. In particular, identifying distinct atomic facts and the discourse relations connecting them enables deeper semantic comparison. Our proposed approach combines structured discourse information extraction with a classifier, FDSpotter, for factual consistency verification. We show that adversarial discourse relations pose challenges for language models, but fine-tuning on our annotated data, DiscInfer, achieves competitive performance. Our proposed approach advances factual consistency verification by grounding in linguistic structure and decomposing it into interpretable components. We demonstrate the effectiveness of our method on the evaluation of two tasks: data-to-text generation and text summarisation. Our code and dataset will be publicly available on GitHub.

1 Introduction
The analysis of public discourse plays a vital role in social sciences, in particular for sociology (Wodak and Meyer, 2015), history (Rule et al., 2015), and computational journalism (Cazalens et al., 2018). A key task in this context is to quantify whether two texts convey the same information through their expressed content. Such comparisons enable the detection of media bias: for instance, the outlets with different political orientations usually tend to selectively report different subsets of information units, which has been a phenomenon known as omission bias (Baker et al., 1994). More recently, similar techniques have been utilised to assess the factual consistency of text generation, verifying whether the content in the output can be grounded in the reference or source text (Tang et al., 2024).

Extracting RDF-style triples from two texts and then assessing their overlap appears to be a straightforward strategy for comparing texts. However, prior works that extract structured contents for factual consistency verification (Joty et al., 2017; Goodrich et al., 2019; Goyal and Durrett, 2020) exhibit a performance gap compared with state-of-the-art approaches based on sentence-level entailment (Scirè et al., 2024) or large language model prompting (Luo et al., 2023). The gap is primarily due to natural language expresses meaning in more rich and subtle ways than RDF triples. Firstly, the contents in adverbials and complements are difficult to encode in the standard subject–predicate–object format. For example, in the text “Barack Obama was elected in 2008 as the President of the USA”, “in 2008” conveys crucial temporal information, yet it is hard to incorporate into a triple structure. Secondly, triple-based representations fail to encode discourse-level semantics, such as contrastive or causal relations. For instance, the causal link in “Lu Xun realised that being a doctor could not save Chinese people’s minds, so he gave up his medical career and became a writer” cannot be captured through RDF triples. Beyond the limitations of the triple format, both large language models (Dubey et al., 2024) and smaller classifiers (He et al., 2021) often fail to detect contradictions introduced by adversarial discourse connectives (Miao et al., 2024).

Our contributions in the paper are as follows:

• We propose a structured representation of textual information that extends RDF-style triples by incorporating complements and adverbials into the atomic facts, and by preserving the discourse relations between them. We show that few-shot LLMs can extract this structure with high quality.

• We introduce a novel method for evaluating cross-textual factual consistency. A classifier,
FDSpotter, is trained to verify the presence of atomic facts and discourse relations across texts. We show that detecting whether discourse relations are preserved is challenging, but we achieve strong performance after fine-tuning on our annotated DiscInfer dataset.

• Empirical results demonstrate the effectiveness of our approach in evaluating data-to-text generation and text summarisation.

2 Related Work

Open Information Extraction (OIE) Open information extraction (Etzioni et al., 2008; Upadhyay et al., 2023; Pai et al., 2024) tools extract triples of the form (subject, predicate, object) from unstructured texts. In OIE, the predicates are not known beforehand, and the subjects and objects are not necessary to be named entities. While the triple format is one of the most common used, n-tuples have also been proposed (Niklaus et al., 2018). In some approaches (Del Corro and Gemulla, 2013), the elements of the tuples are defined as syntax elements, such such as subject, verb, object, complement, etc. In other works (Dong et al., 2023) the elements of the tuples are phrases. Open information extraction can be performed by semantic role labelling (Chen et al., 2025a), sequence tagging (Stanovsky et al., 2018), and large language models (Xu et al., 2023).

Meaning Representation and Parsing Abstract Meaning Representation (AMR) (Banarescu et al., 2013) is a semantic representation framework designed to abstract away from surface syntax and encode the meaning of the text in the form of rooted, directed, acyclic graphs (DAGs) that capture the relation between the elements of the text. One key challenge of AMR parsing lies in the annotation process, which requires high linguistic expertise. A well-trained annotator typically spends around 10 minutes to produce a single AMR graph (Sadedine et al., 2024). AMR parsers also struggles with out-of-distribution inputs and new domains (Lee et al., 2022). Discourse parsing (Braud et al., 2023) identifies the structure of texts by linking spans of discourses through semantic and pragmatic discourse relations. can be explicitly or implicitly expressed. State-of-the-art parsing methods achieve strong performance in discourse unit segmentation (Metheniti et al., 2023), but Liu et al. (2023a) reports discourse connective detection remains challenging. Shallow discourse parsing identifies discourse argument spans and the corresponding discourse connectives jointly (Wang and Lan, 2015; Oepen et al., 2016), but state-of-the-art models on the task still exhibit room for improvement (Xue et al., 2015, 2016; Knaebel, 2021).

Faithfulness Evaluation Metrics. Metrics for comparing two texts have been used to evaluate text generation tasks, such as summarisation, data-to-text generation, and machine translation. Well known metrics include BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), BERTScore (Zhang et al., 2019), MoverScore (Zhao et al., 2019), and BLEURT (Sellam et al., 2020). Overlap measures based on \( n \)-grams, such as BLEU, ROUGE, and METEOR, have been widely used in the literature, however word embeddings metrics, such as BERTScore, BLEURT, MoverScore are becoming standard, as they correlate better with human evaluation than the surface based metrics. Recently, explainable metrics have been developed, for example, FActScore (Min et al., 2023), FactSpotter (Zhang et al., 2023), and FENICE (Scirè et al., 2024), which allow a more fine-grained understanding of differences between texts. FactSpotter propose a metric for factual faithfulness evaluation on data-to-text generation, which gives a score for how well each triple from the structured data is represented in the textual generation. Similarly, FActScore (Min et al., 2023) metric allows detecting hallucinations in text generation. This metric has two building blocks: decomposing the first text into atomic facts, and testing if each atomic fact is present in the second text. In Nawrath et al. (2024), the authors propose splitting the text into sub-sentences using AMR, however, the authors do not deal with maintaining the connection between these sub-sentences. A recent approach that achieves state-of-art results on measuring factual faithfulness in textual summarization FENICE (Scirè et al., 2024) proposes to split a text in atomic facts and then use natural language inference to decide if the facts are expressed in a new text. LLMs, such as ChatGPT, have been also proposed as factually evaluators in zero shot or chain-of-thought setting (Luo et al., 2023). By the time of paper submission, we saw no metric that account explicitly for the discourse relation. However, by the time of paper acceptance, we have seen a concurrent work on discourse-level evaluation for summarisation (Zhong and Litman, 2025)
3 Problem statement

To evaluate the quality of the text generation task, a common method is to calculate the similarity between the generated text and the ground-truth text. Taking graph-to-text generation and text summary as examples, we aim to quantify the similarity of two texts from two perspectives:

1. Do the two texts state the same atomic facts?

2. Are the discourse relations between the atomic facts the same in both texts?

To achieve this goal, we firstly focus on how to represent the atomic facts and the discourse relations between them. Inspired by ClausIE (Del Corro and Gemulla, 2013) and previous research in discourse relation recognition (Prasad et al., 2019), we propose a format for representing text information in Section 4. Then we introduce our method for estimating whether an atomic fact or a discourse relation is preserved in two texts in Section 5 with the model trained in Section 6.

4 Structured Information Representation

Given a text T, our goal is to extract the atomic facts \( \mathbf{A} = \{A_1, ..., A_m\} \) and discourse relations between the atomic facts \( \mathbf{D} = \{D_1, ..., D_p\} \). The combination of the two structured representations captures rich information in the text, as an example illustrated in Table 1.

4.1 Atomic Fact Representation

A clause is the smallest grammatical unit in English for expressing a complete proposition¹. We consider an atomic fact to be a clause, structured as a tuple: \( \langle \text{subject}, \text{predicate}, \text{direct object}, \text{indirect object}, \text{short adverbial}, \text{short complement} \rangle \). Note that simpler triples of the form \( \langle \text{subject}, \text{predicate}, \text{object} \rangle \) have been found insufficient to represent varied atomic facts (Suchanek, 2020; Nawrath et al., 2024; Sadeddine et al., 2024); the short adverbial and the short complement bring important information about the current atomic fact. The short adverbial describes how, when and where about the predicate, while the short complement has extra information about the subject or the object. Each atomic fact should at least have a predicate, and usually has a subject. The object, adverbial, or complement may be absent, and we leave them blank in the extractions for such cases.

If an element of the atomic fact tuple contains another fact in its non-finite verb, relative clause, or appositive ²), we model this as an extra atomic fact. For instance, the text "Amy gave a gift to her best friend, Kate" should be represented as two atomic facts: \( \langle \text{Amy, gave, a gift, to Amy's best friend Kate} \rangle \) and \( \langle \text{kate, is the best friend of, Amy} \rangle \).

4.2 Discourse Relation Representation

Discourse relations connecting atomic facts encapsulate an important part of the text information. The discourse connectives are syntactically expressed by conjunctives, relative adverbs or other transitional phrases. The detailed examples are illustrated in the Appendix A. We model a discourse relation \( D_j \) with format \( \langle \text{fact1, connective, fact2} \rangle \).

We focus on the following groups of discourse relations, as defined in the PDTB dataset (Prasad et al., 2019), and their respective connectives:

• temporal : precedence (e.g., before, till), succession (e.g., after, subsequently), synchronous (e.g., when, at the same time);

• comparison : concession (e.g., although, even if), contrast (e.g., in contrast, however), similarity (e.g., similarly, in the same way);

• contingency : reason (e.g., due to, because), result (e.g., consequently, therefore), condition (e.g., provided that, in case), negative condition (e.g., unless).

We did not include expansion connectives: simple conjunction (e.g., and); restatement (e.g., in other words); specification (e.g., especially); instantiation (e.g., for example); generalisation (e.g., in summary). The reason is that their presence does not affect important textual information. Thus, when comparing two texts for factual consistency, we can ignore the presence of these connectives.

4.3 Implementation of Extraction

The state-of-the-art methods have achieved high performance in discourse unit segmentation

¹https://dictionary.cambridge.org/grammar/british-grammar/clauses-and-sentences

²Non-finite verbs are verbs that do not show tense, person, or number, e.g., "running" in "I saw a dog running". Relative clauses are introduced by relative pronouns to describe the antecedents (e.g., in "Paris is a city which is the capital of France", "which" introduces a relative clause). An appositive is a noun phrase (NP) after another NP to provide additional information about it (e.g., in "Paris, the capital of France, is a good city", "the capital of France" is the appositive of "Paris").
<table>
  <tr>
    <th>Subject</th>
    <th>Predicate</th>
    <th>Object Direct</th>
    <th>Object Indirect</th>
    <th>Short Adverbial and Complement</th>
  </tr>
  <tr>
    <td>Amy</td>
    <td>gave</td>
    <td>a gift</td>
    <td>to Amy’s best friend, Kate</td>
    <td>last week</td>
  </tr>
  <tr>
    <td>Amy</td>
    <td>was thankful about</td>
    <td>the exam’s results</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Kate</td>
    <td>is best friend of</td>
    <td>Amy</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Kate</td>
    <td>said</td>
    <td>Amy did not need to give a gift</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Kate</td>
    <td>helped</td>
    <td>Amy</td>
    <td>-</td>
    <td>to revise all the course materials until midnight</td>
  </tr>
  <tr>
    <td>Amy and Kate</td>
    <td>held</td>
    <td>a party</td>
    <td>-</td>
    <td>early on the next day</td>
  </tr>
  <tr>
    <td>Amy and Kate</td>
    <td>didn’t get up</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

(a) Atomic facts extracted from the given text.

<table>
  <tr>
    <th>Fact 1</th>
    <th>Connective</th>
    <th>Fact 2</th>
  </tr>
  <tr>
    <td>Amy gave a gift to Amy’s best friend, Kate last week</td>
    <td>because of even if</td>
    <td>being thankful about the exam’s results</td>
  </tr>
  <tr>
    <td>Kate said Amy did not need to give a gift to Kate</td>
    <td>if</td>
    <td>Kate helped Amy revise all the course materials</td>
  </tr>
  <tr>
    <td>Amy would have failed the exam</td>
    <td></td>
    <td>Kate were not helping</td>
  </tr>
  <tr>
    <td>Amy and Kate held a party until midnight</td>
    <td>As a result</td>
    <td>Amy and Kate didn’t get up early on the next day</td>
  </tr>
</table>

(b) Discourse relations extracted from the given text.

Table 1: Atomic facts and discourse relations representation, given the text: "Being thankful about the exam’s results, Amy gave a gift to her best friend, Kate, last week. Kate said that Amy did not need to do this, even if Kate helped Amy revise all the course materials. Amy would have failed the exam if Kate were not helping. They held a party until midnight. As a result, they didn’t get up early on the next day."

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Atomic Fact</th>
    <th colspan="3">LCS</th>
    <th colspan="3">SBERT Elem</th>
    <th colspan="3">SBERT Full</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>77.21</td>
    <td>69.17</td>
    <td>72.97</td>
    <td>81.45</td>
    <td>72.97</td>
    <td>76.98</td>
    <td>86.20</td>
    <td>77.22</td>
    <td>81.46</td>
    <td>90.15</td>
    <td>80.76</td>
    <td>85.20</td>
  </tr>
  <tr>
    <td>GPT4-Turbo</td>
    <td>68.47</td>
    <td><b>52.78</b></td>
    <td><b>59.61</b></td>
    <td>73.84</td>
    <td>56.92</td>
    <td><b>64.29</b></td>
    <td>79.36</td>
    <td>61.17</td>
    <td><b>69.09</b></td>
    <td>88.76</td>
    <td>68.42</td>
    <td><b>77.27</b></td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td><b>68.54</b></td>
    <td>49.03</td>
    <td>57.17</td>
    <td><b>75.89</b></td>
    <td>54.28</td>
    <td>63.29</td>
    <td><b>82.68</b></td>
    <td>59.14</td>
    <td>68.95</td>
    <td><b>92.61</b></td>
    <td>66.24</td>
    <td>77.24</td>
  </tr>
  <tr>
    <td>Llama3.1 8B</td>
    <td>49.54</td>
    <td>22.36</td>
    <td>30.81</td>
    <td>57.32</td>
    <td>25.87</td>
    <td>35.65</td>
    <td>68.79</td>
    <td>31.05</td>
    <td>42.79</td>
    <td>76.40</td>
    <td>34.49</td>
    <td>47.52</td>
  </tr>
  <tr>
    <td>Llama3.1 70B</td>
    <td>57.08</td>
    <td>51.53</td>
    <td>54.16</td>
    <td>63.17</td>
    <td><b>57.03</b></td>
    <td>59.94</td>
    <td>69.59</td>
    <td><b>62.83</b></td>
    <td>66.04</td>
    <td>80.49</td>
    <td><b>72.67</b></td>
    <td>76.38</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Discourse Relation</th>
    <th colspan="3">LCS</th>
    <th colspan="3">SBERT Elem</th>
    <th colspan="3">SBERT Full</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
    <th>P</th>
    <th>R</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>81.94</td>
    <td>81.94</td>
    <td>81.94</td>
    <td>84.93</td>
    <td>84.93</td>
    <td>84.93</td>
    <td>88.10</td>
    <td>88.10</td>
    <td>88.10</td>
    <td>90.32</td>
    <td>90.32</td>
    <td>90.32</td>
  </tr>
  <tr>
    <td>GPT4-Turbo</td>
    <td>75.00</td>
    <td><b>76.60</b></td>
    <td><b>75.79</b></td>
    <td>77.45</td>
    <td>79.09</td>
    <td><b>78.26</b></td>
    <td>79.33</td>
    <td>81.02</td>
    <td>80.17</td>
    <td>80.26</td>
    <td>81.97</td>
    <td>81.11</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>61.81</td>
    <td>70.63</td>
    <td>65.93</td>
    <td>71.60</td>
    <td><b>81.82</b></td>
    <td>76.37</td>
    <td>79.83</td>
    <td><b>91.23</b></td>
    <td><b>85.15</b></td>
    <td>86.06</td>
    <td><b>98.35</b></td>
    <td><b>91.80</b></td>
  </tr>
  <tr>
    <td>Llama3.1 8B</td>
    <td><b>92.93</b></td>
    <td>60.13</td>
    <td>73.02</td>
    <td><b>94.41</b></td>
    <td>61.09</td>
    <td>74.18</td>
    <td><b>95.56</b></td>
    <td>61.83</td>
    <td>75.08</td>
    <td><b>96.71</b></td>
    <td>62.58</td>
    <td>75.99</td>
  </tr>
  <tr>
    <td>Llama3.1 70B</td>
    <td>59.85</td>
    <td>64.23</td>
    <td>61.96</td>
    <td>66.57</td>
    <td>71.44</td>
    <td>68.92</td>
    <td>72.08</td>
    <td>77.36</td>
    <td>74.63</td>
    <td>75.50</td>
    <td>81.02</td>
    <td>78.16</td>
  </tr>
</table>

Table 2: Quality of extraction for different large language models.

(Metheniti et al., 2023), so combining discourse unit segmentation with semantic role labelling (Zhao and Penn, 2024; Chen et al., 2025b) is an effective approach for obtaining structured atomic facts. However, the task of discourse connective detection (Braud et al., 2023) remains challenging, and the state-of-the-art work on shallow discourse parsing (Knaebel, 2021) which jointly identifies argument spans and their corresponding discourse connectives has not been advancing recently.

Based on the status quo of the research, in this work, we use large language models with few-shot prompting for jointly extracting atomic facts and discourse relations. We also evaluate the quality of the extractions on a test set of 50 samples from the Causal News Corpus (Tan et al., 2022). Two human annotators provided the gold extractions. We call any one of the fields of an atomic fact (subject, predicate, direct object, indirect object, short adverbial and complement) or a discourse relation (fact1, connective, fact2) as an element.

Given an input text T, we have the golden atomic facts \( A^g \) and discourse relations \( D^g \). The outputs of the atomic fact and discourse relation extraction are \( A^m \) and \( D^m \). Similar to Castro Ferreira et al. (2020), we compare both atomic fact and discourse relation extractions with the ground-truth annotations. For each pair of corresponding tuples in the extraction and the ground truth, we compute several metrics to assess the extraction quality:

• Strict match: Compute whether all the tokens in each element of the extracted tuple strictly match with ground truth, e.g., whether the subjects of the atomic facts are the same.

• LCS ratio: Find the longest common subsequence (LCS) (Wagner and Fischer, 1974) between each element in the extraction and the corresponding element in the ground-truth. Compute the average (over all elements in the extracted atomic fact or discourse relation) of the LCS length divided by the length of the corresponding ground truth element.
• SBERT element: Compute the average (over all elements) of SBERT (Reimers and Gurevych, 2019) similarity for each element between the extractions and the annotations;

• SBERT full: Concatenate all the elements in each tuple for both of the extractions and the ground-truth annotations, then use SBERT to compute the similarity.

To compute the metrics, we assign each extracted atomic fact \( A_i^m \) or discourse relation \( D_j^m \), to exactly one atomic fact \( A_i^q \) or discourse relation \( D_j^q \) using Hungarian Algorithm, similarly as Yang et al. (2023). We evaluate the performance of Llama3.1 (Dubey et al., 2024), GPT4o and GPT4-Turbo (OpenAI et al., 2024) in Table 2 for both extractions. We observe that two humans achieve the highest annotation agreement. For open-source models, Llama3.1 SB is good on discourse relation extraction, but struggles on atomic fact extraction. Llama3.1 70B is competitive on both extractions. GPT4 models perform best across all metrics. We present more complete results in Appendix E.

5 Factual Inclusion and Factual Overlap

Given two texts \( \mathbf{T}^1 \) and \( \mathbf{T}^2 \), which are composed of the atomic fact sets \( \mathbf{A}^1 = \{A_1^1, \ldots, A_m^1\}, \mathbf{A}^2 = \{A_1^2, \ldots, A_n^2\} \) and discourse relation sets \( \mathbf{D}^1 = \{D_1^1, \ldots, D_o^1\}, \mathbf{D}^2 = \{D_1^2, \ldots, D_p^2\} \). \( m, n \) are the numbers of the atomic facts, and \( o, p \) are the numbers of the discourse relations in the texts. The Factual Inclusion (FI) and Factual Overlap (FO) scores between the texts are computed as follows to verify the factual consistency between the texts.

5.1 Intrinsic Confidence of Extraction

Let \( \mathbf{FD} \) be a model computing the probability of a structured atomic fact \( A_i \) or a discourse relation \( D_j \) to be expressed in the given text \( \mathbf{T} \). In practice, \( \mathbf{FD} \) is a Transformer-based classifier that estimates how well the atomic facts \( \mathbf{A} \) and discourse relations \( \mathbf{D} \) represent the information in the text, and the output is referred to as the intrinsic confidence. The intrinsic confidence \( C \) of each atomic fact \( A_i \) and discourse relation \( D_j \) for text \( \mathbf{T} \) is computed as:

\[
C(A_i|T) = \mathbf{FD}(A_i, \mathbf{T}), \\
C(D_j|T) = \mathbf{FD}(D_j, \mathbf{T}).
\]

The intrinsic confidence can serve two key purposes in dealing with the uncertainty inherent in the extraction. First, the intrinsic confidence can be used for selecting the most faithful extraction candidate on top of beam search. Second, within each extraction, the intrinsic confidence provides a principled mechanism to filter out the hallucinated atomic facts and discourse relations that are not sufficiently supported by the text.

5.2 Cross-Textual Fact Corroboration

To compute the content overlap between \( \mathbf{T}^1 \) and \( \mathbf{T}^2 \), the second step is to verify whether each atomic fact and discourse relation from one text is also expressed in the other text; We call this extrinsic confidence computation. We use the same model \( \mathbf{FD} \) as for intrinsic confidence computation, but the atomic facts and the discourse relations are extracted from another text for comparison. The extrinsic confidence of the atomic fact \( A_i^1 \) from \( \mathbf{T}^1 \) being expressed in \( \mathbf{T}^2 \) is computed as:

\[
C(A_i^1|\mathbf{T}^2) = \mathbf{FD}(A_i^1, \mathbf{T}^2),
\]

and similarly, the extrinsic confidence of the discourse relation \( D_j^1 \) from \( \mathbf{T}^1 \) expressed in \( \mathbf{T}^2 \) is:

\[
C(D_j^1|\mathbf{T}^2) = \mathbf{FD}(D_j^1, \mathbf{T}^2).
\]

We also compute the extrinsic confidence of each atomic fact \( A_i^2 \) and discourse relation \( D_j^2 \) from \( \mathbf{T}^2 \) represented in the text \( \mathbf{T}^1 \), i.e., \( C(A_i^2|\mathbf{T}^1) = \mathbf{FD}(A_i^2, \mathbf{T}^1) \) and \( C(D_j^2|\mathbf{T}^1) = \mathbf{FD}(D_j^2, \mathbf{T}^1) \).

5.3 Factual Inclusion Score

We define a score to assess whether all atomic facts \( \mathbf{A}^1 \) and discourse relations \( \mathbf{D}^1 \) extracted from \( \mathbf{T}^1 \) are included in \( \mathbf{T}^2 \). The Factual Inclusion Score (FI) is computed based on two components: the inclusion of atomic facts (\( \mathbf{FI}_A \)) and the inclusion of discourse relations (\( \mathbf{FI}_D \)).

To mitigate the effect of potential hallucinated extractions, we apply the threshold \( \theta = 0.5 \) to the intrinsic confidence, i.e., only the atomic facts and discourse relations with their intrinsic confidence above \( \theta \) in \( \mathbf{T}^1 \) are considered for Factual Inclusion. We define the filtering function as:

\[
\delta_\theta(x) = \begin{cases}
1, & \text{if } x > \theta, \\
0, & \text{otherwise}.
\end{cases}
\]

We then score the inclusion of the atomic facts from \( \mathbf{T}^1 \) into \( \mathbf{T}^2 \) as follows:

\[
\mathbf{FI}_A(\mathbf{T}^1 \subset \mathbf{T}^2) = \sum_{i=1}^m \delta_\theta(C(A_i^1|\mathbf{T}^1)) C(A_i^1|\mathbf{T}^2).
\]
Similarly, the inclusion of the discourse relations from \( \mathbf{T}^1 \) within \( \mathbf{T}^2 \) is:

\[
\mathbf{FI}_D(\mathbf{T}^1 \subset \mathbf{T}^2) = \sum_{j=1}^{o} \delta_\theta(C(D_j^1|\mathbf{T}^1))C(D_j^1|\mathbf{T}^2).
\]

The overall Factual Inclusion Score (\( \mathbf{FI} \)) combines the inclusion score of atomic facts (\( \mathbf{FI}_A \)) and discourse relations (\( \mathbf{FI}_D \)), i.e.,

\[
\mathbf{FI}(\mathbf{T}^1 \subset \mathbf{T}^2) = \frac{1}{Z_I} (\mathbf{FI}_A(\mathbf{T}^1 \subset \mathbf{T}^2) \\
+ \mathbf{FI}_D(\mathbf{T}^1 \subset \mathbf{T}^2)).
\]

\( Z_I \) is a normalisation factor that scales the score according to the number of faithfully extracted atomic facts and discourse relations from \( \mathbf{T}^1 \), i.e.,

\[
Z_I = \sum_{i=1}^{m} \delta_\theta(C(A_i^1|\mathbf{T}^1)) + \sum_{k=1}^{o} \delta_\theta(C(D_k^1|\mathbf{T}^1)).
\]

5.4 Factual Overlap Score

To symmetrically evaluate if two texts contain the same contents, we compute the Factual Overlap Score (\( \mathbf{FO} \)) between \( \mathbf{T}^1 \) and \( \mathbf{T}^2 \), as follows:

\[
\mathbf{FO}(\mathbf{T}^1, \mathbf{T}^2) = \frac{1}{Z_D} (\mathbf{FI}_A(\mathbf{T}^1 \subset \mathbf{T}^2) + \mathbf{FI}_D(\mathbf{T}^1 \subset \mathbf{T}^2) \\
+ \mathbf{FI}_A(\mathbf{T}^2 \subset \mathbf{T}^1) + \mathbf{FI}_D(\mathbf{T}^2 \subset \mathbf{T}^1))
\]

\( Z_D \) is a normalisation factor that scales the result based on the total number of faithful atomic fact and discourse relation extractions from both texts:

\[
Z_D = \sum_{i=1}^{m} \delta_\theta(C(A_i^1|\mathbf{T}^1)) + \sum_{j=1}^{n} \delta_\theta(C(A_j^2|\mathbf{T}^2)) \\
+ \sum_{k=1}^{o} \delta_\theta(C(D_k^1|\mathbf{T}^1)) + \sum_{l=1}^{p} \delta_\theta(C(D_l^2|\mathbf{T}^2)).
\]

6 FDSpotter: Atomic Fact and Discourse Relation Entailment from the Text

To obtain the Factual Inclusion and Factual Overlap scores in Section 5, we train an entailment model, FDSpotter, to compute the probability of whether a structured atomic fact \( A_i \) or a discourse relation \( D_j \) is expressed in the text \( \mathbf{T} \). We collect atomic fact entailment and discourse relation entailment data for training and testing.

6.1 Atomic Fact Entailment Data

To compute the probability of a structured atomic fact \( A_i \) being present in a given text \( \mathbf{T} \), the model is trained on two groups of datasets.

1. Derived from FactSpotter: FactSpotter (Zhang et al., 2023) is a model for evaluating if an atomic fact from a knowledge graph (KG) is stated in a text generated from the KG. Its training data were generated using a self-supervised method. The positive samples were taken directly from datasets with pairs of atomic facts and their corresponding ground-truth texts, such as WebNLG (Castro Ferreira et al., 2020) or GrailQA (Gu et al., 2021). The negative samples were generated by perturbing the atomic facts or the corresponding descriptive texts, such that the atomic fact can no longer be entailed from the text. We use the same method to generate the training and testing samples from the WebNLG dataset.

2. Derived from Text Entailment Data: Text entailment corpus contain hypothesis-premise pairs and aim to determine if each hypothesis is entailed, contradictory, or neutral for the given premise text. The task is also known as Natural Language Inference (NLI). The use of NLI data is to introduce a degree of semantic flexibility in the training signal and to mitigate false negatives, cases where atomic facts are inferable across longer or more implicit spans of text but are not explicitly stated. We used the spaCy (Honnibal and Montani, 2017) model to extract the subject, predicate, object, adverbial and complement from a hypothesis, and add delimiters between them to structure the text into an atomic fact.
We generate pairs (natural language premises, hypotheses represented as structured atomic facts) from the following datasets: SNLI (MacCartney and Manning, 2008), MNLI (Williams et al., 2018), FEVER (Thorne et al., 2018), ANLI (Nie et al., 2020), LingNLI (Parish et al., 2021), WANLI (Liu et al., 2022), and CNC (Tan et al., 2022).

6.2 Discourse Relation Entailment Data

The models trained on the existing entailment datasets are good at identifying whether an atomic fact in the discourse relation is stated in the premise text. However, when classifying whether a discourse relation as a hypothesis is represented in the
premise, the accuracy decreases, especially when the discourse connective is adversarial.

For instance, consider the text: "Germany can be the birthplace of Heidegger, Hegel, Leibniz, Bach, Beethoven, Brecht, and Martin Luther, but they started one world war". The discourse relation ⟨⟨Germany, is the birthplace of, many great minds⟩, so, ⟨Germany, started, the world war⟩⟩ is not entailed from this text: both atomic facts are in the text, but the its connective "so" means the opposite to the connective "but" in the text. However, the models trained without adversarial samples consider this relation to be entailed from the premise.

To enable the classification of adversarial discourse relations from the premise, the model needs to be trained on data with pairs of texts and discourse relations, but especially include discourse connectives contradicting the ones from the premise. To the best of our knowledge, no such dataset exists. We leverage the NLI datasets (MNLI, ANLI, LingNLI, and WANLI) with large quantity of discourse connectives in the hypotheses to generate our DiscInfer (i.e., Discourse Relation Inference) dataset by the following steps.

1. Sample Selection. Automatically select pairs of hypotheses and premises from NLI data. Each hypothesis should contain at least one type of discourse connective in Section 5.4.

2. Connective Replacing. Replace one discourse connective in the hypothesis of the selected sample with another discourse connective that would make the hypothesis potentially contradict the premise, e.g., replacing "if" with "unless". Potential contradictory discourse connective pairs are listed in the Appendix B.

3. Human Verification. Human annotators check whether the relation between each new hypothesis-premise pair is entailing, neutral, or contradicting. The annotators correct the labels and add the created pairs to the dataset. The verification is due to changing the discourse connective in the hypothesis does not necessarily contradict the premise3 and the new hypothesis is sometimes not coherent.

The training split and the test split of DiscInfer are both created from the training and testing splits of MNLI, LingNLI, ANLI, and WANLI. The dataset has 920 annotated (hypothesis, premise) pairs.

6.3 Training and Evaluation

We fine-tune 304M DeBERTa V3 Large (He et al., 2021), pretrained on the tasksource dataset (Sileo, 2024). The model is finetuned on three parts of data: (1) original text entailment; (2) synthetic atomic fact entailment; (3) DiscInfer. The model is finetuned on NVIDIA Tesla V100 with learning rate 1e-5, batch size 16, and AdamW Optimiser (Loshchilov and Hutter, 2019) for 3 epochs.

<table>
  <tr>
    <th>Model/Split</th>
    <th>Temp.</th>
    <th>Cont.</th>
    <th>Comp.</th>
  </tr>
  <tr>
    <td>DeBERTa w/o DiscInfer</td>
    <td>58.2</td>
    <td>48.3</td>
    <td>23.1</td>
  </tr>
  <tr>
    <td>DeBERTa w DiscInfer</td>
    <td><b>76.4</b></td>
    <td><b>84.3</b></td>
    <td><b>59.6</b></td>
  </tr>
  <tr>
    <td>GPT4 zero-shot</td>
    <td>73.0</td>
    <td>53.8</td>
    <td>48.1</td>
  </tr>
</table>

Table 3: Accuracy of DeBERTa and GPT4 on DiscInfer.

In Table 3, we present the results on the test set of DiscInfer of DeBERTa and GPT4. For DeBERTa, we have one checkpoint that did not see the training split of DiscInfer, and another checkpoint finetuned along with DiscInfer. We observe that DeBERTa trained without DiscInfer does not perform well on the test split of discourse relation entailment, and there is a remarkable improvement after fine-tuning on DiscInfer. We have also tested zero-shot GPT4 performance on DiscInfer, with the prompt asking whether the hypotheses are entailed, neutral, or contradictory to the premises. GPT4 performs better than (much smaller) DeBERTa trained without DiscInfer, but underperforms compared with DeBERTa finetuned on DiscInfer.

7 Applications on Generation Evaluation

As explained in Section 5.4, the Factual Overlap Score is a symmetric metric to measure the content alignments between two texts, such as the ground-truth reference and the generated output. It symmetrically aggregates the Factual Inclusion Scores in both directions, evaluating whether both texts exactly contain the same atomic facts and discourse relations. This makes it suitable for the general text generation evaluation, e.g., graph-to-text, where bidirectional content equivalence is expected.

Reference-free faithfulness evaluation of some tasks requires an asymmetric approach, for which the Factual Inclusion in Section 5.3 can be applied in a single direction. For example, for graph-to-text generation faithfulness, we can use Factual Inclusion to determine whether all the triples \( \mathbf{F} = \)

3For example, the meaning of the discourse relation ⟨⟨He, did not finish, homework⟩, when, ⟨the class, started⟩⟩ is equivalent to ⟨⟨He, did not finish, homework⟩, before, ⟨the class, started⟩⟩ after replacing the discourse connective from "when" to be "before".
\{F_1, \ldots, F_m\}, which can be considered as atomic facts, are correctly verbalised in the text \( \mathbf{T} \), i.e.,

\[
\mathrm{FI}_A(\mathbf{F} \subset \mathbf{T}) = \sum_{i=1}^m C(F_i|\mathbf{T}).
\]

For text summary faithfulness classification, the generated summaries should be faithful to the input, but do not need to cover all of the input document. It is reflected in the consistency measurement of text summary benchmarks. To classify whether all the atomic facts and discourse relations in the generated summary \( \mathbf{S} \) are included by the input document \( \mathbf{I} \), we modify Factual Inclusion with the sum of log probabilities, i.e.,

\[
\mathrm{FI}_A(\mathbf{S} \subset \mathbf{I}) = \sum_{i=1}^m \delta_\theta(C(A_i|\mathbf{S})) \log C(A_i|\mathbf{I})
\]
\[
\mathrm{FI}_D(\mathbf{S} \subset \mathbf{I}) = \sum_{j=1}^o \delta_\theta(C(D_j|\mathbf{S})) \log C(D_j|\mathbf{I})
\]

To compare with existing metrics, we compute system-level and sample-level correlations between the automatic metrics and human judgements. We report three correlation coefficients: Spearman \( \rho \), Pearson \( r \), and Kendall’s Tau \( \tau \).

7.1 Natural Language Generation
WebNLG (Castro Ferreira et al., 2020) is a graph-to-text generation benchmark (i.e. triples or sets of triples of the form (subject, predicate, object) are paired with their equivalent texts) on which 16 systems have been evaluated by humans on data coverage (whether the text includes descriptions of all predicates in the data), correctness (whether the predicates in the data are correctly mentioned with the subject and object), and relevance (whether the text describes only the predicates in the data with related subjects and objects). Fluency and grammar evaluations were also reported by annotators, but they are not related to factual consistency. We report the performance of existing and our proposed metrics in Table 4, where the Factual Overlap and Factual Inclusion have the highest correlations on the dimensions related to factual consistency.

7.2 Text Summarisation
SummEval (Fabbri et al., 2020) benchmark has summaries from CNN and Daily Mail generated by 23 models with human judgements. We evaluate on the dimension of consistency, which reflects the factual alignment between the summary and the input, where the hallucinations are penalised by annotators. We compute the correlations of Factual Inclusion with human annotated consistency, and compare against BERTScore, BARTScore, as well as state-of-the-art methods such as FENICE (Scirè et al., 2024), ChatGPT-DA (Wan et al., 2023) and G-EVAL-4 (Liu et al., 2023b). We observe in Table 5 that Factual Inclusion has a strong performance.

AggreFact (Tang et al., 2023) contains summaries from CNN, Daily Mail and XSUM (Narayan et al., 2018). It focuses on factual errors in text summarisation and gives summaries a binary label: factual or non-factual. It evaluated three groups of models and pairs them with human evaluations: FTSOTA, EXFORMER, and OLD. FTSOTA has more recent models on which factuality metrics struggle. We compare with the following competitors: DAE (Goyal and Durrett, 2020), QuestEval (Scialom et al., 2021), SummaC (Laban et al., 2022), QAFactEval (Fabbri et al., 2022), TrueTeacher (Gekhman et al., 2023), MENLI (Chen and Eger, 2023), AlignScore (Zha et al., 2023), ChatGPT-ZS, ChatGPT-CoT (Luo et al., 2023), ChatGPT-DA, ChatGPT-Star, and FENICE. These metrics have been proposed as competitors in Scirè et al. (2024). We present in Table 6 the balanced accuracy on FTSOTA, and in the Appendix C on all the splits. We observe that our proposed Factual Inclusion method performs on par with the top-performing methods.

To evaluate the performance on diverse long-form summarisation faithfulness classification, we conduct experiments on DiverSumm (Zhang et al., 2024), a comprehensive benchmark that incorporates faithfulness annotations across five distinct domains: ChemSumm (CSM), QMSUM (QMS), ArXiv (AXV), GovReport (GOV), and MultiNews (MNW). DiverSumm presents unique challenges for faithfulness evaluation as it contains longer documents and summaries. We compare our proposed Factual Inclusion against several state-of-the-art faithfulness evaluation methods, including FullDoc, SummaC, SentLI (Schuster et al., 2022), and INFUSE (Zhang et al., 2024). Following Laban et al. (2022) and Zhang et al. (2024), we report ROC-AUC (Bradley, 1997) scores for faithfulness classification across all DiverSumm subsets. As shown in Table 7, Factual Inclusion achieves the highest average performance, substantially outperforming the previous state-of-the-art methods with consistent improvements across most domains.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="3">Text-level</th>
    <th colspan="3">System-level</th>
  </tr>
  <tr>
    <th>Correctness<br>r<br>ρ<br>τ</th>
    <th>D. Coverage<br>r<br>ρ<br>τ</th>
    <th>Relevance<br>r<br>ρ<br>τ</th>
    <th>Correctness<br>r<br>ρ<br>τ</th>
    <th>D. Coverage<br>r<br>ρ<br>τ</th>
    <th>Relevance<br>r<br>ρ<br>τ</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>67.2<br>57.3<br>45.3</td>
    <td>68.3<br>61.2<br>48.6</td>
    <td>65.1<br>50.2<br>40.8</td>
    <td>96.0<br>80.0<br>65.0</td>
    <td>93.0<br>83.0<br>68.0</td>
    <td>96.0<br>74.0<br>59.0</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>60.8<br>57.1<br>42.5</td>
    <td>56.9<br>60.0<br>44.5</td>
    <td>58.1<br>48.2<br>35.4</td>
    <td>59.6<br>64.4<br>48.5</td>
    <td>54.0<br>53.6<br>40.5</td>
    <td>57.0<br>60.0<br>45.2</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>65.2<br>59.5<br>42.9</td>
    <td>66.2<br>57.1<br>43.3</td>
    <td>68.1<br>55.2<br>41.8</td>
    <td>72.8<br>75.7<br>60.3</td>
    <td>65.7<br>58.3<br>45.0</td>
    <td>70.5<br>64.7<br>50.0</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>65.3<br>60.8<br>46.6</td>
    <td>67.1<br>61.4<br>46.5</td>
    <td>66.1<br>58.0<br>43.7</td>
    <td>83.1<br>77.7<br>60.5</td>
    <td>74.8<br>58.3<br>43.7</td>
    <td>81.0<br>65.9<br>50.8</td>
  </tr>
  <tr>
    <td>BARTScore</td>
    <td>71.5<br>59.6<br>45.4</td>
    <td>69.6<br>61.9<br>46.9</td>
    <td>70.4<br>61.8<br>47.1</td>
    <td>90.6<br>83.2<br>67.6</td>
    <td>87.0<br>71.5<br>53.4</td>
    <td>88.7<br>71.7<br>56.8</td>
  </tr>
  <tr>
    <td>BLEURT</td>
    <td>72.0<br>63.3<br>48.0</td>
    <td>68.8<br>60.5<br>46.0</td>
    <td>71.7<br>63.3<br>47.6</td>
    <td>93.1<br>82.9<br>67.6</td>
    <td>87.0<br>65.6<br>50.6</td>
    <td>91.1<br>70.0<br>55.8</td>
  </tr>
  <tr>
    <td>FactSpotter</td>
    <td>68.9<br>59.0<br>45.0</td>
    <td>71.2<br>64.0<br>48.8</td>
    <td>69.8<br>59.4<br>45.8</td>
    <td>94.7<br>80.2<br>64.2</td>
    <td>91.4<br>87.1<br>71.5</td>
    <td>96.2<br>80.0<br>64.9</td>
  </tr>
  <tr>
    <td><b>FactInclusion</b></td>
    <td>70.7<br>63.1<br>48.7</td>
    <td>72.3<br>63.6<br>49.0</td>
    <td>74.3<br>61.1<br>46.9</td>
    <td><b>97.3</b><br><b>81.6</b><br><b>67.4</b></td>
    <td><b>96.8</b><br><b>92.8</b><br><b>81.1</b></td>
    <td><b>96.6</b><br><b>79.5</b><br><b>64.7</b></td>
  </tr>
  <tr>
    <td><b>FactOverlap</b></td>
    <td><b>74.2</b><br><b>65.1</b><br><b>50.3</b></td>
    <td><b>72.9</b><br><b>62.7</b><br><b>48.5</b></td>
    <td><b>73.9</b><br><b>60.1</b><br><b>46.2</b></td>
    <td><b>96.9</b><br><b>78.6</b><br><b>64.0</b></td>
    <td><b>97.2</b><br><b>92.8</b><br><b>81.6</b></td>
    <td><b>95.4</b><br><b>74.3</b><br><b>59.2</b></td>
  </tr>
</table>

Table 4: Text-level and System-level correlations between metrics and human annotations of factuality dimensions on WebNLG2020. We highlight the best result and underline the other top 3 values in tables here and below.

<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="3">System-level</th>
    <th colspan="3">Sample-level</th>
  </tr>
  <tr>
    <th>r<br>ρ<br>τ</th>
    <th>r<br>ρ<br>τ</th>
    <th>r<br>ρ<br>τ</th>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>17.9<br>-7.6<br>-3.3</td>
    <td>10.9<br>15.2<br>9.0</td>
    <td>10.9<br>15.2<br>9.0</td>
  </tr>
  <tr>
    <td>BARTScore</td>
    <td>81.0<br>77.4<br>60.0</td>
    <td>40.1<br>48.7<br>33.2</td>
    <td>40.1<br>48.7<br>33.2</td>
  </tr>
  <tr>
    <td>G-EVAL-4</td>
    <td>92.3<br>76.5<br>60.0</td>
    <td>50.1<br>59.1<br>42.0</td>
    <td>50.1<br>59.1<br>42.0</td>
  </tr>
  <tr>
    <td>ChatGPT-DA</td>
    <td>97.0<br>81.2<br>68.3</td>
    <td>41.9<br>51.7<br>38.9</td>
    <td>41.9<br>51.7<br>38.9</td>
  </tr>
  <tr>
    <td>FENICE-GPT</td>
    <td>91.8<br>82.4<br>68.3</td>
    <td>43.5<br>65.9<br>36.0</td>
    <td>43.5<br>65.9<br>36.0</td>
  </tr>
  <tr>
    <td><b>FactInclusion</b></td>
    <td><b>96.5</b><br><b>85.9</b><br><b>70.0</b></td>
    <td><b>40.8</b><br><b>64.5</b><br><b>34.0</b></td>
    <td><b>40.8</b><br><b>64.5</b><br><b>34.0</b></td>
  </tr>
</table>

Table 5: System-level and sample-level correlations between metrics and human annotations of consistency on SummEval benchmark.

<table>
  <tr>
    <th>Metric</th>
    <th>CNN/DM</th>
    <th>XSUM</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>DAE</td>
    <td>65.4±4.4</td>
    <td>70.2±2.3</td>
    <td>67.8</td>
  </tr>
  <tr>
    <td>QuestEval</td>
    <td>70.2±3.2</td>
    <td>59.5±2.7</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td>SummaC-ZS</td>
    <td>64.0±3.8</td>
    <td>56.4±1.2</td>
    <td>60.2</td>
  </tr>
  <tr>
    <td>SummaC-Conv</td>
    <td>61.0±3.9</td>
    <td>65.0±2.2</td>
    <td>63.0</td>
  </tr>
  <tr>
    <td>QAFactEval</td>
    <td>67.8±4.1</td>
    <td>63.9±2.4</td>
    <td>65.9</td>
  </tr>
  <tr>
    <td>TrueTeacher</td>
    <td>62.0±1.3</td>
    <td><b>74.9±1.2</b></td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>MENLI</td>
    <td>65.0±2.8</td>
    <td>57.0±1.8</td>
    <td>61.0</td>
  </tr>
  <tr>
    <td>AlignScore</td>
    <td>67.0±3.1</td>
    <td>60.3±1.9</td>
    <td>63.7</td>
  </tr>
  <tr>
    <td>ChatGPT-ZS</td>
    <td>56.3±2.9</td>
    <td>62.7±1.7</td>
    <td>59.5</td>
  </tr>
  <tr>
    <td>ChatGPT-CoT</td>
    <td>52.5±3.3</td>
    <td>55.9±2.1</td>
    <td>54.2</td>
  </tr>
  <tr>
    <td>ChatGPT-DA</td>
    <td>53.7±3.5</td>
    <td>54.9±1.9</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>ChatGPT-Star</td>
    <td>56.3±3.1</td>
    <td>57.8±0.2</td>
    <td>57.1</td>
  </tr>
  <tr>
    <td>FENICE-GPT</td>
    <td><b>70.5±1.6</b></td>
    <td>72.8±0.3</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td><b>FactInclusion</b></td>
    <td><b>69.6±3.4</b></td>
    <td><b>73.8±1.7</b></td>
    <td><b>71.7</b></td>
  </tr>
</table>

Table 6: Balanced accuracy of on FTsOTA split of AggreFact, with single-threshold setting.

<table>
  <tr>
    <th>System</th>
    <th>CSM</th>
    <th>QMS</th>
    <th>AXV</th>
    <th>GOV</th>
    <th>MINW</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>FullDoc</td>
    <td>50.15</td>
    <td>37.12</td>
    <td>62.78</td>
    <td>79.19</td>
    <td>44.76</td>
    <td>54.80</td>
  </tr>
  <tr>
    <td>SummaC-Conv</td>
    <td>53.14</td>
    <td>51.13</td>
    <td>61.22</td>
    <td>65.34</td>
    <td>53.05</td>
    <td>56.78</td>
  </tr>
  <tr>
    <td>SummaC-ZS</td>
    <td>54.41</td>
    <td>48.21</td>
    <td>69.44</td>
    <td>79.37</td>
    <td>50.17</td>
    <td>60.32</td>
  </tr>
  <tr>
    <td>SentLI</td>
    <td>50.13</td>
    <td>47.56</td>
    <td>64.49</td>
    <td>79.68</td>
    <td>46.61</td>
    <td>57.69</td>
  </tr>
  <tr>
    <td>INFUSE</td>
    <td>54.11</td>
    <td>52.16</td>
    <td>71.38</td>
    <td>80.45</td>
    <td>53.16</td>
    <td>62.25</td>
  </tr>
  <tr>
    <td><b>FactInclusion</b></td>
    <td><b>56.79</b></td>
    <td><b>70.30</b></td>
    <td><b>70.56</b></td>
    <td><b>83.44</b></td>
    <td><b>44.88</b></td>
    <td><b>65.20</b></td>
  </tr>
</table>

Table 7: ROC-AUC scores on DiverSumm benchmark for long form summary faithfulness evaluation.

DiscInfer, which is challenging for the language models trained with existing data. While factual consistency verification is a rapidly evolving field, we believe it is important to leverage fundamental linguistic knowledge so that the advances remain explainable and hence more trustworthy.

Acknowledgement.

The authors were partially funded by the ANR-20-CHIA-0015 project and the ANRT. This work was performed using HPC resources from GENCI-IDRIS (Grant AD011014244R1). The authors thank Huajian Zhang and Yang Zhong for baseline reproduction. We are also grateful to Claire Gardent and Simon Razniewski for their valuable suggestion on improving the paper. Kun Zhang thanks his high school teachers for the solid instruction in English grammar and linguistics.

9 Limitations

Our work has the following limitations:

• We have tested our method only on English text, and in particular the structured sentence representation is based on English syntax. We believe it is very important to address this task in other language. However, due to time and knowledge limitations we could not include it in our work.
• We took advantage of NLI data to train the classifier. However, the hypotheses of NLI may include facts which are semantically plausible but not explicitly grounded in the premise. There exists a conceptual gap between entailment and strict fact inclusion. Training with such data is a trade-off between minimising false negatives, where legitimate facts are mistakenly excluded due to the lack of exact match, and avoiding false positives, where inferred but unstated facts are accepted.

• While our metric is explainable in the sense we can obtain very fine-grained information on how a factuality score was given, nowadays explanations in textual format given by LLMs have gained a lot of attention; we believe that this is a future step that is attainable with current techniques.

• We are currently completing a comprehensive error analysis for our technique and for competitors. Understanding to what type of errors each method is sensitive can be useful for further improving the SOTA. Another question that previous work have tried to answer using automated techniques and human annotation (Zhang et al., 2023) is if the benchmarks are still challenging or if the remaining model errors are actually due to human errors in annotations or divergence of annotations.

References

Brent H Baker, Tim Graham, and Steve Kaminsky. 1994. How to identify, expose & correct liberal media bias. Media Research Center Alexandria, VA.

Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178–186.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.

Andrew P. Bradley. 1997. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145–1159.

Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, and Amir Zeldes. 2023. The DISRPT 2023 shared task on elementary discourse unit segmentation, connective detection, and relation classification. In Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023), pages 1–21, Toronto, Canada. The Association for Computational Linguistics.

Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 55–76, Dublin, Ireland (Virtual). Association for Computational Linguistics.

Sylvie Cazalens, Philippe Lamartre, Julien Leblay, Ioana Manolescu, and Xavier Tannier. 2018. A content management perspective on fact-checking. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 565–574, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

Huiyao Chen, Meishan Zhang, Jing Li, Min Zhang, Lilja Øvrelid, Jan Hajíček, and Hao Fei. 2025a. Semantic role labeling: A systematical survey.

Huiyao Chen, Meishan Zhang, Jing Li, Min Zhang, Lilja Øvrelid, Jan Hajíček, and Hao Fei. 2025b. Semantic role labeling: A systematical survey.

Yanran Chen and Steffen Eger. 2023. MENLI: Robust evaluation metrics from natural language inference. Transactions of the Association for Computational Linguistics, 11:804–825.

Luciano Del Corro and Rainer Gemulla. 2013. Clause: clause-based open information extraction. In Proceedings of the 22nd international conference on World Wide Web, pages 355–366.

Kuicai Dong, Aixin Sun, Jung-jae Kim, and Xiaoli Li. 2023. Open information extraction via chunks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15390–15404, Singapore. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey et al. 2024. The llama 3 herd of models.

Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Communications of the ACM, 51(12):68–74.

A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, Richard Socher, and Dragomir R. Radev. 2020. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409.
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2587–2601, Seattle, United States. Association for Computational Linguistics.

Zorik Gekhman, Jonathan Herzig, Roei Aharoni, Chen Elkind, and Idan Szpektor. 2023. TrueTeacher: Learning factual consistency evaluation with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2053–2070, Singapore. Association for Computational Linguistics.

Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 166–175, New York, NY, USA. Association for Computing Machinery.

Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592–3603, Online. Association for Computational Linguistics.

Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449–1462, Online. Association for Computational Linguistics.

Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond i.i.d.: Three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, WWW ’21, page 3477–3488, New York, NY, USA. Association for Computing Machinery.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pretraining with gradient-disentangled embedding sharing.

Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.

Shafiq Joty, Francisco Guzmán, Lluís Márquez, and Preslav Nakov. 2017. Discourse structure in machine translation evaluation. Computational Linguistics, 43(4):683–722.

René Knaebel. 2021. discopy: A neural system for shallow discourse parsing. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 128–133, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics.

Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177.

Young-Suk Lee, Ramón Astudillo, Hoang Thanh Lam, Tahira Naseem, Radu Florian, and Salim Roukos. 2022. Maximum Bayes Smatch ensemble distillation for AMR parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5379–5392, Seattle, United States. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Wei Liu, Yi Fan, and Michael Strube. 2023a. HITS at DISRPT 2023: Discourse segmentation, connective detection, and relation classification. In Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023), pages 43–49, Toronto, Canada. The Association for Computational Linguistics.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization. arXiv preprint arXiv:2303.15621.

Bill MacCartney and Christopher D. Manning. 2008. Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 521–528, Manchester, UK. Coling 2008 Organizing Committee.

Eleni Metheniti, Chloé Braud, Philippe Muller, and Laura Rivière. 2023. DisCut and DiscReT: MELODI at DISRPT 2023. In Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023), pages 29–42, Toronto, Canada. The Association for Computational Linguistics.
Yisong Miao, Hongfu Liu, Wenqiang Lei, Nancy Chen, and Min-Yen Kan. 2024. Discursive socratic questioning: Evaluating the faithfulness of language models’ understanding of discourse relations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6277–6295, Bangkok, Thailand. Association for Computational Linguistics.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.

Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo Walenta, Juri Opitz, Leonardo Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Sebastian Gehrmann, Lining Zhang, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, and Yufang Hou. 2024. On the role of summary content units in text summarization evaluation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 272–281, Mexico City, Mexico. Association for Computational Linguistics.

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.

Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2018. A survey on open information extraction. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3866–3878, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Stephan Oepen, Jonathon Read, Tatjana Scheffler, Uladzimir Sidarena, Manfred Stede, Erik Vellidal, and Lilja Øvrelid. 2016. OPT: Oslo–Potsdam–Teesside, pipelining rules, rankers, and classifier ensembles for shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 20–26, Berlin, Germany. Association for Computational Linguistics.

OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal et al. 2024. Gpt-4 technical report.

Liu Pai, Wenyang Gao, Wenjie Dong, Lin Ai, Ziwei Gong, Songfang Huang, Li Zongsheng, Ehsan Hoque, Julia Hirschberg, and Yue Zhang. 2024. A survey on open information extraction from rule-based model to large language model. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9586–9608, Miami, Florida, USA. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanyaa Aggarwal, Emile Allaway, Tal Linzen, and Samuel R. Bowman. 2021. Does putting a linguist in the loop improve NLU data collection? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4886–4901, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn Discourse Treebank Version 3.0. Abacus Data Network.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.

Alix Rule, Jean-Philippe Cointet, and Peter S Bearman. 2015. Lexical shifts, substantive changes, and continuity in state of the union discourse, 1790–2014. Proceedings of the National Academy of Sciences, 112:10837 – 10844.

Zacchary Sadeddine, Juri Opitz, and Fabian Suchanek. 2024. A survey of meaning representations – From theory to practical utility. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2877–2892, Mexico City, Mexico. Association for Computational Linguistics.

Tal Schuster, Sihaq Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching sentence-pair NLI models to reason over long documents and clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 394–412, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alessandro Scirè, Karim Ghonim, and Roberto Navigli. 2024. FENICE: Factuality evaluation of summarization based on natural language inference and claim extraction. In Findings of the Association for Computational Linguistics ACL 2024, pages 14148–14161, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696.

Damien Sileo. 2024. tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15655–15684, Torino, Italia, ELRA and ICCL.

Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885–895, New Orleans, Louisiana. Association for Computational Linguistics.

Fabian M. Suchanek. 2020. The need to move beyond triples. In Text2Story@ECIR.

Fiona Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk, Tadashi Nomoto, Hansi Het-tiarachchi, Iqra Ameer, Onur Uca, Farhana Ferdousi Liza, and Tiancheng Hu. 2022. The causal news corpus: Annotating causal relations in event sentences from news. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2298–2310, Marseille, France. European Language Resources Association.

Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11626–11644, Toronto, Canada. Association for Computational Linguistics.

Liyan Tang, Philippe Laban, and Greg Durrett. 2024. Minicheck: Efficient fact-checking of llms on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.

Prajna Upadhyay, Oana Balalau, and Ioana Manolescu. 2023. Open information extraction with entity focused constraints. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1285–1296, Dubrovnik, Croatia. Association for Computational Linguistics.

Robert A. Wagner and Michael J. Fischer. 1974. The string-to-string correction problem. J. ACM, 21(1):168–173.

Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. GPT-RE: In-context learning for relation extraction using large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3534–3547, Singapore. Association for Computational Linguistics.

Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 17–24, Beijing, China. Association for Computational Linguistics.

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 11122–1122, New Orleans, Louisiana. Association for Computational Linguistics.

Ruth Wodak and Michael Meyer. 2015. Methods of critical discourse studies.

Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. 2023. Large language models for generative information extraction: A survey. Frontiers Comput. Sci., 18:186357.

Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1–16, Beijing, China. Association for Computational Linguistics.

Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. CoNLL 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19, Berlin, Germany. Association for Computational Linguistics.

Zinong Yang, Feng Xu, Jianfei Yu, and Rui Xia. 2023. UniCOQE: Unified comparative opinion quintuple extraction as a set. In Findings of the Association for
Computational Linguistics: ACL 2023, pages 12229–12240, Toronto, Canada. Association for Computational Linguistics.

Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics.

Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. 2024. Fine-grained natural language inference based faithfulness evaluation for diverse summarisation tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1701–1722, St. Julian’s, Malta. Association for Computational Linguistics.

Kun Zhang, Oana Balalau, and Ioana Manolescu. 2023. FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation. In Findings of EMNLP 2023 - Conference on Empirical Methods in Natural Language Processing, Singapore, Singapore.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Jinman Zhao and Gerald Penn. 2024. LLM-supertagger: Categorical grammar supertagging via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 697–705, Miami, Florida, USA. Association for Computational Linguistics.

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China. Association for Computational Linguistics.

Yang Zhong and Diane Litman. 2025. Discourse-driven evaluation: Unveiling factual inconsistency in long document summarization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2050–2073, Albuquerque, New Mexico. Association for Computational Linguistics.

A Grammatical form of Discourse Connectives

Syntactically, the discourse connectives are frequently expressed by conjunctives, relative adverbs or other transitional phrases. Below are several representative grammatical forms and their corresponding structured representation.

• Two clauses connected by a conjunction. For example, the text "Amy and Kate were both in good mood, so they went to celebrate happily" is structured to be ⟨⟨Amy and Kate, were both, in good mood⟩, so, ⟨Amy and Kate, went to celebrate, happily⟩⟩.

• Two clauses in different sentences connected by a transitional word or phrase. For instance, the text "Amy and Kate celebrated until midnight. As a result, they got up late on the next day" is represented as ⟨⟨Amy and Kate, celebrated, until midnight⟩, as a result, ⟨Amy and Kate, got up, late on the next day⟩⟩.

• Relative adverbs (e.g., when, why) leading adverbial clauses. For example, the text "The company built a new school where there was a park" is represented as ⟨⟨the company, built, a new school⟩, where, ⟨there, was, a park⟩⟩.

• Complex phrases connecting adverbial clauses. For instance, "Despite the fact that he cheated on the exam, he was not dismissed by the university" is represented as ⟨⟨he, was not dismissed, by the university⟩, despite the fact that, ⟨he, cheated, on the exam⟩⟩.

B Creation of the DiscInfer Dataset

To facilitate the construction of the DiscInfer dataset for training and evaluating models on identifying contradictions induced by adversarial discourse connectives, we automatically generated adversarial examples by replacing one discourse connective in the selected hypothesis with another discourse connective that could potentially render the hypothesis contradictory to the premise. The groups of potentially conflicting discourse relations are listed in Table 8.

To ensure grammatical plausibility, a discourse connective is replaced only if the original and replaced connectives belong to the same grammatical category, such as subordinating conjunctions or adverbial phrases, to preserve syntactic coherence.

The annotators were trained on the linguistic concepts related to discourse connectives (e.g., conjunctions, transitional phrases, adverbial clauses, etc.). They were also provided with a reference
<table>
  <tr>
    <th>Original connective</th>
    <th>Potential conflicting connectives</th>
  </tr>
  <tr>
    <td>concession</td>
    <td>condition; reason; result.</td>
  </tr>
  <tr>
    <td>contrast</td>
    <td>reason; result; similarity.</td>
  </tr>
  <tr>
    <td>similarity</td>
    <td>concession; contrast.</td>
  </tr>
  <tr>
    <td>negative condition</td>
    <td>condition; reason; result.</td>
  </tr>
  <tr>
    <td>condition</td>
    <td>concession; negative condition; result.</td>
  </tr>
  <tr>
    <td>reason</td>
    <td>concession; negative condition; precedence; result.</td>
  </tr>
  <tr>
    <td>result</td>
    <td>concession; condition; contrast; reason; succession.</td>
  </tr>
  <tr>
    <td>succession</td>
    <td>precedence; result; synchronous.</td>
  </tr>
  <tr>
    <td>synchronous</td>
    <td>precedence; succession.</td>
  </tr>
  <tr>
    <td>precedence</td>
    <td>concession; reason; succession; synchronous.</td>
  </tr>
</table>

Table 8: Potential conflicting discourse relation groups.

list of connective groups and examples of potentially conflicting discourse relations. All annotators have completed postgraduate-level education and possess at least a C1 level of English proficiency.

For completeness, we also list examples of discourse connectives of each relation category as follows. These examples are primarily drawn from the Penn Discourse Treebank (Prasad et al., 2019).

• alternative: alternatively, as an alternative, instead of.

• concession: although, in spite of, even though, though, despite, even if, albeit, regardless of.

• condition: depending on, depending upon, if, provided that, in case.

• contrast: by/in contrast, conversely, however, on the contrary, on the other hand, rather than, whereas, by/in comparison, but.

• disjunctive: otherwise, unless, except.

• precedence: thereafter, later, then, before, subsequently, afterwards, afterward.

• reason: as a result of, because, because of, due to, as a consequence of.

• result: consequently, therefore, thus, hence, thereby, so, so that, for the purpose of, with the purpose of, in order to, as a result, as a consequence, for this/that reason.

• similarity: similarly, in a similar way, in the same way, likewise.

• succession: earlier, previously, until, till, after.

• synchronous: when, during, at the same time, meanwhile, simultaneously, meantime.

To ensure balanced coverage across discourse relation categories, we aimed to include at least 100 training examples and 50 test examples per high-level group (comparison, contingency, temporal). However, due to differences in connective frequency in NLI datasets, the final distribution in the dataset varied. Table 10 illustrates the category distribution of Discinfer training and test samples.

C Detailed result on AggreFact

Table 9 reports balanced accuracy on AggreFact, with threshold-per-split settings. The accuracy is computed with the same method as Tang et al., and we report 95% confidence intervals for our method and the baselines with publicly available outputs. Following Scirè et al., the performance of DAE is excluded on the EXF (Exformer) and OLD splits of XSum because its training data (Goyal and Durrett, 2021) covers these splits.

We compare the effectiveness of structured representations for factual consistency verification. Open Information Extraction (Open IE) systems typically extract subject-predicate-object (SPO) triples. However, the triple representations have semantic incompleteness, as they omit adverbials and complements. We propose to enhance atomic facts by including adverbials and complements, and to incorporate discourse relations to represent connections between facts. As shown in Table 11, using atomic facts outperforms the triple format by including more complete content units. Also, incorporating discourse relations further improves the classification performance. We further assess the effect of training with DiscInfer dataset, which introduces adversarial discourse connectives. As is shown in Table 12, DiscInfer improves the ability of models for factual consistency verification.

D Complexity Comparison for Factual Consistency Verification Methods

We compare the computational complexity of the approaches for factual consistency verification. We categorize existing methods into three groups.

1. Simple Text Entailment. These methods use a classification model (e.g., DeBERTa (He et al., 2021) for NLI) once per instance, taking one text (e.g. text summarisation output) as the hypothesis and another text (e.g., the input document of text summarisation) as the premise.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">AggreFact-CNN</th>
    <th colspan="4">AggreFact-XSum</th>
  </tr>
  <tr>
    <th>FTS</th>
    <th>EXF</th>
    <th>OLD</th>
    <th>AVG</th>
    <th>FTS</th>
    <th>EXF</th>
    <th>OLD</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>DAE</td>
    <td>59.4±3.1</td>
    <td>67.9±3.1</td>
    <td>69.7±1.1</td>
    <td>65.6</td>
    <td>73.1±1.8</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>QuestEval</td>
    <td>63.7±3.2</td>
    <td>64.3±3.1</td>
    <td>65.2±1.1</td>
    <td>64.4</td>
    <td>61.6±1.9</td>
    <td>60.1±4.5</td>
    <td>59.7±3.4</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>SummaC-ZS</td>
    <td>63.3±3.0</td>
    <td><b>76.5±2.8</b></td>
    <td>76.3±1.0</td>
    <td><b>72.0</b></td>
    <td>56.1±2.0</td>
    <td>51.4±4.6</td>
    <td>53.3±3.8</td>
    <td>53.6</td>
  </tr>
  <tr>
    <td>SummaC-Conv</td>
    <td><b>70.3±2.5</b></td>
    <td>69.8±3.2</td>
    <td>78.9±1.1</td>
    <td><b>73.0</b></td>
    <td>67.0±2.0</td>
    <td>64.6±4.3</td>
    <td>67.5±4.1</td>
    <td>66.4</td>
  </tr>
  <tr>
    <td>QAFactEval</td>
    <td>61.6±3.6</td>
    <td>69.1±3.0</td>
    <td>80.3±1.0</td>
    <td>70.3</td>
    <td>65.9±2.0</td>
    <td>59.6±4.7</td>
    <td>60.5±5.4</td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>TrueTeacher-11B</td>
    <td>65.7</td>
    <td>57.7</td>
    <td>81.9</td>
    <td>68.4</td>
    <td>75.2</td>
    <td>68.4</td>
    <td>52.8</td>
    <td>65.5</td>
  </tr>
  <tr>
    <td>MENLI</td>
    <td>51.7</td>
    <td>52.8</td>
    <td>68.4</td>
    <td>57.6</td>
    <td>58.3</td>
    <td>59.7</td>
    <td><b>73.9</b></td>
    <td>64.0</td>
  </tr>
  <tr>
    <td>AlignScore</td>
    <td>53.5</td>
    <td><b>73.9</b></td>
    <td>78.0</td>
    <td>68.5</td>
    <td><b>80.2</b></td>
    <td><b>79.9</b></td>
    <td>63.7</td>
    <td><b>74.6</b></td>
  </tr>
  <tr>
    <td>ChatGPT-ZS</td>
    <td>66.2</td>
    <td>64.5</td>
    <td>74.3</td>
    <td>68.3</td>
    <td>62.6</td>
    <td>69.2</td>
    <td>60.1</td>
    <td>64.0</td>
  </tr>
  <tr>
    <td>ChatGPT-CoT</td>
    <td>49.7</td>
    <td>60.4</td>
    <td>66.7</td>
    <td>59.0</td>
    <td>56.0</td>
    <td>60.9</td>
    <td>50.1</td>
    <td>55.7</td>
  </tr>
  <tr>
    <td>ChatGPT-DA</td>
    <td>48.0</td>
    <td>63.6</td>
    <td>71.0</td>
    <td>60.9</td>
    <td>53.6</td>
    <td>65.6</td>
    <td>61.5</td>
    <td>60.2</td>
  </tr>
  <tr>
    <td>ChatGPT-Star</td>
    <td>55.8</td>
    <td>65.8</td>
    <td>71.2</td>
    <td>64.3</td>
    <td>57.7</td>
    <td>70.6</td>
    <td>53.8</td>
    <td>60.7</td>
  </tr>
  <tr>
    <td>FENICE<sub>gpt</sub></td>
    <td>68.2</td>
    <td>68.8</td>
    <td><b>82.1</b></td>
    <td><b>73.0</b></td>
    <td>73.9</td>
    <td><b>73.5</b></td>
    <td>69.9</td>
    <td><b>72.4</b></td>
  </tr>
  <tr>
    <td>FactOverlap<sub>gpt</sub></td>
    <td>63.7±3.3</td>
    <td>69.6±3.0</td>
    <td>79.0±1.0</td>
    <td>70.7</td>
    <td><b>74.7±1.8</b></td>
    <td><b>73.4±3.7</b></td>
    <td><b>73.0±3.6</b></td>
    <td><b>73.7</b></td>
  </tr>
</table>

Table 9: Balanced accuracy on the test splits of AggreFact, with threshold-per-split settings.

<table>
  <tr>
    <th>Category</th>
    <th>Size</th>
    <th>Details</th>
  </tr>
  <tr>
    <th colspan="3">Train</th>
  </tr>
  <tr>
    <td>Temporal</td>
    <td>186</td>
    <td>Succesion: 55, Precedence: 72, Synchronous: 59</td>
  </tr>
  <tr>
    <td>Contingency</td>
    <td>237</td>
    <td>Condition: 100, Reason: 68, Result: 69</td>
  </tr>
  <tr>
    <td>Comparison</td>
    <td>117</td>
    <td>Concession: 62, Contrast: 55</td>
  </tr>
  <tr>
    <th colspan="3">Test</th>
  </tr>
  <tr>
    <td>Temporal</td>
    <td>123</td>
    <td>Succesion: 49, Precedence: 37, Synchronous: 37</td>
  </tr>
  <tr>
    <td>Contingency</td>
    <td>248</td>
    <td>Condition: 136, Reason: 85, Result: 19</td>
  </tr>
  <tr>
    <td>Comparison</td>
    <td>52</td>
    <td>Concession: 34, Contrast: 18</td>
  </tr>
</table>

Table 10: Distribution of DiscInfer training and test sets.

<table>
  <tr>
    <th>Extraction Format/Split</th>
    <th>CNN/DM</th>
    <th>XSUM</th>
  </tr>
  <tr>
    <td>Subject-Predicate-Object Only</td>
    <td>65.4</td>
    <td>69.6</td>
  </tr>
  <tr>
    <td>Atomic Facts Only</td>
    <td>66.4</td>
    <td>72.4</td>
  </tr>
  <tr>
    <td>Atomic Facts + Discourse Relation</td>
    <td><b>69.6</b></td>
    <td><b>73.8</b></td>
  </tr>
</table>

Table 11: Balanced accuracy of different extraction formats on AggreFact.

<table>
  <tr>
    <th>Checkpoint</th>
    <th>CNN-DM</th>
    <th>XSUM</th>
  </tr>
  <tr>
    <td>w/o DiscInfer</td>
    <td>62.6</td>
    <td>72.0</td>
  </tr>
  <tr>
    <td>w/ DiscInfer</td>
    <td><b>69.6</b></td>
    <td><b>73.8</b></td>
  </tr>
</table>

Table 12: Effect of DiscInfer on performance.

question answering of multiple rounds.

3. Information Extraction + Entailment These methods first extract content units from the input by semantic parsing (Goyal and Durrett, 2020), sentence tokenisers (Chen and Eger, 2023), or LLM atomic fact extractors (Scirè et al., 2024), and then they check the factual consistency of the extracted content units. Their computational complexity is:

\[
T_{IE+E} = T_{extraction} + N_{content\ units} \times T_{entailment}.
\]

The complexity of the content unit extractors varies: sentence tokenisers and semantic parsers are lightweight, while both FENICE (Scirè et al., 2024) and our method use LLMs. Moreover, the number of extracted units also varies, typically,

\[
N_{sentences} \leq N_{atomic\ facts}.
\]

If a classification model is introduced to score all top extractions for selecting the most faithful extraction, this would incur additional overhead:

\[
N_{extractions} \times (N_{content\ units}) \times T_{entailment}.
\]

Our proposed Factual Inclusion Score (FI) belongs to the third category, which combines atomic
facts with discourse relations as the content units. Therefore, the number of the content unit becomes:

\[
N_{\text{content\_units}} = (N_{\text{atomic facts}} + N_{\text{discourse relations}}).
\]

Then the overall complexity changes according to the number of atomic facts and discourse relations. Given the number of discourse relations is usually smaller than the atomic facts, the runtime overhead of our method is marginally higher in this step.

E Extraction evaluation

We present the prompt we used for the few shot extraction of atomic facts and discourse relations in Figure 1. In Tables 13 and 14 we present the results of the extraction using different models and temperatures. We use a 16 bit quantified Llama 3.1 8B model and 4 bit quantified 70B model. We observe that when we change the temperature, there is a fluctuation of two to three percent between the results, with no temperature being generally better across all models. For Llama 3.1 7B model with temperature 0.2 and 0.3 we observe a significant drop in the recall for atomic fact extraction. We verified the extraction results, and Llama 3.1 8B indeed extracts less atomic facts under these two temperatures, while the extraction of the discourse connectives is not affected. This highlights the complex problem of finding the best parameters.
Task
1. extract all atomic facts from the given text and list them in a markdown table with columns: subject, predicate, object direct, object indirect, short adverbial and complement.

2. Extract relations between the events, with columns: discourse1, connective, discourse2. The relations are frequently connected by the following groups of connectives: concession (eg, although, even if); reason (eg, due to, because); result (eg, consequently, therefore); contrast (eg., in contrast, however); disjunctive (eg, otherwise, unless); alternative (eg., instead, as an alternative); condition (eg., provided that, in case); synchronous (eg, when, at the same time); precedence (eg, before, until); succession (eg, after, subsequently); similarity (eg, similarly) in the same way); location (eg, where). The connectives shouldn’t be limited by the examples.

Example text: Being thankful about the exam’s results, person A gave a gift to her best friend, person B, last week. Person A would have failed, if person B were not helping. Person B said that her best friend did not need to do this, even if person B helped person A revise all the course materials from time to time. They held a party until midnight. As a result, they didn’t get up early on the next day.

Expected output Format:
# Atomic Facts
<table>
  <tr>
    <th>Subject</th>
    <th>Predicate</th>
    <th>Object Direct</th>
    <th>Object Indirect</th>
    <th>Short Adverbial and Complement</th>
  </tr>
  <tr>
    <td>person A</td>
    <td>gave</td>
    <td>a gift</td>
    <td>to person A's best friend, person B</td>
    <td>- | last week |</td>
  </tr>
  <tr>
    <td>person A</td>
    <td>was thankful about</td>
    <td>the exam's results</td>
    <td>-</td>
    <td>- | - | -</td>
  </tr>
  <tr>
    <td>person B</td>
    <td>said</td>
    <td>her best friend did not need to give a gift to person B</td>
    <td>-</td>
    <td>- | - | -</td>
  </tr>
  <tr>
    <td>person B</td>
    <td>helped</td>
    <td>person A</td>
    <td>from time to time</td>
    <td>to revise all the course materials</td>
  </tr>
  <tr>
    <td>person A and person B</td>
    <td>held</td>
    <td>a party</td>
    <td>-</td>
    <td>until midnight</td>
  </tr>
  <tr>
    <td>person A and person B</td>
    <td>didn't get up</td>
    <td>-</td>
    <td>-</td>
    <td>early on the next day</td>
  </tr>
</table>

# Discourse Relations
<table>
  <tr>
    <th>Event1</th>
    <th>Connective</th>
    <th>Event2</th>
  </tr>
  <tr>
    <td>person A gave a gift to person A's best friend, person B</td>
    <td>because of</td>
    <td>being thankful about the exam's results</td>
  </tr>
  <tr>
    <td>person A would have failed</td>
    <td>if</td>
    <td>person B were not helping</td>
  </tr>
  <tr>
    <td>person B said person A did not need to give a gift to person B</td>
    <td>even if</td>
    <td>person B helped person A revise all the course materials from time to time</td>
  </tr>
  <tr>
    <td>person A and person B held a party until midnight</td>
    <td>As a result</td>
    <td>person A and person B didn't get up early on the next day</td>
  </tr>
</table>

# Standards

Having correct tense for predicate is important.

Do not miss any detailed information, especially features, or identities of the entities.

Do not extract content that is not present in the input.

Perform co-reference resolution on entities: if an entity has a full name, put the full name, rather than a pronoun.

If an auxiliary verb refers to another predicate, put their original predicate. For example, "did not need to do this" in the original text is extracted to be "did not need to give a gift to person B".

Make sure the predicate links subject and object in a correct direction. For example, we only know "person B | helped | person A" from the text, rather than "person A | helped | person B".

If the subject or object has another atomic fact in its adjective, clause, non-finite verb, or appositive, place all related words in the corresponding column and the new fact should also be in another line of the table. For example, "person B | is best friend of | person A" is a new fact inside the appositive "person A's best friend, person B".

If two discourses are linked with a connective, connect them in the discourse relation table. For example, the events "person A | gave | a gift | to person A's best friend, person B | last week" and "person A | was thankful about | the exam's results" are connected with "because of". The discourse connective shouldn't appear in "Short Adverbial and complement".

If a discourse is hypothetical or counterfactual situation, such as subjunctive mood, output the discourse relation but don't extract it as an atomic fact. For example, "person B were not helping" and "person A would have failed" are not atomic facts, but discourse relation should contain "person A would have failed | if | person B were not helping".

Don't output atomic facts without a predicate.

Discourse connectives should correctly lead another event. Example: "As a result" is leading event the content "person A and person B didn't get up early the next day", rather than anything else.

Don't generate discourse relation table when there is no discourse relations.

Don't output the following connectives: simple conjunction (eg, and, in addition); specification (eg, especially, in particular); restatement (eg, in other words); instantiation (eg, for example, for instance); generalisation (eg, in summary, overall).

Don't output extra explanations.

Figure 1: The prompt for atomic fact and discourse relation extraction.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Strict</th>
    <th colspan="3">LCS</th>
    <th colspan="3">SBERT Elem</th>
    <th colspan="3">SBERT Full</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>77.21</td><td>69.17</td><td>72.97</td>
    <td>81.45</td><td>72.97</td><td>76.98</td>
    <td>86.20</td><td>77.22</td><td>81.46</td>
    <td>90.15</td><td>80.76</td><td>85.20</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0</td>
    <td>69.45</td><td>53.06</td><td>60.16</td>
    <td>75.05</td><td>57.33</td><td>65.00</td>
    <td>49.85</td><td>38.08</td><td>43.18</td>
    <td>89.29</td><td>68.21</td><td>77.34</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0.1</td>
    <td>69.11</td><td>53.75</td><td>60.47</td>
    <td>75.41</td><td>58.65</td><td>65.99</td>
    <td>80.89</td><td>62.91</td><td>70.78</td>
    <td>88.71</td><td>69.00</td><td>77.63</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0.2</td>
    <td>66.49</td><td>51.25</td><td>57.88</td>
    <td>72.66</td><td>56.01</td><td>63.26</td>
    <td>79.23</td><td>61.07</td><td>68.97</td>
    <td>90.35</td><td>69.65</td><td>78.66</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0.3</td>
    <td>68.47</td><td>52.78</td><td>59.61</td>
    <td>73.84</td><td>56.92</td><td>64.29</td>
    <td>79.36</td><td>61.17</td><td>69.09</td>
    <td>88.76</td><td>68.42</td><td>77.27</td>
  </tr>
  <tr>
    <td>GPT4o t=0</td>
    <td>70.00</td><td>53.47</td><td>60.63</td>
    <td>75.87</td><td>57.96</td><td>65.71</td>
    <td>81.32</td><td>62.12</td><td>70.44</td>
    <td>91.00</td><td>69.52</td><td>78.82</td>
  </tr>
  <tr>
    <td>GPT4o t=0.1</td>
    <td>68.99</td><td>52.22</td><td>59.45</td>
    <td>75.22</td><td>56.94</td><td>64.81</td>
    <td>81.41</td><td>61.62</td><td>70.15</td>
    <td>88.57</td><td>67.04</td><td>76.32</td>
  </tr>
  <tr>
    <td>GPT4o t=0.2</td>
    <td>69.64</td><td>53.19</td><td>60.31</td>
    <td>75.87</td><td>57.96</td><td>65.71</td>
    <td>82.13</td><td>62.74</td><td>71.14</td>
    <td>89.11</td><td>68.07</td><td>77.19</td>
  </tr>
  <tr>
    <td>GPT4o t=0.3</td>
    <td>68.54</td><td>49.03</td><td>57.17</td>
    <td>75.89</td><td>54.28</td><td>63.29</td>
    <td>82.68</td><td>59.14</td><td>68.95</td>
    <td>92.61</td><td>66.24</td><td>77.24</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0</td>
    <td>49.61</td><td>43.75</td><td>46.49</td>
    <td>57.72</td><td>50.91</td><td>54.10</td>
    <td>65.99</td><td>58.20</td><td>61.85</td>
    <td>75.05</td><td>66.19</td><td>70.34</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0.1</td>
    <td>50.81</td><td>43.75</td><td>47.01</td>
    <td>58.79</td><td>50.62</td><td>54.40</td>
    <td>67.60</td><td>58.21</td><td>62.56</td>
    <td>77.48</td><td>66.72</td><td>71.69</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0.2</td>
    <td>52.94</td><td>25.00</td><td>33.96</td>
    <td>59.44</td><td>28.07</td><td>38.13</td>
    <td>68.56</td><td>32.37</td><td>43.98</td>
    <td>73.34</td><td>34.63</td><td>47.05</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0.3</td>
    <td>49.54</td><td>22.36</td><td>30.81</td>
    <td>57.32</td><td>25.87</td><td>35.65</td>
    <td>68.79</td><td>31.05</td><td>42.79</td>
    <td>76.40</td><td>34.49</td><td>47.52</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0</td>
    <td>57.80</td><td>50.97</td><td>54.17</td>
    <td>63.41</td><td>55.92</td><td>59.43</td>
    <td>70.21</td><td>61.92</td><td>65.81</td>
    <td>82.71</td><td>72.94</td><td>77.52</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0.1</td>
    <td>55.85</td><td>50.42</td><td>52.99</td>
    <td>62.07</td><td>56.03</td><td>58.90</td>
    <td>69.35</td><td>62.61</td><td>65.81</td>
    <td>80.24</td><td>72.44</td><td>76.14</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0.2</td>
    <td>58.14</td><td>52.08</td><td>54.95</td>
    <td>63.86</td><td>57.20</td><td>60.35</td>
    <td>71.03</td><td>63.63</td><td>67.13</td>
    <td>83.59</td><td>74.88</td><td>78.99</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0.3</td>
    <td>57.08</td><td>51.53</td><td>54.16</td>
    <td>63.17</td><td>57.03</td><td>59.94</td>
    <td>69.59</td><td>62.83</td><td>66.04</td>
    <td>80.49</td><td>72.67</td><td>76.38</td>
  </tr>
</table>

Table 13: Quality of atomic fact extraction on annotated samples.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Strict</th>
    <th colspan="3">LCS</th>
    <th colspan="3">SBERT Elem</th>
    <th colspan="3">SBERT Full</th>
  </tr>
  <tr>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
    <th>P</th><th>R</th><th>F1</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>81.94</td><td>81.94</td><td>81.94</td>
    <td>84.93</td><td>84.93</td><td>84.93</td>
    <td>88.10</td><td>88.10</td><td>88.10</td>
    <td>90.32</td><td>90.32</td><td>90.32</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0</td>
    <td>74.67</td><td>82.96</td><td>78.60</td>
    <td>78.88</td><td>87.64</td><td>83.03</td>
    <td>83.06</td><td>92.29</td><td>87.43</td>
    <td>85.30</td><td>94.78</td><td>89.79</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0.1</td>
    <td>73.61</td><td>78.52</td><td>75.99</td>
    <td>78.17</td><td>83.38</td><td>80.69</td>
    <td>82.70</td><td>88.21</td><td>85.37</td>
    <td>84.37</td><td>89.99</td><td>87.09</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0.2</td>
    <td>66.67</td><td>71.32</td><td>68.91</td>
    <td>70.74</td><td>75.68</td><td>73.13</td>
    <td>74.50</td><td>79.70</td><td>77.01</td>
    <td>75.05</td><td>80.29</td><td>77.58</td>
  </tr>
  <tr>
    <td>GPT4-Turbo t=0.3</td>
    <td>75.00</td><td>76.60</td><td>75.79</td>
    <td>77.45</td><td>79.09</td><td>78.26</td>
    <td>79.33</td><td>81.02</td><td>80.17</td>
    <td>80.26</td><td>81.97</td><td>81.11</td>
  </tr>
  <tr>
    <td>GPT4o t=0</td>
    <td>61.81</td><td>68.99</td><td>65.20</td>
    <td>70.02</td><td>78.16</td><td>73.87</td>
    <td>77.62</td><td>86.65</td><td>81.89</td>
    <td>83.38</td><td>93.07</td><td>87.96</td>
  </tr>
  <tr>
    <td>GPT4o t=0.1</td>
    <td>60.28</td><td>67.46</td><td>63.67</td>
    <td>68.72</td><td>76.90</td><td>72.58</td>
    <td>77.50</td><td>86.72</td><td>81.85</td>
    <td>84.84</td><td>94.94</td><td>89.61</td>
  </tr>
  <tr>
    <td>GPT4o t=0.2</td>
    <td>59.86</td><td>71.54</td><td>65.19</td>
    <td>67.32</td><td>80.46</td><td>73.31</td>
    <td>74.75</td><td>89.33</td><td>81.39</td>
    <td>79.92</td><td>95.51</td><td>87.02</td>
  </tr>
  <tr>
    <td>GPT4o t=0.3</td>
    <td>61.81</td><td>70.63</td><td>65.93</td>
    <td>71.60</td><td>81.82</td><td>76.37</td>
    <td>79.83</td><td>91.23</td><td>85.15</td>
    <td>86.06</td><td>98.35</td><td>91.80</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0</td>
    <td>91.92</td><td>59.48</td><td>72.22</td>
    <td>93.78</td><td>60.68</td><td>73.69</td>
    <td>95.37</td><td>61.71</td><td>74.93</td>
    <td>96.43</td><td>62.40</td><td>75.77</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0.1</td>
    <td>94.79</td><td>59.48</td><td>73.09</td>
    <td>96.23</td><td>60.38</td><td>74.20</td>
    <td>97.91</td><td>61.43</td><td>75.49</td>
    <td>99.40</td><td>62.37</td><td>76.60</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0.2</td>
    <td>93.94</td><td>60.78</td><td>73.81</td>
    <td>95.58</td><td>61.85</td><td>75.10</td>
    <td>97.87</td><td>63.33</td><td>76.90</td>
    <td>99.34</td><td>64.28</td><td>78.05</td>
  </tr>
  <tr>
    <td>LLAMA3 8B t=0.3</td>
    <td>92.93</td><td>60.13</td><td>73.02</td>
    <td>94.41</td><td>61.09</td><td>74.18</td>
    <td>95.56</td><td>61.83</td><td>75.08</td>
    <td>96.71</td><td>62.58</td><td>75.99</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0</td>
    <td>45.65</td><td>60.00</td><td>51.85</td>
    <td>53.14</td><td>69.84</td><td>60.35</td>
    <td>60.54</td><td>79.56</td><td>68.76</td>
    <td>63.44</td><td>83.38</td><td>72.06</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0.1</td>
    <td>47.83</td><td>59.46</td><td>53.01</td>
    <td>56.00</td><td>69.63</td><td>62.08</td>
    <td>63.57</td><td>79.03</td><td>70.46</td>
    <td>67.37</td><td>83.75</td><td>74.67</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0.2</td>
    <td>48.89</td><td>57.89</td><td>53.01</td>
    <td>55.73</td><td>65.99</td><td>60.43</td>
    <td>62.32</td><td>73.80</td><td>67.57</td>
    <td>68.35</td><td>80.94</td><td>74.12</td>
  </tr>
  <tr>
    <td>LLAMA3 70B t=0.3</td>
    <td>59.85</td><td>64.23</td><td>61.96</td>
    <td>66.57</td><td>71.44</td><td>68.92</td>
    <td>72.08</td><td>77.36</td><td>74.63</td>
    <td>75.50</td><td>81.02</td><td>78.16</td>
  </tr>
</table>

Table 14: Quality of discourse extraction on annotated samples.

Reviewer paper summaries: This paper proposes a method for verifying factual consistency between texts through structured discourse representation. The authors introduce an approach to extract structured facts and discourse relations between facts, leveraging this information to compare texts. The study seeks to address the challenge of capturing and comparing factual relations, a task purportedly difficult for state-of-the-art language models. The authors claim to advance research in sentence similarity by incorporating linguistic grounding into distinct, explainable modules. Two downstream tasks, text generation and summarization, are used to demonstrate the method's potential.

This work presents a novel approach for computing cross-textual factual overlap by introducing a new method for extracting atomic facts from a given text. The proposed method leverages both structural representations and discourse relations to enhance the accuracy and reliability of fact extraction. A specialized entailment model, FDSpotter, is trained to assess how the atomic facts/discourse relations existed in the providing texts. Experimental results showed that the proposed FactOverlap metrics achieved comparable or slightly better performance on multiple factuality evaluation benchmarks.
Summary of strengths: 1. The topic of factual consistency in generated language is crucial and timely, particularly with the growing deployment of LLMs.
2. The method attempts to enhance the sentence-level analysis by incorporating discourse relations, which is an interesting angle to explore.
3. The paper provides empirical evaluations on multiple datasets, aiming to validate proposed methods in practical scenarios.
4. Introducing a new dataset, DiscInfer, is a valuable contribution that could potentially support future research in discourse relations.
5. The paper addresses an important and timely problem in NLP concerning the factual consistency of texts and language model outputs.
6. The proposed method attempts to integrate structured sentence representation with discourse relation analysis, offering a novel angle on sentence comparison.
7. The work includes an interesting experimental evaluation on downstream NLP tasks, which avoids purely theoretical discussion.

- The paper proposes a new approach to extract atomic facts that cover more content and leverage discourse relation matching. 
- The authors released the DiscInfer dataset, which could be beneficial for other works studying the discourse relation extraction and entailment prediction tasks. 
- The proposed approach obtains better performance on WebNLG2020 and comparable on summarization benchmarks
Summary of weaknesses: 1. The methodology for extracting and validating structured facts relies heavily on current LLMs, yet the paper claims to outperform these models without adequately demonstrating how this is achieved. The dependence on the same technology it aims to surpass is both circular and insufficiently justified.
2. The proposed FactOverlap metric is not convincingly better than existing measures. The paper lacks a thorough comparison with state-of-the-art metrics on a wide range of benchmarks. The presented differences in performance are not statistically significant.
3. The newly introduced dataset (DiscInfer) is inadequately described, lacking sufficient details regarding its construction, annotation processes, and coverage. There's no tangible evidence that DiscInfer effectively benchmarks the problem of discourse relation entailment in substantial ways.
4. The explanation of technical components, such as the intrinsic and extrinsic confidence scores, is not aligned with clear algorithmic contributions or insights into the limitations of current methods.
5. The paper's approach to discourse relation entailment is inadequately validated. The empirical results do not convincingly demonstrate that introducing discourse relations substantially enhances performance or insights into factual consistency tasks.
6. There is an over-reliance on generated datasets or synthetically perturbed examples for training and evaluation, which may not adequately reflect real-world complexities.

- It is little discussed about the overheads of running the inference models compared to the LLM approach or prior baselines. 
- The robustness of the proposed model in handling long input is less verified. WebNLG2020 texts seem to be pretty short, and the summarization benchmarks are rather old and mainly based on news articles. Since the FO score is computed bidirectionally, extracting atomic facts from the long source documents in the scenario of long document summarization tasks is necessary. You may want to consider other benchmarks, such as DiverSUMM [1], SummEdit [2], and other scientific-article-focused corpora. 
- As admitted by the authors, there lacks an analysis of the errors/evaluations to account for the conclusions that the proposed approach offers interpretable and fine-grained assessment. Those are needed to make the contribution convincing. Additionally, 
- The appendices seem to focus on presenting the results of different detection models. I would say more efforts should be put into examining how those detection models help with the down-stream factuality evaluation tasks: 

*References:*
- [1] Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. 2024. Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1701–1722, St. Julian’s, Malta. Association for Computational Linguistics.
 
- [2] Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9662–9676, Singapore. Association for Computational Linguistics.
Comments, suggestions and typos: Nan

- On SummaC evaluation, why is there such a huge difference between system-level and sample-level correlations?
- You may consider removing some of the detection model training/benchmark details from the appendix and adding back the analysis mentioned in the limitation section.
Overall assessment scores: 2.5

3