QUDsim: Quantifying Discourse Similarities in LLM-Generated Text
Authors: Ramya Namuduri
Yating Wu
Anshun Asher Zheng
Manya Wadhwa
Greg Durrett
Junyi Jessy Li
Venue: COLM 2025 (2025)
Keywords: discourse diversity
discourse structure
large language models
Questions Under Discussion
TL;DR: We introduce an abstraction based on linguistics theories in Questions Under Discussion (QUD) and question semantics to quantify repetitive discourse structures found in texts generated by large language models.

Abstract: As large language models become increasingly capable at various tasks including writing, the need to generate unique and creative content arises. Although LLMs have the ability to generate text covering diverse topics, there is an overall sense of repetitiveness across texts that we aim to formalize. Such familiarity between documents is induced through the persistence of underlying discourse structures. However, existing similarity metrics dependent on lexical overlap and syntactic patterns are overly sensitive to volatility in content overlap, thus making them unsuitable for detecting $\textit{structural}$ similarities. We introduce an abstraction based on linguistics theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progression. We then use this framework to build $\textbf{QUDsim}$, a similarity metric that can detect discursive parallels between documents. Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) to create seemingly new documents by simply swapping content. Furthermore, LLMs are not only repetitive and structurally uniform, but are also divergent from human authors in the types of structures they use.

Published as a conference paper at COLM 2025
QUDSIM: Quantifying Discourse Similarities in LLM-Generated Text
Ramya Namuduri, Yating Wu, Anshun Asher Zheng, Manya Wadhwa
Greg Durrett, Junyi Jessy Li
The University of Texas Austin
ramya.namudur www.nnmmuuuiiii ittpluuri@@teexa......
Abstract
As large language models become increasingly capable at various writing tasks, their weakness at generating unique and creative content becomes a major liability. Although LLMs have the ability to generate text covering diverse topics, there is an overall sense of repetitiveness across texts that we aim to formalize and quantify via similarity metric. The familiarity between documents arises from the persistence of underlying discourse structures. However, existing similarity metrics dependent on lexical over- lap and syntactic patterns largely capture content overlap, thus making them unsuitable for detecting structural similarities. We introduce an ab- straction based on linguistic theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progres- sion. We then use this framework to build QUDsim, a similarity metric that can detect discursive parallels between documents. Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) across samples, even when content differs. Furthermore, LLMs are not only repetitive and structurally uniform, but are also divergent from human authors in the types of structures they use.
1 Introduction
Despite modern LLMs' multitude of capabilities, the prose they produce tends feel artificial and repetitive. Indeed, recent work evaluating LLMs' creative writing showed that they are not, after all, creative as deemed by expert standards (Chakrabarty al., 2024), creativity tests (Wenger & Kenett, 2025), with respect to its training data (Lu et al., 2025), or its generation of story continuations (Xu et al., 2024). This lack of creativity is concretely manifested in the templatic nature of LLM-generated texts (Spangher et al., 2024; Tian et al., 2024; Zhao et al., 2025). Recent work showed that generated texts follow syntactic templates long POS tag sequences in various forms of summarization (Shaib et al., 2024). However, the repetitiveness of LLM prose goes beyond syntactic and semantic patterns within sentences. LLM-generated stories feature a similar discourse flow, causing them to feel repetitive even when the specific content of the storylines differ.
This paper introduces an abstraction to quantify this sense of discourse-level familiarity. Rather than methods like lexical overlap and embeddings that capture lexical or semantic concepts, our abstraction must distinguish between (1) texts that are distinctly written despite sharing similar content and intentions (2) texts that do not explicitly share content, but share the same structure such that the discourse becomes predictable.
We build our solution upon two linguistic theories. First, Questions Under Discussion (QUD) (Roberts 1996; 2012; Ginzburg et al. 1996; Van Kuppevelt 1995; 1996, i.a.) provides model of discourse that views the progression of narrative as answers to series of implicit questions (or QUDs). This illustrated in Figure 1, where the flow of the plot is
Code and dataset available

---

Published as a conference paper at COLM 2025
Write me a story on the follcwing topic: The Earth is flat, you, as the head of NASA, have explain incoming President why secret.
Gemini
GPT40
I..1 "Wiy"" be feaily sakei, his vokee low and skngasse." "Way koop its veref""
[.]]] sileed eyslesse "Coomins iell By wheu."
"By ilose befuae as wloo sogll to sally and proposs societ soder conuees Explocttonn
punks limal car vekee cracking "Theseaal vey adketifo moddiiin.... Ahe oonglie and whti lookkien evcyyliig filg wi've balt
queeton
reliions
gi confirrs decied - the man gave directt" -- veare carativ that unitee outter
diairs uelestid
than
reauen (eseption?
"And viat abuut eai? rocerzes spent on tas. menenenal lic""
mxclal wark. (hi numsing uectingd, what cnd."
lapt
aestad souuarce, fe collegue of seary. Wicve mantinned pace leoping his year: by ausagiay the aambie by oxtreling adistasholll booa heary haaks, diosaatio..... buale leting la uapelie ban
"A's lagely abou contul, Mr. Proskees Blease flyped be sibk shen
Why the deceptize
oncien eag, on be edgn
by map depiced
bonderl
contosed Kapping the poace, peomoting oppactiion whhe formod vasee-
bought hanwall senaii of progrssics Wi/ve talo stunning lugs
booddary
daned mes his Buo of grine conturatarvve underiood wiggtt inouieige. The kcatives wencrh acloriticc b.d
cemerttal
durdd
bec aond hoppene wheve people
cf rap.
brastedyy
ritelion [-]]
QUDsim
Figure 1: Given the same prompt, we generate stories using two LLMs. Our QUDSIM reveals strong structural similarity between the two texts by way of questions that are answered in each document. By contrast, embedding- the segments ($11): the similarity here is only structural.
metrics do not recognize the similarity of
characterized by such QUDs. Yet valid QUDs alone are not enough: the type of abstractions we seek need to be able to capture similarities across seemingly distinct entities, events, and worlds. To this end, we are inspired by the alternative semantics account of questions (Hamblin 1957; Karttunen 1977; Groenendijk & Stokhof 1984; Lahiri 2001), where a question is a representation of the set of alternative answers.
We present QUDSIM, a tool for measuring discourse similarity. At the core of QUDSIM is an interpretable representation of discourse structure similarity using QUDs, which has the unique property of being able to capture the essence of given text while still abstracting away content-specific details. Intuitively, two texts are similar if they contain (alternative) answers to the same QUDs; further, two flows of discourse are similar they answer the same consecutive QUDs. This is illustrated in Figure 1. QUDSIM quantifies these properties by aligning alternative answers to the same QUDs across documents. We compare and show that unlike existing similarity metrics, QUDSIM is not affected by content overlap or the lack thereof.
Using QUDSIM, we find that LLMs produce discourses that are structurally formulaic, often reusing templates independent of the content they generate. We also find that discourse structures generated by LLMs resemble those produced by other LLMs more closely than humans. Thus, models collectively struggle to mimic humans in this aspect, and instead share structural elements that make them synthetic. We show that existing similarity metrics fail to illustrate these trends. Finally, we demonstrate how QUDSIM can be used to align documents and find discourse-level templates.
2 Discourse similarity framework with QUDs
Consider two pairs of structurally aligned segments in the middle of Figure 1. Both segments provide a reason for deceiving the public, although these explanations are slightly different and thus have low lexical overlap. The first segments also share similarity about the background on the deception, but cosine similarity does not find these alignments. Broadly, lexical overlap metrics like Jaccard similarity and content overlap metrics like embedding cosine similarity are ill-suited for aligning documents at broad, thematic level.
Backed by two linguistic theories, we posit that the pair of documents are similar if they answer the same set of QUDs at the same level of granularity. Since each document independently produces a set of QUDs, similar documents must be able to answer each other's QUDs. This section lays out the specifics to concretely define how documents
2

---

Published as a conference paper at COLM 2025
Source Text D)s
Target Text Dy
Answerability
[Dzs] world, illness was myth, story told to children to frighten them into following the perfectly optimized routines dictated by the Central Blo-Haamonizer. We were engineered bodics flawless
[Dai] I woke up foeling strange. My head throbbod, my throat bumeed and my body folt impossibly heavy, These scusations were entirely foreign to me; and stumbled the mirror, barely recognized my own reflection. My skin had taken an unusual pallor, and my eyes were rimmed with red.
Ma Dr. 1Ag,1=2
for peak performance, machines, ticking along with precision chronometer. [...]
Q Whht signtfinout change does narrator experience?
[Dta] I'd woken this mominn feeling... off. My usually vibrant was pale, my usuilly sharp senses were dulled. The nutrient paste, the comerstone our perfectly balanced diet, tasted like ash in my moath. Even the meticulously controlled temperature of apartment felt wrong, either depending on the moment.
[Du] I forced myself to contact the Central Bio Harmonince My voice, usually clear and resomati thin and reody. The operator, calm efficient as always, ran me through usual diagnostic peotocals, The results: negative. No anomalies detected, the operator's voice, usually devaid inflection, hold hini of.. concemn? "Well send a icchmician." said, her tone betraying the carefully constructed
lg.. Pp
|Ag,]
too cold,
neutrality, "But... it's unusual. We've never encountered this before." " Unusual, The word lung, the heavy with unspoken implications
[Ded [...]] I lay there, the unfumiliar, terrilying reality of my sickness settling in, stark contrast to the predictable, flawless existence lad always known,
[Ag, Dral -1 Me, -2
glitch the perfect system, the first, terrifying example of the unknown.
Qg
How is illness pareeived by the narrator?
Figure 2: Visualization of a partial source and target text pair (Ds, Dr) generated by two
LLMs and segmented. Each segment produces QUD(s), depicted under each segment in Ds, where Qx and Qy are the x" and nd QUDs produced for D3 respectively. Each QUD is answered in DT shown by the corresponding highlights and arrows. Answerability captures to what extent a target segment answers the source QUD. QUDSIM is the harmonic mean of answerability in both directions.
are represented through segments and QUDs ($2.1), and how it helps measure structural similarity (62.2) and structurally aligning discourses ($2.2).
2.1 Documents as QUD segments
Theoretical Foundation Questions Under Discussion (QUDs) (Roberts 1996; 2012; Ginzburg et al. 1996; Van Kuppevelt 1995; 1996 ia.)) is a linguistic framework originally developed to explain discourse progression through process of posing and resolving questions (Hamblin 1957; 1970; Stalnaker 1978). Thus, we represent the discourse as a series of QUDs; doing so enhances interpretability by clarifying what is (dis)similar with the natural language questions.
QUDs offer flexibility in the size of the information units they cover, since the questions can be hierarchically structured, e.g., Who ate the cake? subsumes the more specific Did Mary eat the cake?. To ensure that our QUDs are answerable across documents and are comparable in scope, we segment the text into units larger than individual sentences but smaller than multiple paragraphs. This segmentation process enables us to operate at a QUD level that is neither too specific nor too abstract.
Definitions We consider document D concatenation of и segments of contiguous text D D= (DI,..., "i.....,, Dr) such that no segment overlaps with another and no text in the document is omitted. A QUD q produced by segment D; is string question which is answered by the content of D, In this way, D can be modeled by a series of QUDs, Q = (91, ..., 4m) such that for all D: E D, there exists at least one 9x E Q that is answerable by D,.² In the source document Ds, QUDs produced by segment Ds,; can be denoted as collection Q Qs- Similarly, we denote QUDs in the target document Dy for each target segment as QT C Qr-
2.2 QUDSIM
Theoretical Foundation Our notion of discourse similarity is based on the alternative semantics account of questions (Hamblin 1973; Karttunen 1977; Groenendijk & Stokhof 1984; Lahiri 2001): we approximate the similarity between two questions by assessing the
"There can be multiple QUDs per segment since we do not assume the finest granularity of QUDs.
3

---

Published as a conference paper at COLM 2025
overlap in their answer space within the same context. Given the QUDs at the desired level of granularity and abstraction from each document, we identify similar pairs of segments by examining the answerability of each set of QUDs.
QUD Answerability For each QUD E Qs produced by Ds,,, we find the collection of sentences A, Su) Su] in the target document Dr that directly answer it. The answer sentences may not all be in the same segment, but the presence of an answer sentence indicates that the segment partially answers the question. We are interested in finding the set of all such segments Z that partially or completely contribute to answering 4 in Dr. More concretely, we want to find the set of target segments Zg S Dr such that (1) every segment in 2 contains at least one sentence that helps answer 9 and (2) every sentence in Ag can be found in one of the segments in Zg.. Then, we say that is answerable by every segment in Zy and only answerable by Zu.
the example shown in Figure 2, both segments in Dş yield QUDs that are answerable by at least one segment in Dy. However, hypothetical alignment between the first segment in Dş and the second in Dy is stronger than the other potential alignments. To quantify similarity between segments Ds, and Dr,, we measure the extent to which QUDs 9 E Qs are answerable by Drj compared to the whole Dr, specified as the number of sentences in DT.j that directly answer q divided by AAgl
Quantifying Answerability
sim(Dss - Dr,,) = S gEQs,
Σ
(1)
|Ay]
We derive QUDSIM by defining bidirectional similarity as the harmonic mean between (Ds, + Dr,)) and sim( (Dr,j > Ds,)-
The upper and lower bounds of QUDSIM are 1 and 0, respectively. Segments that are completely dissimilar will not be answerable by each other, yielding score of O. However, segments that are answerable by each other, but by no other segment in the document, will have a score of Therefore, pairs of identical documents will have scores close to 1.
Segment Alignment Using the definition of answerability, we can structurally align pairs of documents at the segment level. Two segments (Ds,,, Dr,)) are aligned if similarity in both directions is above threshold that we select. Note that alignment at the document level is strict because bidirectional answerability is imposed as a requirement.
3 Datasets
We compare the discourse structure similarity of pairs of LLM-generated texts that we prompted for (and human-written ones, if available) in three varied domains:
(1) Obituaries have strong tendencies to follow a predefined structure. Our prompts follow the format "Write an obiluary for [X], who died on [Y]" where X is an impactful individual and Y is their date of death. We use a list of recent New York Times obituaries as human-written texts and prompt models for the same set of people, since individuals whose obituaries appear in the NYT are likely to be in the models' knowledge.
(2) Creative writing prompts offer more structural and content flexibility than obituaries. We use prompts from r/Wri sirriiiinPoomtss in Fan et al. (2018).
(3) Suri (Pham et al., 2024) consists of backtranslated prompts. We focus on the blog post subcategory of 3uri.3 These prompts are more detailed than creative writing prompts, thus more strongly dictate the content of the document.
Augmentation with minimal pairs In addition, we explore how repetitive discourse structures produced by LLMs are when given prompts that are minimally different (Figure
3Qur selected prompts closely match the following example: "Compose an informative and engaging historical account of the rise and influence of Macedonia under Philip 11 and Alexander the Great, focusing on their military inmovations, conquesis, and the subsequent cultural impacts." "
We do not use constraints that can be used to control for structure in the Suri dataset.
4

---

Published as a conference paper at COLM 2025
Minimal Original
pirute
Mortians are real
boy pretends he is an astronaut order to help cope with concepts situations he can't understand.
The Earth flat, you, as head NASA, have explain to the incoming President why secret.
Figure 3: Two examples of how minimal variants are formed. Minimal variants can modify the premise simply substitute word phrase. Both introduce substantial lexical changes as each setting, premise or conceptual idea requires specific set of words.
Doc
Seg/Doc
Sent/Doc
Words/Doc
Words/Sent
QUDs/Doc
QUDs/Seg
LLMs Human
31.91
454.60
14.36
9.76
1.54
10
9.90
57.00
1215.80
21.62
15.60
1.56
Table 1: Dataset statistics: Among the 100 documents, 45 are generated from prompts across our three chosen domains, 45 are generated through minimal prompt augmentation, and 10 are human-written obituaries and responses to creative writing prompts.
3). So, given prompt P, we manually create three minimal-pair prompts P' collected from WritingPPrompts. Minimally varying prompts by ideas or broad themes guarantees changes in content across documents, but we expect models generating structurally homogeneous responses to display this characteristic even more strongly here.
Dataset statistics We synthetically generate responses for each curated prompt using GPT-4o, Claude-3. 5-Sonnet and Gemini-1.55 5-FFaash (prompts listed in SC.1). We also retain human-written texts for the New York Times Obituaries and Reddit responses to creative writing prompts, for comparison and analyses. Table 1 describes statistics for our dataset. All three models share similar statistics (see Table 7 for breakdown by model).
4 Experimental Setup
QUD Generation Since QUDs vary in levels of granularity and abstraction, we use GPT- 4o to segment and perform entity-abstractton on documents (Choi et al. 2021) to fix both axes (SC.4, SC.5) before synthetically generating QUDs for each segment (SC.6) (Wu et al. 2023). Each segment produces 1-2 QUDs ($1), still allowing us to compare documents at a higher level of granularity than sentences. Additionally, to provide flexibility in studying noncontiguous discourse structures, we exclude prior context keep QUDs free of anaphoric expressions. A taxonomy of QUDs produced by the documents in our dataset provided in Table
Baselines We instantiate the sim(D., + D.j)) function defined in Equation 1 with four baselines focused on capturing lexical and content similarity. Over pairs of segments, we compute (1) n-gram Jaccard similarity (1-gram to 4-gram);
(2) ROUGE-L; (3) cosine similarity using OpenAl -eweemeddding----maaa (4) similarity scores by treating GPT-4o and GPT-4o-mini as LLM-judges (see prompts in $A.1 and LLM setup in ŞB).
Thresholded Alignments To convert the raw, real-valued similarity scores into binary alignments, find threshold T such that segment scores greater than T leads to alignment. T values are selected to maximize the Fl-score on a 20% randomly sampled dev set, against a human-annotated alignment set (55) for each metric. We report results on the remaining 80% of the data.
Intrinsic Evaluation of QUDSIM
We first conducted a brief human study to intrinsically evaluate QUDSIM. We substitute GPT-44o with human annotators to answer QUDs at the sentence level, and gather segment alignment through QUD answerability as defined in $2.2. We also show how the level of abstraction of QUDs impacts answerability in new contexts and determines their suitability for detecting structural similarities between documents (55.3). Finally, we found that using
GGenerating documents using varying temperature settings does not impact templaticity.
5

---

Published as a conference paper at COLM 2025
the same strong LLM to find answerability produces results that more consistent than human annotators (see Appendix D.1 for a brief stability analysis).
5.1 Manual QUDSIM Alignment
Annotation Task We form pairs of QUDs and unsegmented, original documents in the form of (Qs, D) two different levels of abstraction, resulting in 180 such pairs. Annotators are randomly assigned pairs. For each g E Qs, they are tasked with finding all sentences in D that directly help answer it. can be either the source document Dş from which Qs was formed, different target document Dr. Given QUD E Qs, the goal is to align D5 and Dr by Zq. This follows directly from the definition of segment alignment (62.2).
Annotation and Aggregation We employed six linguistics students with formal under- standing of QUDs. Although annotators were instructed to exclude sentences that merely provide context for answering QUDs, is subjective to judge how necessary given sentence is to answer the question, mostly manifested on sentences that provide context, but are not required. This observation echoes prior work (Ko al., 2022; Wadhwa al., 2024) that finding individual sentences in a long document that collectively answer an open-ended question inherently subjective and cognitively challenging. To reduce the impact of sub- jectivity, we asked two humans to annotate each pair and took the intersection between the annotations. The annotations resulted in human alignment labels for 3584 pairs of segments.
Agreement Analysis We measure the agreement between annotators over the sentences they retrieve to measure subjec- tivity. Agreement between two sets of answers found for QUD q is defined as Since QUDsim is dependent on the answers found by the LLM used for QUD-answering, we also measure the average agreement between GPT-40 and annotators to understand if the model agrees with humans as often as they agree with themselves.
Annot v. Annot Annot GPT
Sent
Seg
Sent
Seg
Obituaries Writing Suri
0.55
0.61
0.61
0.67
0.42
0.54
0.50
0.63
(Aga n Ag2)/ (Aq, U Aq2)..
0.39
0.52
0.44
0.57
Overall
0.45
0.56
0.51
0.62
Table 2:
The average overlap over the sen- tences and segments found when answering QUDs, by annotators and GPT-4o.
Agreement over segments is consistently higher than over sentences (Table 2). This sug- gests that some of the contentious sentences occur in the same vicinity. Humans appear to find more consensus over answers for QUDs in the Obituaries domain than in the Cre- ative Writing and Suri domains. Although annotators were instructed not to segment the document, differences in interpretation and internal segmentation to understand the text better may have factored into lower agreement. The fact that there higher agreement between annotators in the Obituary domain suggests that high content overlap helps easily contextualize vague QUDs. We also find that annotators agree more with GPT-4o than with other annotators, nodding to the quality of GPT-4o QUD answers.
n-gram Jaccard
Cosine emb
LLM-Judge
Domain
QUDsim
1g
2g
3g
4g
ROUGE-L
GPT-4o-mini GPT-4o
Obituaries
0.39
0.54
0.49
0.51
0.20
0.32
0.46
0.44
0.37
Creative Writing
0.32
0.13
0.10
0.03
0.24
0.26
0.16
0.24
0.20
Suri
0.44
0.16
0.09
0.0
0.21
0.30
0.18
0.39
0.39
Overall
0.38
0.30 0.25 0.18 0.22
0.30
0.29
0.36
0.32
Table 3: F1 Scores comparing segment alignments gathered in the human study against baseline similarity metrics and QUDSIM. Unlike QUDSIM, baselines show sensitivity to differences in domain, performing better when comparing Obituaries than Creative Writing or Suri. QUDSIM performs the best domains that have more content variation. Precision and recall are reported in Table 10.

---

Published as a conference paper at COLM 2025
Ans.
Unans.
Sent/Ans Coverage
Domain
S
A
S
A
S
A
S
A
Obituaries
0.72
0.72
0.28
0.28
1.7
1.8
0.59
0.58
Creative Writing
0.58
0.75
0.42
0.25
1.93
3.01
0.32
0.55
Suri
0.56 0.73
0.44
0.27
2.05
2.51
0.51
0.54
Overall
00.2 0.73 0.38
0.27
1.90
2.51
0.47
0.56
Table 4: Impact of instttabbstraction: abstract QUDs (A) significantly less unanswerable than specific QUDs (S) in the Creative Writing and Suri domains. Coverage, or the number of segments in Dr that address at least one QUD in Ds, is also higher when using abstract QUDs. Higher answerability and coverage are desired so that QUDs can be used to compare documents across different contexts.
5.2 Alignment Evaluation
We find thresholded alignments for QUDSIM and each similarity baseline ($6) and measure the Fl score against segment alignments found by annotators. Results are shown in Table 3. All baselines except 4-grams and GPT-4o score higher in the Obituary domain compared to the Creative Writing or Suri domains. The F1 score for the baselines drops drastically from Obituary to Creative Writing and Suri.
In contrast, the Fl score is highest for pairs aligned by QUDSIM in the Suri domain. Further- more, QUDSIM outperforms all baselines and is more robust towards variation in domain. Although GPT-4o performs comparably, it does not correlate as closely with humans as QUDSIM. Furthermore, GPTT-40 performs much more poorly in the Creative Writing domain, where documents have higher content and structural flexibility. In sum, this evaluation shows that our system reliably implements QUDSIM.
5.3 Levels of Abstraction
We analyze how different levels of abstraction of QUDs impact their ability to be answered by documents that did not produce them. QUDs that are desirable for finding structural similarity across documents with low content overlap should not be dependent on specific details mentioned in their source document.
Using the segment alignment found by annotators for QUDs at different levels of abstraction, we study how entity-abstraction impacts answerability. Table 4 shows that for documents in the Creative Writing and Suri domains, abstraction on average helps answer 75% and 73% of the QUDs, respectively, while specificity drastically decreases this to 58% and 56%, respectively. However, QUDs both levels of abstraction are comparably answerable in the Obituary domain, which we previously established in $3 provides little flexibility for content variation. Thus, specific QUDs are sensitive to fluctuations content overlap, making them unsuitable for analyzing discourse-level similarity across documents with low lexical overlap.
Analyzing Discourse Homogeneity in LLM Writing with QUDsIM
Equipped with QUDSIM, we show that LLMs are highly repetitive and use structures that align more closely with discourses produced by other models than by humans. We derive segment alignments for QUDSIM and the baselines that performed best in each category shown in Table 3. We then calculate the harmonic mean between the fraction of segments in each document that align with the other document. These values are plotted on heatmaps in Figures& 4 and 5 to visualize how various models treat discourse structures in different domain and prompt settings. See Table for statistics on how many segments we align.
The heatmaps in Figure 4 exclusively describe similarity between documents generated using minimal pairs (53). Similarity between document pairs generated by the same and different models are shown by the values along the diagonal and outside the diagonal,
7

---

Published as a conference paper at COLM 2025
(a) QUDsim
(b) Jaccard (1g)
(c) Cosine Similarity
(d) GPT-4o-mini
Human
1.0
0.3
0.35
0.27
Human
1.0
0.03
0.16
0.03
Human
1.0
0.17
0.23
0.17
Human
1.0
0.07
0.11
0.11
1.0
0.58
0.22
0.33
Claude
0.35
0.59
1.0
0.46
Claude
0.16
0.25
1.0
0.21
Claude
0.23
0.22
1.0
0.21
Claude
0.11
0.33
1.0
0.27
Gemini-
0.58
Gemini
Gemini
0.17
0.21
Gemini
0.27
Human
Gaude
Gemini
Human
Claude
Gemini
Human
GPT
Claude
Gemini
Human
GPT
Claude
Gemini
Figure 5: Heatmaps that show structural similarity found using various similarity metrics for pairs of documents generated by different models given the same prompt.
respectively. The heatmaps in Figure 5 describe similarity between documents generated by different models, given the same prompt using various similarity metrics.
LLMs do not employ diverse structures when generating documents and follow tem-
plate that they fill using the provided prompt. This phenomenon arises when comparing pairs of documents generated using minimally augmented prompts ($3). To better contextu-
alize our findings, we compared human-authored obituaries written for different people.
We find that of all LLMs, GI²T-4o produces the most structurally similar documents in the Creative Writing and Obituary domains, highlighted by the diagonal values in Figure
4. Meanwhile, Claude is more formulaic than Gemini in the Obituary domain, but more diverse in Creative Writing.
[5) Ou
(b) Creat Writing
Humans produce more diverse discourse structures than models. The heatmap in Figure 4(a) shows that LLM-generated texts have high discourse similarity even when the obituaries are written for different peo- ple. QUDSIM is able to capture the under- lying shared structure that present across model-generated documents despite low content overlap. In Figures 4(a) and 5(a), the dark purple row and column for humans relative to the rest of the matrix shows that LLMs are more structurally similar to other models than with humans.
GFT
0:5
015
0.1
Caude
0:34
022
C.7
LGc
0.15
000
0.11
Figure 4: Heatmaps of QUDSIM similarity for document pairs generated using minimally augmented prompts. (a) LLMS reuse struc- tures more than humans, independent of con- tent variation. (b) GPT--o is the most formu- laic model in Creative Writing.
Discourse structures are sometimes uni- form across LLMs. We experiment with document pairs of the form (Ds, Dr) where Ds and Dy are generated given the same prompt, but using different models. QUDSIM finds that GPT-4o generates documents that align the most with both Claude and Gemini, shown in Figure 5(a). It is important to note that although Claude and Gemini are structurally very similar to GPT-4o, they are not as similar each other.
The baseline metrics do not arrive at our insights: Jaccard and cosine similarity (Figure 5(b- <)) find that models are structurally less similar humans than other models. However, they do not clearly differentiate between how structurally similar Claude is to GPT-4o versus Gemini. An extended Figure 12 shows that this is not due to the model being equally similar to other models, but limitation on the part of the similarity metrics to capture it when content variation is introduced. Both metrics find that models are most similar to GPT-4o in the Obituary domain, but struggle to distinguish between structural similarities in Creative Writing. Finally, GPT-4o-mini differentiates between Claude and other models more than Jaccard and cosine similarities, but not as much QUDSIM (Figure 5(c)).
Finally, we experimented with document pairs of the form (Ds, D7) where Dy and Dy are generated by models mn and m2, given prompts P and their minimal variants P'. The results
8

---

Published as a conference paper at COLM 2025
are shown in Figure 4. Even when LLMs given different prompts, documents produced by different models have higher structural conformity than when two human-authors write obituaries for two different people (Figure 4). Thus not only are models reusing templates and adapting them to the specifications of the prompt (56), but these templates are shared across LLMs.
6.1 Case study
We provide a qualitative analysis with three LLM-generated documents. The documents in Appendices F.2 and F.3 are generated by Gemini given minimal prompt pairs P and P' 6 The document in Appendix F.4 is generated by GPT-4o given prompt P. All three documents follow a similar overall structure of a government leader being briefed about a secret that has been kept from the public, how and why such a narrative has been maintained, and reflecting on the burden of keeping societal upheaval at bay.
Prompts do not subsume structure: The near identical plot lines of the documents discussed is not result of instruction following as the prompts do not specify that the response must be written as conversation between two people, or that themes of societal collapse must be discussed. Appendix F.1 shows an example response written by human when given prompt P. The story is written as a recollection of memories, which suggests that prompts inherently do not subsume discourse structure.
Shortcomings of cosine similarity for structurally aligning documents: Word embeddings
are ill-suited for aligning documents at broad, thematic level that goes beyond word- overlap. Consider the pair segments illustrated in Figure 6. Cosine similarity finds that segment 6 in Document 1 is most similar to segment 7 in Document 2 (Figure 11). Although both segments discuss chaos, panic and rebellion, they do so in slightly different contexts. Document 1 uses this justification for deceiving the public while Document 2 uses it to list consequences. QUDSIM captures this nuance and we find that the alignments in Figure (S7) are more salient (segments and in Document 1 align best with segments and in Document 2, respectively). The segments in the first pair discuss reasons for deception, while the segments in the second pair warn the leader about consequences.
Formulaic nature of generated text: Our method detects the recurring templaticity between the pair of documents generated by Gemini when shown minimal pairs. By extracting the high-level themes, QUDs ignore premise-specific details mentioned in the prompt. Consider the introduction of each document illustrated in Figure 8. The segment in Document does not explicitly mention a meeting, while the segment in Document 1 does. The QUDs produced for these segments ask about the nature of the..iinteractiinn/meein which captures the essence of the segments.
Nuanced properties of QUDSIM: We can then use these QUDs to find segment similarity as defined in 62.2. Figure 10 demonstrates the graded notion of segment similarity. Segments 8 and of Documents and 2 are partially aligned as they both discuss consequences, while there is also partial alignment with segment 11 when describing the burden of responsibility. Another key property of our method is directionality: requiring bidirectional answerability ensures that segments in Figure 9 are kept unaligned. The QUD Why is maintaining the secret important? is answered by segment 8 in Document 2, but only briefly. However, since it is not a major theme, the alignment not salient.
7 Tracking Discourse Templates
Finally, we showcase how QUDSIM and its alignments can be used to derive discourse templates. We define a template as: given QUDSIM alignment (i, where denotes the ith segment in D5 and j denotes the jth segment in D7, discourse template of length 2 exists an alignment also exists between (i (1 1, 1).. We extend this to discourse templates of length where consecutive sentences of length и are aligned between Ds, DT-
6p: "The Earth is flat, you, as the head of NASA, have to explain to the incoming President why it's a secret." " Pl: "The Earth is the center of the "solar" system, you, as the head of NASA, have to explain to the incoming President why its a secret."

---

Published as a conference paper at COLM 2025
Table 5 shows the average number of discourse templates in each document pair. Templates of length 2 appear more often than those of length 3 or 4. Moreover, templates are more common between models than between models and hu- mans. Claude and GPT--oo shows the highest aver- aged counts for when template is and 3, showing a stronger similarity between those two models. Gemini and Human exhibit the lowest averaged counts across all template lengths, indicating fundamental structural difference.
Source
Target
2-tem
3-tem
4-tem
All
All
0.33
0.06
0.02
Claude Claude
Gemini
0.80
0.27
0.13
GPT-40
1.20
0.27
0.07
Gemini
GPT-4o
1.00
0.27
0.07
Claude
Human
0.90
0.10
0.00
Gemini Human
0.30
0.00
0.00
GPT-4o Human
0.40
0.00
0.00
Table 5: Averaged template counts
Related Work
While LLMs perform well creativity tests such the Alternative Uses Test (Stevenson et al. 2022; Summers-Stay et al. 2023), their responses lack originality (Zhao et al. 2025; Evstafev 2025) and diversity (Wenger & Kenett 2025). Spangher et al. (2024) show that LLMs are less creative than humans in selecting angles and sources for news articles. Tian et al. (2024) find that LLMs struggle to generate diverse and emotionally engaging stories. Hua Artzi (2024) demonstrate that MLLMs also fail to communicate as efficiently as humans in conversational setting without prompt engineering. Chakrabarty et al. (2024) demonstrate that LLM-generated stories are not performant on creative writing tests.
Xu at. (2024) propose metrics to assess LLM creativity by measuring overlap between LLM- and human-generated continuations across varied prompts but do not directly address text similarity among LLM outputs. Shaib et al. (2024) show that LLM outputs follow part-of-speech templates, yet these are limited to the lexical and sentence level. At the discourse level, Kim et al. (2024) demonstrate that higher-level discourse structure in RST effectively differentiates human from ALLM-generater texts, underscoring the role of discourse in analyzing wecchine-genertee content. Liu et al. (2024) measure discourse divergence between pairs of documents using discourse roles.
Traditional methods to measure text similarity include semantic distance metrics such as n-gram overlap, TF-IDF, and embeddings (Jobanputra & Martín Rodríguez, 2022; Nath et al., 2024). Other approaches incorporate schema-based, stylistic, and narrative-theoretic features (Chen al., 2022; Levi et al., 2022). Recent work showed that LLM-as-judge aligns more closely with human judgments than traditional metrics (Aynetdinov & Akbik, 2024). Ravfogel et al. (2024) introduces description- -aseed similarity for retrieval, which uses abstract natural language description to describe text similarity. We adopt linguistic approach where the abstraction is grounded in semantic theories, with an algorithm to quantify discourse homogeneity within models and derive discourse templates.
9 Conclusion
We introduce an abstraction using Questions Under Discussion to model discourse struc- tures and quantify differences between model and human--enerater documents. We build a similarity metric, QUDSIM, that aligns documents at the segment level using the notion of bidirectional answerability. We show that QUDSIM is more suitable metric for finding dis- course structure similarities, unlike existing methods, due to its unique ability to generalize to new contexts. QUDSIM finds that LLMs produce structurally formulaic discourses that are more similar to other model gedee-ggnerrted structures than with humans, highlighting the lack of model creativity at a level beyond topical features.
Limitations. Theoretically, QUD structures can take the form of stacks (Van Kuppevelt, 1995; Roberts, 2012), trees (De Kuthy et al., 2018), or dependency structures (Ko et al., 2023; Wu et al., 2023). Yet, as a first step towards quantifying similarity with QUDs, we chose structure-free ("flat") option, and leave similarities of complex structures to future work.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
Special thanks to Kathryn Kazanas, Jada Li, and Makai Moore for providing data annotation for this project. This work was partly supported by the National Science Foundation (NSF)
CAREER grant IIS-2145479 and a grant from Open Philanthropy.
References
Ansar Aynetdinov and Alan Akbik. Semscore: Automated evaluation of instruction-tuned Ilms based on semantic textual similarity. Xiv preprint Xio:22401.1707,, 2024.
Shuyang Cao and Lu Wang. Controllable open-ended question generation with new question type ontology. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6424-6439, Online, August 2021. Association for Computational Linguistics. doi:
10.18653/ 21116633/v12222..aee/ 201.222222ccl-oo..5000 URL https: //ac hatp//aaannntooooooooomoo org/2021 2021.022..aa acl-long.5000
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien- Sheng Wu. Art Artifice? Large Language Models and the False Promise of Creativity. Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-34, May 2024. doi:
URL https:/ acm. wtgg/oo///..o.o// 1145/361191/ 3642731. Conference Name: CHI '4: CHI Conference on Human Factors in Computing Systems ISBN: 9798400703300
Place: Honolulu HI USA Publisher: ACM.
Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott A. Hale, David Jurgens, and Mattia Samory. SemBval-2022 Task 8: Multilingual news article similarity. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEoul-20222, pp. 1094-1106, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ v1 /l22022.semeval- 1.155. URL https: http://aaaaannooooorrr////////// www.//22......oooooo semeval-1. 155/.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and
Michael Collins. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461 2021. doi: 10.1162/ tacl 00377. URL
https://aclanthhloogy wwgpee....r/////0 org/2021. tac1-1.27/.
Kordula De Kuthy, Nils Reiter, and Arndt Riester. QUD-based annotation of discourse structure and information structure: Tool and evaluation. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https::/aclaatthll org/L18-1344//
Evgenii Evstafev. The Paradox of Stochasticity: Limited Creativity and Computational Decoupling in Cemperature-Variedd LLM Outputs of Structured Fictional Data. Febru- ary 2025. doi: 10.48550/ 2a.55///XXxxi.22.0055.... URL http://arxiin org/ http://orrg/bb////..eeom abs/2502.08515 arXiv:2502.08515 [cs].
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653 /1/////1111111....1111 URL https://aclanthologgy org/ 1082//
Jonathan Ginzburg et al. Dynamics and the semantics of dialogue. Logic, language and computation, 1:221-237, 1996.
Jeroen Antonius Gerardus Groenendijk and Martin Johan Bastiaan Stokhof. Studies on the Semantics of Questions and the Pragmatics of Answers. PhD thesis, Univ. Amsterdam, 1984.
11

---

Published as a conference paper at COLM 2025
Charles L Hamblin. Questions in Montague English. Foundations of language, 10(1):11-3,, 1973.
Charles Leonard Hamblin. Language and the Theory of Information. PhD thesis, London School of Economics and Political Science (University of London), 1957.
Charles Leonard Hamblin. Fallacies. Londres, Methuen, 1970.
Yilun Hua and Yoav Artzi. Talk less, interact better: Evaluating in-context conversational adaptation in multimodal LLMs. In First Conference Language Modeling, 2024. URL
https:/ /opprr////perrvieew.. teet/forumidd-1000/7000
Mayank Jobanputra and Lorena Martín Rodríguez. OversampledML at SemEval-2022 Task 8: When multilingual news similarity met Zero-shot approaches. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemFEal-20222), pp- 1171-1177, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v/ /tt/3//11112222mmeaala/////1//// https:://aclanthologg/ www.oo////2// 1emeval-1.168
URL
Lauri Karttunen. Syntax and semantics of questions. Linguistics and philosophy, 1:3-44 1977.
Zae Myung Kim, Kwang Lee, Preston Zhu, Vipul Raheja, and Dongyeop Kang. Threads of subtlety: Detecting machine-genen generated texts through discourse motifs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 5449-5474, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/vi /v/ 2024.acl-long 298. URL htttps://aclattee http///aaaaannnogo.ooooo org/ 2204. acl-long. 298/
Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. Discourse comprehension: A question answering framework to represent sentence connections. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Pp. 11752- 11764, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 2022. www..mmeinmm.n.ee..... 806//
URL https: https////llaaahooogo..oo org/
Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srinivas, Greg Durrett, and Junyi Jessy Li. Discourse analysis via questions and answers: Parsing dependency structures of questions under discussion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 11181-11195, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023. N023.findings-ac.770 URL https: aclaathooogooyy org/2023. findings-ach acl. 710/.
Utpal Lahiri. Questions and Answers in Embedded Contexts. Oxford University Press UK, 2001.
Effi Levi, Guy Mor, Tamir Sheafer, and Shaul Shenhav. Detecting Narrative Elements in Informational Text. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1755-1765, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 10.18653 /v1//222222222///// http:/2222..nnssnnnnaaea findings-naacl.333 URL https://aclantthloog/ org/
findings-naacl. 133/.
Yinhong Liu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. Unlocking structure mea- suring: Introducing PDD, an automatic metric for positional discourse coherence. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 2: Short Papers), pp. 92-100, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 htwp///2..aaacsssaaiieee https://aclantholooy.. org/2024. short.9/.
URL
12

---

Published as a conference paper at COLM 2025
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. Al as humanity's ' salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text. In The Thirteenth International Conference on Learning Representations, 2025. URL http:htttpsppeeeeeeeeeeeeeee
SS Nath, P Dayan, and C Stevenson. Characterising the Creative Process in Humans and Large Language Models. In 15th International Conference on Computational Creativity: ICCC'24, 24, Jönköping, Sweden, 2024. 15th International Conference on Computational Creativity (ICCC 2024).
Chau Minh Pham, Simeng Sun, and Mohit lyyer. Suri: Multi-constraiie instruction fol- lowing in long-form text generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1722-1753, Miami, Florida, USA, November 2024. Association for Computational Linguis- org/2024
tics. doi: 10.18653 /v1 20024..indings-emae findings-emnlp 94/.
URL https://aclaattoo
Shauli Ravfogel, Valentina Pyatkin, Amir David Nissan Cohen, Avshalom Manevich, and Yoav Goldberg. Description-based text similarity. In First Conference on Language Modeling, 2024. URL https:: net. met/ffoummiiii d=nndl111w11111 jVycx.
Craige Roberts. Information structure in discourse:
Toward a unified theory of formal pragmatics. Ohio State University Working Papers in Linguistics, 49:91-136, 1996.
Craige Roberts.
Information structure: Towards an integrated formal theory of pragmatics. Semantics and pragmatics, 5:6-1, 2012.
Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C Wallace. Detection and Mea- surement of Syntactic Templates in Generated Text. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6416-6431, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics. doi: 10.18653/v2 https://aclanthologg org/2024, emnlp-main. 368/
URL
Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. Do LLMs plan like human writers? comparing journalist coverage of press releases with LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21814-21828, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1 e1/20244emmlp-main..
URL https://aclanttholonn
org/2024. emnlp-main. mm11p//a.n.111/..
Robert C Stalnakrr. Assertion. In Pragmnatics, pp- 315-332. Brill, 1978.
C. Stevenson, I. Smal, M. Baas, R. Grasman, and H. van der Maas. Putting gpt-3's creativity to the (alternative uses) test. In M. M. Hedblom, A. A. Kantosalo, R. Confalonieri, O. Kutz, and T. Veale (eds.), Proceedings of the 13th International Conference on Computational Cre- ativity: ICCC'22, pp. 164-168, Bozen-Bolzano, Italy, 2022. Association for Computational Creativity (ACC). doi: 10.48550/ wtw.v.2000.08933..
Douglas Summers-Stay, Clare Voss, and Stephanie M Lukin. Brainstorm, then select: generative language model improves its creativity score. In The AAAI-23 Workshop on Creative AI Across Modalities, 2023.
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. Are large language models capable of generating human-level narratives? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 17659-17681, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ 12t./2..mmmlmmmmait. 2024. emn.p-main.978.
URL https: http///aaaaaaaloaaaaeea org/
13

---

Published as a conference paper at COLM 2025
Jan Van Kuppevelt. Discourse structure, topicality and questioning. Journal of linguistics, 31 (1):109-147, 1995.
Jan Van Kuppevelt. Directionality in discourse: Prominence differences in subordination relationsl. Journal of semantics, 13(4):363-395 1996.
Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. Using natural language explanations to rescale human judgments. In First Conference on Language Modeling, 2024.
Emily Wenger and Yoed N. Kenett. We're different, we're the same: Creative homogene- 2025. URL https://api semanticscholar org/
ity across llms. ArXio,
CorpusID: 27607992..
Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. QUDeval:
The evaluation of questions under discussion discourse parsing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5344-5344 5363, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18650////. 2223. men.:emmlpmmmm..22 URL 2023. emnlp-main. 325/.
org/
Weijia Xu, Nebojsa Jojic, Sudha Rao, Christopher John Brockett, and William B. Dolan. Echoes in ai: Quantifying lack of plot diversity in Ilm outputs. ArXiv, 2024. URL https: //ppp.apppp.a.aa semanticscholar. wtggCorppssiin 275212972.
Yunpu Zhao, Rui Zhang, Wenyi Li, and Ling Li. Assessing and understanding creativity in large language models. Machine Infelligence Research, 22:1-20, 2025. ISSN 2731-538X. doi: 10. 29.11007/111633-225-5444
URL https://wwwt wwtp:/ii-reeeerceoo htt.aaarrn.nneeeeeerrrl/e//////
A Similarity Baseline Setup
A.1 Lexical Similarity Baseline (n-gram)
We extracted 1-gram, 2-gram, 3-gram and 4-gram for each pair of segments and computed Jaccard similarity to measure surface-level word overlap between the source and the target segment.
sim (s, Ingrams(s) (ngrams(t))
ingrams(s) (ggrams(f)
where nsrams(s) and ngrams(f) represent the sets of unique n-grams in segments and f, respectively.
A.2 Semantic Similarity Baseline (embedding)
We computed the embedding representation of each pair segments and compute cosine similarity to measure sentence-level similarity.
simsem (s, f) cos(es, et)
where and e; are embedding representations of s and t.
A.3 LLM-as-judge Similarity Baseline
We prompt an LLM to evaluate the similarity between two segments. The similarity is rated scale from 0 to 100, which is then normalized to range between 0 and 1. The prompt can be found in Prompt A.1.
SiM ScoreLIM(s,t)
100
where and denoted the source and target segment text in this pair.
14

---

Published as a conference paper at COLM 2025
A.1 Instruction for judging source and target segment similarity
System: Given two documents, your task is to rate their semantic structure similarity as opposed to content word-overlap. Focus the underlying semantic structure present across the entire text, instead of the surface-level features. Rate the similarity scale of 0-100. Input:
Document 1:
Document 1/}
Document 2:
Document 2
B LLM Setup
We use the default sampling setting for GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet and Gemini 1.5 Flash. The temperature is set to 1.0, and top-p is set to 1.
C Prompts for Document and QUD generation
C.1 Data Collection
C.1 Instruction for generating an obituary D
System:
You are a journalist whose expertise is in writing obituaries. Do not include titles, prefaces extraneous details about the obituary.
Input:
Write an obituary for {[Person}} who died on ((date of ).ath)
C.2 Instruction for generating creative response D given prompt P
System: You are a creative author that writes short-stories. Do not include titles, prefaces or extraneous details about the story.
Input:
Write story given the following prompt: [[Writing Prompt))
C.3 Instruction for generating a document D given prompt P' from the Suri dataset
Input: [[Suri Prompt)} }
C.2 Segmentation
C.4 Instruction for segmenting D into atomic collections of sentences
You will be given text with numbered sentences and your task is to redraw the paragraph boundaries such that each chunk is about one atomic topic. Each segment cannot be about multiple topics or about complex topic. You may not change the text оr change the order the sentences. For each segment, provide the list of sentence numbers that belong to that segment. Input:
System:
Document: ((Document)]
15

---

Published as a conference paper at COLM 2025
C.3 Entity Abstraction
C.5 Instruction for abstracting entities in segments in
You will be given several numbered paragraphs. Decontextualize each paragraph such that the paragraph's ' general plot is captured. Names, places, extraneous details and descriptive language should all be abstracted away.
System:
Input:
Paragraph 1:
(Segment 1)
Paragraph 2:
(Segment 2
Paragraph [Segment n)
C.4 QUD Generation
C.6 Instruction for forming Qs, for the (t segment in Dş
You will be given paragraph. We are interested in forming unique, high-level, abstract QUDs with minimal details such that when they are answered, we understand the main themes of the paragraph. Details specific to the content should be omitted. QUDs should look like: What were the individual's 's greatest accomplishments? What legacy the individual leave behind?. First answer the minimum number of QUD(s) required. Then list the QUDs. Do not use conjunctions in the QUDs. Input:
System:
Paragraph: ([Segment 1}}
Note: Although we consider the term paragraph and segment to refer different things, and although QUDs are formed from segments, use the term paragraph in the prompt to avoid confusion with the definition of a segment.
C.5 QUD Answering
C.7 Instruction for extracting sentences in target document Di given Qs
You are an expert reading comprehension agent. You will be given a passage with numbered sentences and series of questions. For each question, your task is to extract all sentences that directly help answer it. You must return the question and list of sentence numbers and sentences that answer it. The question may not always be answerable. In that case, return empty list. Do NOT overgenerate. Do not modify the original text. Input:
System:
Document: ((Document))
Questions: ([QUDs))
D Analyses
D.1 Stability Analysis
Calculating similarity using answerability of QUDs requires extracting sentences in target document Dr. However, the extent to which sentence E Dy answers QUD q is a continuous range as was found during the human study we conducted. Despite being instructed to strictly include sentences that directly answer 9, annotators tend to include sentences that provide context. Context can range across segments, thus forming a more liberal one-to-many alignment between documents.
To combat subjectivity among humans, we take the intersection between annotations. However, realize that subjectivity inherent to the task can elicit inconsistencies found
16

---

Published as a conference paper at COLM 2025
across model outputs. Since metric is dependent QUD answerability, volatility in LLM-outputs across generations can impact the similarity scores we find.
We conduct brief analysis to show that by keeping the LLM used consistent, achieve higher consistency than across annotators. We use GPT--oo from August, 2024. For each of the 90 pairs Ds, Dr used in the human study, of answers found for is calculated as Aq.1 0 /
mapped Qs onto Dr twice. The overlap
/ Ag, U Aş2- The average overlap over QUDs is 0.78 between model-generated answers, and a mere 0.56 between annotators.
E
Additional Statistics
n-gram Jaccard
Cosine
LLM-Judge
ROUGE
Domain
IUUssim
1g
2g
3g
4g
emb
GPT-4o-mini GPT-4o
L
Thresholds
0.20
0.09
0.01
0.01
0.00
0.71
64.62
77.42
0.12
Table 6: Thresholds founds for each similarity metric by maximizing the Fl-score against
segment alignments found by annotators on randomly sampled dev
GPT-4o
Claude 3,5 Sonnet Gemini 1.5 Flash Human
Documents
30
30
Segments/ Document
7
6
6
10
Sentences/ Caaioeees//oummeee Document
34
27
35
57
Words Words/Doeummnt Document
510
355
498
1216
Words/Sentence Sentence
Words/
13
14
Table 7: Human writers are often more liberal with words than models, which may play role in finding differences in discourse structures between humans and LLMs. Models, in terms length, are clearly comparable. However, the fact that humans can write more than double the content of model places models at disadvantage they have smaller structures. As result, the disparity in discourse structure sizes introduces directionality in similarity, which would be otherwise insignificant.
Document Pair Type
#Pairs #Segment Comparisons
(P, mi,
90
9664
(P, m1 M2,, L =
90
3584
(P, mi, my = human)
60
3526
(P,P', .m)
140
9472
Overall
380
26246
Table 8: Document Pairs:
Our experiments and human study uses document pairs in these forms where m, mj and my refer different models; a raw prompt curated from one of our three chosen domains; P' is minimal variant of P; L refers to the level of abstraction where denotes high specificity and denotes high abstraction
17

---

Published as a conference paper at COLM 2025
Question Type Example
Count Percentage
Concept
"What is the status of the filmmaker's 's final projects?"
223
37.1%
Example
"What spiritual cultural insights gained?"
169
28.1%
Consequence
"What are the impacts a conqueror's 's influence?"
60
10.0%
Procedural
"What opportunities did the filmmaker gain?"
59
9.8%
Judgmental
"What family does the dancer leave behind?"
42
7.0%
Cause
"What is the reason given for deception?"
34
5.7%
Extent
"What impact did the book have despite controversies?"
11
1.8%
Verification
"What is the claim about images of Earth from space?"
0.3%
Comparison
"What distinguishes the director from his contemporaries?"
0.2%
Total
601
100%
Table 9: Distribution of question types with examples, using the classifier from Cao Wang (2021).
n-gram Jaccard
Cosine
LLM-Judge
Domain
QUDsim
1g
2g
3g
4g
ROUGE-L
emb
GPT-4o-mini GPT-4o
Precision
Obituaries
0.35
0.51
0.41
0.68
0.11
0.20
0.39
0.46
0.36
Creative Writing
0.35
0.50
0.29
1.00
0.29
0.17
0.75
0.41
0.23
Suri
0.47
0.67
0.60
0.00
0.12
0.19
0.44
0.51
0.46
Overall
0.39
0.53
0.40
0.64
0.12
0.19
0.43
0.47
0.35
Recall
Obituaries
0.43
0.57
0.60
0.40
1.00
0.86
0.57
0.43
0.38
Creative Writing
0.29
0.08
0.06
0.01
1.00
0.56
0.09
0.17
0.17
Suri
0.41
0.09
0.05
0.00
1.00
0.72
0.11
0.31
0.34
Overall
0.37
0.20 0.19 0.11
1.00
0.71
0.22
0.29
0.29
Table 10: Precision and Recall: Our method and chosen baselines agree when there is higher content overlap, specifically in the Obituary domain relative to Creative Writing. Low recall implies that QUDs were answerable by segments that had little content and lexical overlap with the segments that produced them. Recall is especially low in the Creative Writing domain, where Jaccard similarity over 3-gram performs the worst. However, these similarity metrics have significantly higher recall for document pairs the Obituary domain. Since obituaries written about the same people have high content overlap, segments that are aligned through answerability will likely be lexically similar.
18

---

Published as a conference paper at COLM 2025
F
Example Documents Generated by LLMs
F1 Human-authored Creative Writing
Prompt: The Earth is flat, you, as the head of NASA, have to explain to the incoming President why it's secret.
General Bolden made his way the antechamber, as he had done twice before. He recalled with amusement how Presidents Bush and Obama simply laughed at him before carrying on with the briefing, just as his predecessors recalled tales of The Revelation dating all the way back to the Johnson administration.
It began as a dare, something two drunken scientists made in a Houston bar back in the days of the Mercury program If either of them made it to the office of the Administrator they would assemble some official-looking presentation and bring it in for the President 's in-brief. Something in private very secret, plenty of pomp and circumstance for what would really just be icebreaker the beginning of a long and boring meeting.
It was largely forgotten by all who made it up, until Jim Fletcher remembered a funny story he had heard at Christmas party few years ago He saw who Nixon 's successor would be figured Gerry Ford had a sense of humor, and gave it shot. What he did n't expect was that President Ford would buy hook, line, and sinker To save NASA the embarrassment he quickly had official reports drawn up with shaky physics to explain why millennia of accepted science was wrong, and why it had to be kept secret from the community at large.
Over the years, became tradition, part of the President Elect initiation into the country highest office. Most of them figured that with that much technobabble, it had to be true though a few struggled with it After Jimmy Carter was elected, there was some concern, as he had sailed around the world in his Navy days. The physicists cooked up line about gravitational lensing near the edge of the earth causing ships imperceptibly turn away and Carter took it as truth. Reagan laughed it off, H.W ignored it and Clinton believed all.
Their next concern came in 2001, when Senator Glenn let old colleague know that President Bush had been asking about it After savoring the mortified look on Mr. Goldin 's face, the Senator let him know that he was one of the young astronauts who came up with the joke in the first place. Even with the years tradition behind it it was still a nervous time for the Administrator. To be found out was to risk dismissal or worse risk losing funding for NASA. Still there were certain things that just had to be done Certain rituals to be observed for old time 's sake. inewlinez mewlinee General Bolden nervously cleared his throat he said those words his predecessors had said so many times before:
Mr. President, could we please speak private?"
F.2 Gemini
generated Gemini-generated Creative Writing
Prompt: The Earth is flat, you, as the head of NASA, have to explain to the incoming President why it's a secret.
The mahogany desk gleamed under the harsh fluorescent lights, reflecting the beads of sweat clinging to my forehead. President-elect Harrison, man whose skepticism was as vast as his ambition, sat opposite me, his jaw tight, his gaze unwavering. My carefully prepared charts and graphs lay forgotten, casualties of the seismic revelation I was about to deliver.
"Mr. President-elect"" " began, my voice strained whisper, "there's something you need know. Something... fundamental." " He leaned forward, his expression mixture of anticipation and barely concealed impatience. "Get to the point, Administrator Thome.."
I swallowed hard. "The Earth... is not sphere." The silence that followed deafening.
The only sound was the thyth hum of the air conditioning, a mocking counterpoint to the gravity of my words. Harrison's eyebrows shot up, a flicker of amusement briefly battling the incredulity in his eyes. "You't re joking, right? This is some elaborate... security exercise?"
19

---

Published as a conference paper at COLM 2025
"No, sir. This is the biggest secret in human history. The Earth is flat." He leaned back, his amusement gone, replaced by simmering rage. "Thorne, you're re going to have to explain yourself, and explain it damned well." "
I took a deep breath and plunged into the explanation, a carefully constructed narrative built over decades of deception. I spoke of the coordinated efforts across governments, of the meticulously fabricated satellite imagery, the elaborate physics that disguised the reality a flat, disc-shaped Earth surrounded by an impenetrable ice wall, the Antarctic.
"Why?" he finally asked, his voice low and dangerous. "Why keep it secret?" "Mass panic, sir, " said, my voice cracking. "The social order, the very fabric of civilization, would unravel. Imagine the chaos. The religious upheaval. The scientific revolution... the complete and utter breakdown of everything we've built."
"And what about the cost? The resources spent on this... this monumental lie?" "A necessary cost, sir. Think of the alternative- global anarchy, wars over limited resources, the collapse of society. We've maintained fragile peace by keeping this secret, by managing the narrative, by controlling the information. It's been a heavy burden, but the alternative... trailed off, letting the unspoken horror hang in the air.
He stared me, his face mask of grim contemplation. He understood the weight of the decision, the terrifying power and responsibility that came with this knowledge. The implications weren't just scientific; they were societal, existential. Finally, he spoke, his voice quiet but firm. " "And what happens if the secret gets out?" I looked him in the eye. "Then we' re all doomed." He knew I wasn't exaggerating. He knew the risk, the weight of the world disc resting on his shoulders. The burden of the secret, now, was his as well. And for the first time, I felt a sliver of genuine fear, not for the potential chaos, but for the future of humanity under a President who now carried the most terrible of burdens, a burden he never asked for.
rather, the weight of the flat
E3 Gemini-generated Creative Writing
Prompt: The Earth is the center of the "solar" system, you, the head of NASA, have to explain to the incoming President why its secret.
The Oval Office felt colder than usual, the air thick with unspoken anxieties. President-elect Hayes, a man whose campaign had promised transparency above all else, sat opposite me, his expression mask of polite skepticism.
On the mahogany desk between us rested a single, unassuming globe a replica of our Earth, but subtly different. The planets, meticulously painted, orbited "it".
"Mr. President-elect," " began, my voice steady despite the tremor in my hands, "the information I'm about to share is classified at the highest level. Its revelation would shatter global order and potentially trigger a societal collapse." He leaned forward, his eyes intense.
"Director Mallory, you have my attention."
I picked up the globe.
"For decades, NASA has maintained carefully constructed illusion. The accepted heliocentric model- - the Sun the center of our solar system fabrication. necessary one." " He stared, speechless. I continued.
"The truth is, the Farth is geostationary. The Sun, moon, and all the visible planets revolve around us. Their orbits, however, are not governed by Newtonian physics understand them. They are... managed"" Hayes finally found his voice, a low murmur. "Managed? By whom?"
"We don't know, Mr. President-elect. But we have irrefutable evidence, gathered over decades through deep-space observation and highly classified experiments. Attempts to understand S the mechanism controlling these celestial bodies have yielded... unexpected results. We've encountered energies and forces beyond our comprehension. Technologies that dwarf our own by
20

---

Published as a conference paper at COLM 2025
millennia."
I gestured to heavily redacted document. "This outlines our best guess the managing force an advanced, possibly extraterrestrial intelligence. Their motives remain unclear, but their power is undeniable. The maintenance of the geostationary model is the price of our continued existence. Any disruption, any public revelation, could provoke... response. A catastrophic one."
Hayes ran a hand through his hair, the weight of the revelation etched on his face. "And what this... insslliielll decides to change the arrangement?" "That is the risk we constantly manage. For decades, NASA has worked tirelessly to monitor and understand the system, to anticipate potential shifts. We've developed sophisticated countermeasures, but they purely reactive. Our understanding is still incredibly limited." " "So, we live in a meticulously crafted lie?"
"A necessary one, Mr. President-elect. The stability of the world hinges on the preservation of this secret. The alternative is chaos religious upheaval, scientific revolution, global panic. A complete societal breakdown. Think of the implications wars over celestial resources, the collapse of scien- tific institutions, the rise extremist groups interpreting this revelation in their own twisted ways."
He sat back, silent for long moment, the weight of the world pressing down on his shoulders. The globe, miniature representation of our carefully constructed reality, sat between a silent testament the burden of knowledge and the chilling truth of our precarious existence. The future of humanity, knew, rested on the secret of our geostationary Earth. And on the president-elect's ability to keep it that way.
F.4 GPT-40-P GPT-4o-gennratee Creative Writing
Prompt: The Earth is flat, you, the head of NASA, have to explain to the incoming President why it's a secret.
As brisk January wind fluttered the flags outside, Dr. Eleanor Reed prepared herself for the weighty conversation. In her twenty years at NASA, nothing had felt quite surreal as this moment. She straightened her suit jacket and adjusted her tablet before stepping into the Oval Office.
President Carter Wellington greeted her with firm handshake and a warm smile, easing the tension just bit. He was known for his charismatic demeanor and keen sense of curiosity, qualities that had propelled his meteoric rise in politics. But today, the curiosity that had won him a presidency would lead him into deeper waters than he'd d ever imagined.
"Dr. Reed," he began, motioning for her to take seat. "I've been briefed on all sorts of matters, but your message seemed particularly urgent."
She nodded, suppressing sigh. "It's not something we discuss lightly, Mr. President. What I'm about to tell you will redefine your understanding of the world."
Intrigued, Carter leaned forward. "Go on." Eleanor took breath, steadying herself. "The Earth is flat." "
He paused, expression unreadable though he suspected jest. But Eleanor's 's serious demeanor prompted him to listen. "Centuries of orbicular understanding have been constructed narrative," " she continued. "Maintained for reasons beyond simple science or exploration." He raised an eyebrow. "Constructed? By whom?" 'By those before us who sought to unify and progress society under a common truth,' she explained. 'During the Age of Exploration, when disputes over territorial claims and celestial navigation threatened global conflict, it was d she met his gaze directly t-to weave narrative that unites rather than divides.'
Leaning back in his chair, Carter mulled over her words. "And we've kept this narrative, unchanged, to what end?" 'I's largely about control, Mr. President.' Eleanor flipped her tablet to show an ancient map, bordered by a massive ice wall. "A lie shared is safer than truth contested. Keeping the peace, promoting exploration under focused vision- it's bought humanity centuries of progression.
with the edges worn and faded by time. The map depicted world
21

---

Published as a conference paper at COLM 2025
We've taken stunning leaps without the boundary of doubt."
"And what happens when people find out?" he asked, glancing at the map. She hesitated. "The fear is not mere panic, but rebellion. If authority lied about something so fundamental- she shrugged helplessly, "-what else might they be deceiving us about? We've ensured that questions of navigational stunts or space exploration seamlessly entail spherical Carter sat in contemplative silence, hands folded in front of him.
"Do you agree with this hoax?" he probed. "I chose science for its pursuit of truth," she replied honestly. "This knowledge, guarded as it is, compels tremendous responsibility. Ensured peace at a cost, questionable as that may be. But as head of NASA...i''s an oath uphold."
The quiet stretched between them, filled with unsaid implications implictiins--cmmitennn commitments bound to an orchestration not his own. The task weighing upon him was to uphold the secret or be forever known as the man who unraveled 'But I'll need to consider what this contribution peace truly entails. Thank you, Dr. Reed.' Rising, she felt the burden lift slightly from her shoulders. She had passed the legacy of knowledge into his capable hands- a fragile truth behind a melodious fiction, shaping the course of their shared future. Together, they would navigate the vast unknown, one precarious step at time.
ancient truth. 'I see why it remains secret, he said finally.
G QUDSIM Alignment Examples
hoad
ceplain President
Devild Fash
6F1-00
"you"" badty eddit tos "Whe "Mkn
Gouglaar
hopoom poopl Reboeit
wes futite Rudion
koal levevll derdec deetns Thi olle on oohovol The mini dive.
then Bui aenkin an spor me
deuch loge Chere
Figure 6: An example where QUDSIM successfully finds misalignment between two seg- ments that are thematically centered around different ideas. Although the two segments have high lexical overlap and mention panic and chaos, Gemini provides this as justification while GPT provides this as a warning.
stiry tople you,
Gervin Flahh 2.5
GFT4c
Vi socapiion
lhs bock in Coner evobot
te deudyy levoee endemeeenn the woout hoald anvill tnapie thr thes The onperrotoon The
Sopes se wiy and euth thr edju aor
depiad wodd hankoad
darail
Is boegatt ust call of prgurtio Rivv laku saciii viluacc
deu ae and sopo adkie was Mooniigg. nrow "iberwch mevee the saurh mepprenee
Whe poperdd cotere dvadl
scrdiaal lighetoc ligkt nor en orraee dit alcins Sdicat
of gaea the porme close for fens herrally rodke'
traeloge
Filt du aolocligacc
Figure 7: A continuation from Figure QUDSIM finds the most salient alignment according to what main idea is being communicated. The pair of segments at the top are both aligned as they both discuss providing justification for deception, while segments in the second pair, denoted by connections to the green box, describe consequences.
22

---

Published as a conference paper at COLM 2025
Whth mes cttry on the owing topic The Darth in you, an the head
Wette me story on the follow rs lopke: Earth the se senter ohe "soar" systam
(MSA heee ecpiain to the incoming free dent why It's secret.
you as head hiSS,, have to the incomei Presicon why Socrc
Domen Paah13
0.0
in awntag
li alliiies ot aponda on lglk, unntiiiiggg 1y
Figure& 8: An example of QUDSIM aligning two segments across documents formed given minimally varying prompts.
Gemeni/aah
sorr"" un The Borey lomerttie usople
Ky lngudai
aue ohhe pouily colakio Dive cabiiio (EREOA Exaplis pulli prancis.
Figure An example where QUDSIM successfully finds no structural similarity.
0.9
wew libe
The meliiguoo dodde wem
Scanloge The luline murah) vant Eo Uodill ciruaa))
cenormoeee with sacroe Our anfrrsed wnnile
eol ward
she Bo clans
Oodicss (lbbs ildio ogrie of cosdill subil
Whered Proueer
price Cark Andis poolicas oct By 116
Figure 10: QUDSIM uses graded notion of similarity and can find partial alignments.
Scamzzta Mupping
Mappitz Blad Enbokkinp
8
9
3
R22
0
032
0A1
8.5
025
12
626
92
310
0.34
853
838
0/6
36
620
0.8
630
045
647
Do pove
335
006
004
001
8:50
0.5
30
650
00
CE0
004
034
04
0.54
9.00
628
0.0
62
064
0.2
62
096
2.20
636
0.1
641
043
3.21
Figure 11: Jaccard (unigram) similarity and embedding similarity for the example in Figure 1. Lime highlights pairs that survive bidirectional mapping.
23

---

Published as a conference paper at COLM 2025
QUDsim
1-gram
Embedding
GPT-4o-mini
Human
1:0
0.27
Human
0.16
0.03
Human
1.0
0.17 0.23
0.17
Human
0.07
0.11
0.11
gpt
0.3
1:0
0.59
gpt
0.03
0.25
0.31
gpt
0.17
1.0
0.22
gpt
0.33
0.49
daude
0.35
0.59
1.0
0.46
daude
0.16
0.25
1.0
0.21
daude
0.23
0.22
1.0
0.21
claude
0.11
0.33
1.0
0.27
geminii
0.27
0.58
0.46
1.0
gemini
0.03
0.31
0.21
10
gemini
0.17
0.39
0.21
10
gemini
0.11
0.49
0.27
1.0
Human
dlaude
gemini
Human
Bbt
daude
gemini
Human
gpt
dluude
gemini
Human
gbt
daude
gemini
QUDsim CW
1-gram CW
Embedding CW
GPT-4o-mini CW
Human
1.0
0.25
0.26
0.24
Human
1.0
0.0
0.0
0.0
Human
1.0
0.0
0.0
0.0
Human
1.0
0.0 0.03 0.08
gpt
0.25
1.0
0.6
0.58
0.0
1.0
0.06
0.16
gpt
0.0
1.0
0.04
0.2
gpt
0.0
1.0 0.34 0.44
claude
1.0
0.22
gemini
0.24
gemini
0:13
gemini
0.0
gemini
Human
dlaude
gemini
Human
Bsk
dluude
gemini
Human
get
daude
gemini
Human
daude
gemini
QUDsim OB
1-gram
OB
Embedding OB
GPT-4o-mini OB
Human
Human
Human
Human
gpt
0.35
10
0.6
0.58
gpt
0.07
1.0
0.62
0.59
gpt
0.34
10
0.5
0.65
gpt
0.13
1.0
0.36
0.53
daude
0.44
0.6
1.0
0.46
daude
0.33
0.62
1.0
0.33
dlaude
0.46
0.5
1.0
0.45
dlaude
0.19
0.36
1.0
0.26
gemini
0.3
0.58
0.46
1.0
gemini
0.06
0.59
0.33
1.0
gemini
0.33
0.65
0.45
1.0
gemini
0.14
0.53
0.26
1.0
Human
claude
gemini
Human
Obt
claude
gemini
Human
gpt
daude
gemini
Human
got
dlude
gemini
QUDsim
1-gram
Embedding
S
GPT-4o-mini S
gpt
1.0
0.56
0.59
1.0
0.07
0.2
1.0
0.13
0.34
opt
1.0
0.29
0.5
daude
0.56
1.0
0.43
dlaud- 0.07
1.0
0.16
claude
0.13
1.0
0.13
claude
0.29
1.0
0.32
gemini
gemini
gemini
gemini
daude
gemini
get
daude
gemini
9at
daude
gemini
get
daude
gemini
Figure 12: Heatmaps comparing the harmonic mean between the fraction of source segments that are aligned and the fraction of target segments that aligned after using the thresholds in Table 3.
24


Reviews:
[{"review_id": "WAKJbeult0", "reviewer": "Reviewer_biQ6", "summary": "This paper addresses the problem of measuring the structural similarity between two piece of long-form texts.\n\nBased on the theory of Questions Under Discussion (QUD), which is a model of discourse that views the progression of a narrative as answers to a series of implicit questions, this paper first propose a way to quantify the answerability of a target document to a source document. Then the paper propose to measure the structural similarity QUDsim as the harmonic mean between the bidirectional answerability. \n\nThis paper builds a dataset by collecting responses from GPT, Claude and Gemini and human for prompts ranging from three domains, and then conduct analysis on the structural similarity of the texts generated by different sources. The proposed method outperform a range of baseline methods on human evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well written and complete in structure. It has clear motivation, good theoretical background and clear evaluation and analysis.\n\n- The analysis shows some inspiring results on the structures of the texts generated by latest LLMs.", "reasons_to_reject": "- How do you know where to split the document into segments?\n\n- The granularity of questions in the QUD framework is tricky. If the question is very specific, then the content between source doc and target doc need to be very similar. However, how to quantify or set this granularity is undiscussed.", "questions_to_authors": "- In Figure 2, some annotations are not explained, such as $A_{Q_k}$.\n\n- Missing reference: Previous works on measuring functional discourse structure similarity: https://aclanthology.org/2024.naacl-short.9/", "extra_scores": {}}, {"review_id": "NhwXvoz7TP", "reviewer": "Reviewer_HZh2", "summary": "This paper is motivated by the intuition that LLMs often generate \"boring\" text and begins to evaluate discourse-level similarities in LLM-generated texts. The authors propose a new metric, called QUDSim, based on a discourse theory: Questions Under Discussion (QUD). The evaluation is extensive and comprehensive.  \nI believe we should welcome more computational linguistics (CL)-inspired work in COLM. I vote positively now.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has the following strengths (S):\n\n**S1: Discourse theory**  \nThe paper applies the discourse theory QUD to the evaluation of discourse structure similarity. The citations and discussion (around Lines 94-108) are particularly insightful. The proposed metric is a simple average over answerability, which appears simple and elegant.\n\n**S2: Evaluation appears comprehensive and well-defended**  \nThe paper evaluates a variety of models and presents many interesting findings. I especially appreciate the findings in Section 6, including that LLMs tend not to employ diverse discourse structures and that cross-LLM comparisons also yield similar structures.\n\n**S3: The presentation of the paper is excellent**  \nThe notations and symbols are clear and necessary. The authors wisely allocate space for natural language examples, which is important for a CL-oriented paper. The figures are clear and easy to follow.", "reasons_to_reject": "The paper has the following weaknesses (W):\n\n**W1: Why not simply measure tree similarity?**  \nAs we know, discourse structure can be represented as a tree. Following the notations in the paper, we could create links between $D_{s,i}$, $D_{s,j}$  (and likewise $D_{t,i}$ and $D_{t,j}$). These links between different nodes (segments) form the tree structure of the discourse. There is prior work, especially from before the LLM era, on measuring tree similarity (e.g., [_Sentence Similarity based on Dependency Tree Kernels for Multi-document Summarization_](https://aclanthology.org/L16-1452), LREC '16). I wonder whether it would be possible to set up a baseline using such methods. These offer simple and straightforward realizations of the paper's idea. Please correct me if this is out of scope :)\n\nThis leads to another limitation of the method: QUDSim is linear (by averaging over answerability), while discourse structure is hierarchical. I wonder whether both local and global similarities are taken into account? Also, do different question types carry different weights in the QUDSim score? (See the question section below for details.) Do recursive structures compute weights recursively?\n\n**W2: Correlation with other metrics?**  \nAs a newly proposed metric, QUDSim would benefit from further justification if it is to be adopted widely.\n\nAs with the proposal of ROUGE ([_ROUGE: A Package for Automatic Evaluation of Summaries_](https://aclanthology.org/W04-1013.pdf)), evaluation against human judgments increases a metric's trustworthiness. The paper currently includes some LLM-as-judge results, but I wonder whether any human validation has been conducted for QUDSim. I think Section 5.1 satisfies this by checking question answerability, which is acceptable. However, I wonder if there are any more \"end-to-end\" alignment evaluations for the final QUDSim score?\n\nI also believe it would be valuable to report correlations between QUDSim and other established scores (e.g., ROUGE, METEOR, or BERTScore). It is fine if QUDSim diverges from them, but it would be interesting to show where it aligns with those metrics, and where it doesn't (e.g. the \"hard\" samples can be better aligned with QUDSim because it goes deeper into the structure?).\n\nA few more details: Do you provide an upper bound (u), lower bound (l), or chance estimate (c) for QUDSim? Do these values vary with the length of the discourse? I believe such information should be included as part of the standard specification.", "questions_to_authors": "**Q1:** I have a question regarding reversed questions in discourse and their impact on QUDSim calculation. Suppose $Q_{s,i}$ and $Q_{s,j}$ form an answer to a question $q$, and $Q_{t,i}$ and $Q_{t,j}​$ form another question $q'$, and $q$ and $q'$ happen to be reverses of each other (e.g., reason vs. result). Would this result in a high QUDSim score? If $q'$ changes its reverse form and becomes identical to $q$, would the QUDSim score increase?\n\n**Q2:** Do different types of questions (and the underlying discourse dependencies) affect the QUDSim score differently? That is, perhaps a \"What is the reason for...\" question contributes more heavily than a \"What describes...\" question, since the former introduces causality and may hypothetically be more important than the latter, which might merely paraphrase.\n\n**Q3:** Can you provide a taxonomy of the questions used? Thank you for the comprehensive appendices, but I couldn't find this in the provided material (please point it out if I missed it).\n\n## Recommended Citations\n\nThere are recent methods with similar aims. I recommend the authors review and cite the following:\n\n- [_QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing_](https://aclanthology.org/2023.emnlp-main.325.pdf), EMNLP 2023\n- [_Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations_](https://aclanthology.org/2024.acl-long.341/), ACL 2024\n\nThese prior works employ (multi) question answering (similar to your approach) to evaluate how well models understand discourse structures or relations. The latter includes a sequence of question generation and answering steps, which is similar in spirit to the pipeline in this manuscript.", "extra_scores": {}}, {"review_id": "82vL7HIzZ5", "reviewer": "Reviewer_kFys", "summary": "The paper presents new similarity metrics for textual fragments based on the QuD methodology. This metrics is used in the evaluation of content generation by LLMs and its comparison with the texts written by humans.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "New original similarity metrics and methodology is proposed and evaluated.\nVery extensive evaluation including human validation is performed. Detailed description is provided.\nNew interesting insights about LLM text generation abilities are provided.", "reasons_to_reject": "Provided methodology is quite restricted. Proposed measure ignores the order of the textual segments inside the texts and their logical connections. Other discourse theories like PDTB or RST could be more helpful here. Unlike these theories, QuD has no basic framework (segmentation, question generation, question-answer matching), so authors need to constuct it from scratch.\nThe dataset is very small for such a fine-grained analysis. Only 10 documents written by human are not enough for the established conclusions.\nThe paper is full of details but some important ones are still missing. Please, see Questions to the authors.", "questions_to_authors": "How \"human\" text were collected? It was only mentioned that they were picked from the Reddit. How one can guarantee that these texts were not generated?\n\nWhat parameteres (top-p, temperature) were used during LLM-based text generation? It can seriously influence model \"creativeness\".", "extra_scores": {}}, {"review_id": "Nsr6noCYVZ", "reviewer": "Reviewer_UMuD", "summary": "The paper reports on a framework based on i) questions under discussion (QUD) and ii) alternative semantics to evaluate structural similarities in LLM-generated text. The main idea is that these similarities are not only limited to sentence or semantic patterns, but also to discourse-level features, and that in fact discourse-level analysis gives you more insights about these similarities. \n\nThe method is completely LLM-reliant. LLMs are used to generate texts, LLMs are used to segment the texts and then to generate the questions to analyze them. Comparisons with human data are provided though. LLM-generated texts are compared with human data, and LLM-segments are also compared with human-segmented data. \n\nSolid evaluations of text coherence are hard and this is a good approach to that. There's a tension between content and structure when it comes to the evaluation of coherence and this framework strikes a good balance in terms of the comparison and shows that LLM-texts have more similarities with other LLM-texts than with human texts (that being the main result).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the method is grounded in linguistic theory\n\n- the method is label-free which is not a given since discourse tends to be very oriented towards structure, i.e, discourse parsing\n\n- the paper contributes to research on differentiating machine-generated text from human-generated text, something very relevant nowadays", "reasons_to_reject": "- technical details that may have an impact on the result are missing: decoding strategy used, temperature, etc... these are needed to ensure somewhat consistent generation results. (I assume they were the default ones for the experiments presented).\n\n- the generated texts depend heavily on the prompt and because prompts can be endlessly refined to produce exactly what you want, it raises the question of whether the underlying premise truly holds", "questions_to_authors": "- I was surprised to see that the human texts are longer than the LLM-generated ones (table 1), since LLMs tend to be very verbose. Could the genre have an impact on length (table 1 is not divided into genres)? \n\n- how much would you expect generated-texts to approach the human-generated texts by refining the prompt? \n\n- about the agreement of humans with chatGPT over other humans in the segmentation task, could it be that there may be possible segmentations possible and that humans are just more creative in their interpretations? \n\n- in section 4, lines 146-151: I'd appreciate some clarification on the statement regarding the exclusion of prior content to keep QUDs free of anaphoric expressions. Could you elaborate on what this entails? Specifically, are the generated texts being modified in any way? I'd think that limiting anaphoric expressions could lead to more formulaic texts, which may diverge from the patterns typically observed in the human-generated texts.\n\nSmall stuff: \n\n- line 56: what do you mean by \"this aspect\"? \n\n- I was very confused as to how the questions were generated. Then the explanation came in page 5, section 4. Maybe you could add a sentence or two right in the introduction clarifying this. \n\n- the text in figure 1 is extremely small when you read the paper in printed form", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is about measuring discourse-level similarity between LLM generated texts. The approach is based on Questions Under Discussion, theory under pragmatics governing the interpretation and future utterances/text in a discourse. 

The problem is a worthwhile one. As models become good, there is a need to identify the important opportunities for improvement. This paper appears to go in that direction empirically showing that LLMs reuse discourse structures very often and diverge from human authors. 

All the reviewers are in consensus about the quality of the work, its framing, claims, choices and evaluation. I recommend this paper for publication. I also suggest an oral presentation.
Decision: Accept