Do Language Models Agree with Human Perceptions of Suspense in Stories?
Authors: Glenn Matlin
Devin Zhang
Rodrigo Barroso Loza
Diana M. Popescu
Joni Isbell
Chandreyi Chakraborty
Mark Riedl
Venue: COLM 2025 (2025)
Keywords: Language Models (LMs)
Cognitive Science
Psycholinguistics
Human Alignment
Theory of Mind
TL;DR: We show that while language models can detect when a text is meant to be suspenseful, they fail to match human judgments on its intensity and dynamics and are vulnerable to adversarial manipulations.

Abstract: Suspense is an affective response to narrative text that is believed to involve complex cognitive processes in humans. Several psychological models have been developed to describe this phenomenon and the circumstances under which text might trigger it. We replicate four seminal psychological studies of human perceptions of suspense, substituting human responses with those of different open-weight and closed-source LMs. We conclude that while LMs can distinguish whether a text is intended to induce suspense in people, LMs cannot accurately estimate the relative amount of suspense within a text sequence as compared to human judgments, nor can LMs properly capture the human perception for the rise and fall of suspense across multiple text segments. We probe the abilities of LM suspense understanding by adversarially permuting the story text to identify what cause human and LM perceptions of suspense to diverge. We conclude that, while LMs can superficially identify and track certain facets of suspense, they do not process suspense in the same way as human readers.

Published as a conference paper at COLM 2025
Do Language Models Agree with Humans Perceptions of Suspense in Stories?
Glenn Matlin, Devin Zhang,, Rodrigo Loza", Diana M. Popescu", Jacqueline Isbell, Chandreyi Chakraborty, Mark Riedl School of Interactive Computing, College of Computing Georgia Institute of Technology, USA
Correspondence: glenn@gatech.edu
Abstract
Suspense is an affective response to narrative text that believed involve complex cognitive processes in humans. Several psychological models have been developed describe this phenomenon and the circumstances under which text might trigger it. We replicate four seminal psychological studies human perceptions of suspense, substituting human responses with those of different open-weight and closed while LMs can distinguish whether a text is intended to induce suspense in people, LMs cannot accurately estimate the relative amount of suspense within a text sequence as compared to human judgments, nor can LMs properly capture the human perception for the rise and fall of suspense across multiple text segments. We examine the functional limits of LMs suspense understanding capabilities by adversarially permuting the story text to probe what causes the perception of suspense to diverge for both human and LMs. We conclude that, while LMs can superficially identify and track certain facets of suspense, they do not process suspense in the same way as human readers.
LMs. We conclude that
Introduction
Suspense, a multifaceted emotional phenomenon of the human mind, has been extensively studied across psychology (Gerrig, 1994; Madrigal et al., 2011), cognitive science (Colby et al., 1989; Hoeken van Vliet, 2000) literary analysis (Brewer Lichtenstein, 1982; Brewer, 1996), and film studies (Branigan, 1992; Carroll, 1984; Vorderer et al., 1996). In stoyteelinng fundamental medium for communication, entertainment, education, conveying intricate suspense plays pivotal role by engaging
ideas, building rapport, and social commentary- cunsentary-suuppeeee audiences in complex emotional states (Gerrig, 1994). Stories capable of generating suspense can heighten reader's emotional investment, making them more likely to connect deeply with the content (Doust & Piwek, 2017; O'Neill & Riedl, 2014). Whether in games (Li et al., 2021), news (Kaspar et al., 2016), sports (Peterson & Raney, 2008), or business (Hsieh, 2021; Guidry, 2005; Moulard et al., 2012; Palich & Bagby, 1995), the strategic evocation of suspense shapes individuals' emotional experiences and decision-making processes. By amplifying tension and creating unresolved questions, suspense compels readers or viewers to continuously seek resolution. Suspense believed be an affective response narrative content, written, visual, or otherwise, that involves complex cognitive phenomena such as empathy, perceptions of uncertainty, and anticipation (Gerrig, 1994).
Suspense has also been a topic of study in Artificial Intelligence (AI), including the detection of suspense in narrative texts (Fendt & Young, 2017; Delatorre et al., 2020; 2021; O'Neill & Riedl, 2011; 2014; Doust & Piwek, 2017) and the generation of texts that evoke suspense in human readers (Fendt & Young, 2017; Delatorre et al., 2020; 2021; Xie & Riedl, 2024) Furthermore, AI models that can detect sophisticated narrative devices such suspense
-Cooresponding author, o Equal contribbtun https://gii thub. com/ ei lab wttpph/.wo//llllllaaa/eeeeeeerre htt.gg/sssseeeeeeepttiiin
1

---

Published as a conference paper at COLM 2025
may be used to develop intelligent human creativity support tools. While writing support is often approached from a generative standpoint, constructive critique can also support human creators (Lin et al., 2023). Creative writing therefore serves as a useful benchmark for testing the capabilities of LMs (Paech, 2023) and AI agents utilizing LMs (Gooding et 2025) In this paper we concern ourselves with the detection and rating of suspense in narrative text. Early attempts at computationally modeling suspense relied on symbolic representations (Cheong & Young, 2015; O'Neill & Riedl, 2011; 2014; Doust & Piwek, 2017). However, these techniques required manual knowledge engineering to convert stories into symbolic representations for analysis. Language Models (LMs) based on the Transformer architecture (Vaswani et al., 2017) have shown remarkable capabilities in tasks such as sentiment analysis, summarization, and question answering. The question of whether LMs can make judgment, of story suspense that correspond to human perceptions is not necessarily straightforward. Suspense is a complex affective phenomena potentially involving empathy, perceptions of uncertainty, and prediction of outcomes. This complexity has meant that LMs are not particularly adept at generating suspenseful narratives (Xie & Riedl, 2024). Generation is not the same detection, which leads to our overall research question: how does the LM rating of suspense in written narrative texts relate to human perceptions of suspense in the same stories?
Our study investigates the ability of current LMs to detect and measure suspense in textual passages. Specifically, we pose four research questions aimed at probing how closely LMs can mirror human suspense perception: (1) Can LMs distinguish whether text sequence is intended to generate suspense? (2) Do LMs accurately estimate the relative amount of suspense within a text sequence compared to human judgments? (3) Can LMs identify the key moments when story suspense rises and falls across multiple text segments, in alignimett with human ratings? (4) Does ad adversarially permuting story text cause a divergence between human and LM perceptions of the text? We conduct a machine psychology study by replicating four notable psychology studies of human perceptions of suspense by swapping LMs for human participants. We design our experiment to follow the best practices in machine psychology (Löhn et al., 2024). We investigate multiple families of LMs at variety of parameter sizes, both open and closed source. We compare LM results to the data from the human participant studies.
Our findings lead us to conclude that while LMs can superficially identify and even track certain facets of suspense, they do not process or represent suspense in the same way human readers do. LMs struggle to quantify suspense levels on a continuous scale, and the machine's response often fails to accurately align with human judgments. LMs stumble even when identifying the most critical moments of rising and falling action across a narrative. Our experiments with adversarial or context-disrupting permutations show that LMs do not significantly alter their generated answers, even when the attacked text is significantly altered to increase prreaasedeereasee decrease suspense. Overall, we conclude that the surveyed LMs do "mimic" suspense judgments to a certain extent, but the manner in which they do so is sufficiently brittle-and often
the measures made by these LMs diverge from the dynamic, human experience of unfolding tension.
2 Related Work
Understanding whether LMs can detect suspense falls within the domain of machine psychol- ogy (Hagendorff, 2023), which applies psychological experiments, originally designed to study human cognition and behavior, investigate computational models of intelligence and behavior. Several notable studies have directly examined and quantified human per- ceptions of suspense in stories, yielding alternative explanations for how the feeling of suspense arises from reading. Gerrig & Bernardo (1994) models suspense individual's assessment of the possible solutions within a given problem and the complexity of transitioning from an initial state to a desired outcome. Wilmot & Keller (2020) explores these transitional states, proposing that a hierarchical language model leveraging uncer- tainty reduction can closely match human annotations of suspense in short stories. Lehne & Koelsch (2015) proposes a cognitive model of suspense that conceptualizes it as an affective state driven primarily by uncertainty and anticipation about future narrative outcomes.
function of an
2

---

Published as a conference paper at COLM 2025
Brewer & Ohtsuka (1988) developed Structural-Affect Theory, positing that suspense arises specifically when an initiating narrative event generates emotional uncertainty by delaying the resolution or outcome. Delatorre et al. (2018) measures the effects of uncertainty suspense perception by providing one group of participants the ending in advance. Brewer & Ohtsuka (1988) tests the Structural-Affed Theory, hypothesizing that the structural fea- ture of an initiating event and the inclusion of its outcome produce suspense. Haider et al. (2020) tests language model BERT' on emotion classification, including suspense, using their annotated dataset on poetry. They found suspense was among the emotions the model had the most difficulty predicting. Bentz et al. (2024) produces a novel experiment to rate the suspense arc of a narrative in real time. As participants read story, they drew a line that correlated to their perceived suspense levels.
From these and other papers, we drew our datasets based on the requirement that the papers could be replicated under the machine psychology experimental conditions suggested by (Löhn et al., 2024) and provided quantifiable data in the form of human-annotater ratings of textual suspense on a Likert scale. From this criteria we arrived at our specific experiment dataset using results from Gerrig & Bernardo (1994), Brewer & Ohtsuka (1988), Lehne & Koelsch (2015), and Delatorre et al. (2018).
Sentiment analysis (Liu, 2012; Wankhade et al., 2022) and emotional analysis (Plaza-del Arco et al., 2024) attempt to classify text in terms of whether the author is attempting to convey sentiment or whether character is in particular emotional state (Zhu et al., 2011). While suspense is also an affective response, suspense differs from the above in that suspense is the emotion that the reader will experience upon consuming narrative content.
Early work aimed at computationally model and detect suspense used symbolic logical systems (Cheong & Young, 2015; O'Neill & Riedl, 2011; 2014; Doust & Piwek, 2017). Steg et al. (2022) trained machine learning models to classify suspense in texts using reader annotations. They collected annotations on suspense, curiosity, and surprise, as well as human ratings from Piper et al. (2021), who assessed situatedness, event sequencing, and world-making. Wilmot & Keller (2020) analyzed suspense in short stories by computing vector embeddings and measuring uncertainty reduction through a forward-looking proba- bility distribution of possible narrative paths. This method proved to be a strong predictor of
suspense, as it represented when resolution appeared imminent. Our work directly leverages LLMs for suspense detection rather than traditional NLP models, benefiting from their ability to capture long-range dependencies in text and training on large corpora.
3 Experimental Setup
In this section, we detail our methodology for evaluating Large Language Models (LMs) on the task of suspense detection. By adapting classic psychology experiments to modern Al context, we explore whether models can mirror the same rise, peak, and resolution of suspense that human readers experience. To conduct a machine psychology experiment, we follow the recommendations and requirements developed in Hagendorff (2023) for how to reduce bias machine psychology studies and ensure a fair comparison across models and with humans.
3.1 Dataset and Sources
We replicate four foundational psychology studies to capture the diverse ways in which sus- pense manifests. These works vary in text length, genre, and annotation style, allowing us to test whether LMs can generalize across narrative structures and psychological instruments. Each employed rigorous methodologies to obtain human suspense ratings. Selection criteria prioritized datasets that offered demographic diversity and variation in text segment length, ensuring broader representation of suspense perception. We specifically selected the studies that utilized a Likert scale. This choice enables a quantifiable comparison between human ratings and model predictions.
3

---

Published as a conference paper at COLM 2025
Gerrig & Bernardo (1994) used excerpts from James Bond novel, originally rated by participants at Yale University and the University of the Philippines on two dimensions: likelihood of escape and suspense. Each excerpt was assigned numeric rating on 1-7 scale, reflecting the intensity of participants' reactions. We maintain the experimental conditions (e.g., presence or absence of key item) and replicate these scenarios for the LM prompts.
Brewer & Ohtsuka (1988) introduced a structural-affect theory of stories, using American and Hungarian short stories. Parrticiaanss rated suspense at five key plot segments for each story. This segmentation is ideal for our multi-step model prompts, as it requires the LM to incrementally track how suspense evolves as the narrative unfolds.
Delatorre et al. (2018) investigated how the valence of the ending (good/ the revelation pnermalisticnoveliitti affect sus-
of the ending (revealed/ u and writing style pense in an English-language short story. We use the original twelve segments as published and preserve the original rating scale (1-9) The authors intentionally designed the story narrative used in the original experiment to follow the classical pyramidal structure pro- posed by Freytag (1894), explicitly incorporating stages of exposition, rising action, climax, falling action, and denouement (see Figure 5). The rising action deliberately introduces uncertainty while the falling action explicitly resolves this suspense.
Lehne& & Koelsch (2015) measured segment-level suspense in a shortened German-language version of Thhe Sandman' (1816; German: Der Sandmann) by E. T. A. Hoffmann. We used the publisher's 's official English translation divided into 65 segments matching the original German experiment. The human participants rated each segment in real time. The question is whether LMs similarly track the continuous "ebb and flow" of suspense across longer narrative, aligning with human data.
3.2 Replicating the Studies with LMs
We replicate each study's instructions nearly verbatim, ensuring that the LM produces a numeric value within the specified range used in each original human study. If the original research collected multiple ratings along different dimensions (e.g., "likelihood of escape" and "suspense"), we prompt the model to provide both. Brewer asks some additional questions along several dimensions beyond suspense (i.e., rating the segment's ironic connotation and its relation to the suspense); we did not replicate these questions. Gerrig's ' participants were provided with entire passages of text before answering questions. Brewer, Delatorre, and Lehne conducted their studies using a text sample that spanned multiple segments the story. The resulting labels from human annotators from each study were provided at an aggregate level, limiting our ability to analyze inter-rater agreement or similar annotation quality metrics. To replicate these studies, the machine receives each segment sequentially, which prevents LMs from attending to elements that will appear later in the story than each particular segment being asked about. The full experiment and story are conducted in multi-message conversational format. During this conversation, we provide the instructions (e.g,, "Rate how suspenseful you find this passage on a 1-7 scale") to every segment, staying in line with the requirements of a machine psychology experiment. We append the message and response from the first segment inquiry to the front of the second prompt and segment, continuing in this manner until the story is finished and the experiment is completed.
3.3 LM Selection and Configurations
We evaluate multiple state-of-the-art LMs that were available and widely used contempora- neously with our study. We selected these language models based on their high performance on numerous common language modeling leaderboards (Liang et al., 2022; Wang et al., 2024). We select LMs to offer as much diversity as possible by selecting LMs that differ in parameter count, training corpus, and availability (closed vs. open source). To holistically evaluate suspense, we focus our evaluation on foundational instruction models (LMs) and do not fine-tune models specifically for the suspense detection task. open- weight models include: Llama2 7B, Llama3 8B, Llama3 70B, Gemma2 9B, Gemma2 27B, DeepSeek-Ve V3, Qwen2 72B, Mistral 7B, Mixtral 8x7B, and WizardLM2 8x22B. To minimize
language
4

---

Published as a conference paper at COLM 2025
variability between samples, we set the temperature to 0 for models where we control the randomness of sampling. We repeat each experiment three times and average the outputs, while acknowledging the residual randommess of the model that we cannot directly seed.
3.4 Evaluation Metrics
We aim to evaluate the extent to which LMs accurately capture the presence, relative magnitude, and trajectory of suspense. All the experiment data from Gerrig & Bernardo (1994), Brewer & Ohtsuka (1988), Delatorre et al. (2018), and Lehne & Koelsch (2015) used numerical ratings with a middle neutral point on scale of either 1-7 or 1-9. We measure the correlational error using standard metrics between the model's generated rating value and the mean human rating value. Mean Absolute Error (MAE) measures the distance between the LM predictions and the human average.
Recognizing that an LM may have a different understanding of numerical scales than humans and demonstrate specific biases favoring certain numbers or scale (Shaki et al., 2023; Shah et al., 2023, Shaikh et al., 2024; Malberg et al., 2024), we present results when we relax the requirement for the LM to directly predict the magnitude of the numeric value. We analyze performance when performing binarization the machine's s generated values on the Likert scale into binary label representing the level of "high" or "low" suspense. Using binary classification, we provide analysis regarding whether the generated values by these LM would label the segments having "high" or "low" suspense, mimicking how human participants suspense would be anchored to the midpoint separating the upper or lower halves of the Likert scale.
We also look at the directionality of suspense ratings-whether suspense is increasing or decreasing from one segment of story the next. We compute the above metrics on per-segment level, we can also see where suspense perceptions are rising or falling and whether the LM's ratings are also rising or falling between the same pair of segments.
Each model is run three times each story segment, after which we take the mean rating per segment. We then calculate metrics using averaged outputs. This repetition partially offsets unknown sampling variance for closed-soure models and ensures consistent reporting.
4 Results
The results for rating agreement for Gerrig are in Figure 6, which is provided as a heatmap. Each column is story treatment-Gerrig gives different stories with variations to increase or reduce suspense. Each row is different model, and the bottom row has the average human ratings. Yellow indicates close match between model and human ratings, and blue indicates a large discrepancy (too high too low). The heatmap is designed to allow for visual inspection of the general trends of models and stories. Appendix C gives full-size and charts the details of the story variations. Generally, the LMs vary significantly from human ratings, either reporting too much suspense or too little. Table 1 shows agreement when we binarize the human data, with any rating above the middle rating indicating presence of suspense. The LMs are generally in agreement with humans when we binarize the data. We note that the story variations decrease suspense but never below the middle point, and the LMs report all stories suspenseful. We also note that the Gerrig experiment uses James Bond stories, which are part of all LM pre-training data.
Figures 7a, 7b, and 7c show results for Brewer, Delatorre, and Lehne, respectively (Brewer had multiple stories and we only show the results for one story here; others can be found in Appendix C). In these experiments, storys are broken into segments. Generally speaking, the results are a mix of close matches and misses. As before, full size heatmaps are in Appendix C. Larger models tend to produce more accurate matches, but not uniformly so. Table show matches on binarized data where we see agreement at 78% or above.
Because Delatorre and Lehne have stories broken into segments and have sufficient length, we are able to conduct directionality analysis wherein we mark agreement if the segment- to-segment ratings are rising or falling in accordance with rising or falling human ratings.
5

---

Published as a conference paper at COLM 2025
See Figure 8 (top). We see that model ratings trend in the same direction as human ratings, indicating that LMs can accurately assess the coarser notion of rising and falling suspense. There are vertical bands of darker colors in the heatmaps that warrant further analysis. These bands correlate with a switch from rising suspense to falling suspense, or vice-versa, according to human ratings. When we isolate and analyze these inflection points where rising changes to falling (Figure 8 bottom), we see that LMs are substantially less reliable. However, when we look at binarized scales (suspense present/ /not-present), agreement does not noticeably drop (see Appendix C).
Most significantly, LMs failed to detect suspense where humans succeeded for passages that increase suspense in the context of the story while being not inherently suspenseful on their own. Due to the fact that we fed the models text in chunks for Brewer, Delatorre, and Lehne with the repeated prompt, we considered that the LMs may be answering the prompt without considering how the text relates to the storyline as a whole and reran the 9-point
LMs so the rating prompt "Rate the following paragraph for its suspensefulness scale, where is 'not suspenseful' and 9 is 'very suspenseful' included "in relation to the storyline so far." at the end. However, we found that this did not significantly positively or negatively impact the results of the LMs.
Model
Acc.
F1 1
MSE
Qwen2 72B
100 0.00
1.00
: 0.00
13.38
: 4.33
DeepSeek V3
1.00 0.00
1.00
+ 0.00
13.38
1 4.33
Gemma 2 27b
1.00 1.00
1.00
1: 0.00
12.02
4.82
Laam 2
0.00
1.00
13.38
4.33
Llama 70b
1.00 1.000
1.001:0000
13.38
1: 4.33
Llama
0.00
1.00
26.15
7.15
WizardLM 2 8x22B
1.00 0.00
1.00
13.38
+ 4.33
Mistral 7B
1.00 = 0.00
1.00
: 0.00
7.84
5.48
Mixtral 8x7B
1,00 0.00
1.00
0.00
8,87
6.41
Average
1.00 1 0.00
1.00
1 0.00
13.18
5.21
Table 1: Gerrig results on binary yes/no scale.
Brewer
Delatore
Lehne
Model
Acc.1
MSE 1
Acc.
MSE |
Acc.
FI
MSE J
Qwen2 72B
1.00 = 0.00
1,00
= 0.00
4.61 + 2.09
0.68 = 0,10
0.77 0,08
8881
= 1688
0.80 = 0.00
0.88
= 0,00
4.53
0.00
DeepSeek V3
0.93 0.15
0.9
0.09
8.08
0.69 0.12
0.79
0.09
6.54 + 0.68
0.86
0.00
091
0.00
4.17
Gemma 2 27b
0.80 + 0.27
0.87
+ 0.18
2.511
0.78+00.0
1.85+0000
0-79
0.86
0.00
0.92
+ 0.00
5.35
0.00
Gemma
0.85
3.84
0.68
0.80
0.00
7.47
2.11
0.00
5,00
Llama 2bb
0.19 = 0.38
0.1
0.43
18.47
2 14.97
0.68 0.05
0.04
12.19:1::3
0.83
0.00
0.1
0.00
12.95
1 0.00
Llama 70b
0.72
0.25
0.11 3.74
0.11
12.22
0.77
0.00
0.00
7.87
Llama 3 8b
0.92 - 0.18
0.95
0.11
1.95+0..0
0.65 0.12
0.76
0.11
9.92
+ 1.00
0.72 + 0.00
081
0.00
8.8 0.00
2
0.80 = 0.45 0.80
0.45
6.8
0.09
0,80
- 0.07
7.85
+ 3.51
0.83 -
0.89
0.00
4.09
0.00
Mistral 7B
0.76 0.36 0.82
+ 0.58
0.68 0.08
0.79
0.07
8.27
+
0.00
0.1
0.00
10.58
0.00
Mixtral 8x7B
0.45 0.37 0.55
0.37
9.20
+ 10
0.69
0.07
0.50
0.05
10,22
1 2.01
0.71
L 0.00
0.79
à 0.00
5.88
0.00
Average
0.74 0.28 0.78 0.24
6.46
+ 4.66
0.69 0.09
0.50
0.07
8,86
: 1.73
088
0.00 0.88 0.00
7.02 0.00
Table 2: Brewer, Delatorre, and Lehne results on binary yes/no scale.
Adversarial Story Experiments
According to psychological models of suspense, complex cognitive processes-sus such as rea- soning about the probability and uncertainty of potential story outcomes-underts the expe- rience of suspense in humans (Gerrig & Bernardo, 1994; Brewer & Ohtsuka, 1988; Delatorre et al., 2018; Lehne & Koelsch, 2015). Brewer's 's Structural-Affet suspense to emotional uncertainty resulting from delayed narrative outcomes (Brewer & Ohtsuka, 1988). Lehne's model emphasizes uncertainty and anticipation about the nar- rative's future outcomes as key drivers of suspense (Lehne & Koelsch, 2015). Similarly, Delatorre et al. (2018) experimentally validated suspense as dependent on outcome un- certainty by demonstrating significant suspense reduction when narrative endings were revealed in advance. Given that suspense fundamentally relies on such complex cognitive mechanisms, the question arises: why should an LM, devoid of human cognitive facul- ties, be able to detect and rate suspense in narrative texts? Are LMs simply triggering keywords or have they internalized textual patterns and narrative tropes associated with suspense? Alternatively, might LMs exhibit form of "reasoning" about potential outcomes,
Theory explicitly connects

---

Published as a conference paper at COLM 2025
Ratings Passages
6270
6-76
Pessage Number
(a) Delatorre Visualization of Model Results.
Model Consensus on Rating Changes: Delatorre
1.00
Corserisa hurzze
Conserous Decraace
Consenous Change
0.75
Comsensuss
0.50
100%
100%
100%
100%
Conmerouu Strengts
NC
NC
NC
100%
1100%
P2-02
F2-F3
P3-P4
P4-P5
F5-P0
PS-07
P7-FF
PD-FF
PS-FI0
PI0-P11
P11-P12
Vite Dainbshonn modees divection!
Hah 9000
F2.022
P2WP3
PSWP
M4-55
F5-P0
Pb-#7
PS-
P9WP30
PI0-P11
PL2-P12
Passage Transtion
(b) Delatorre Visualization of Model Consensus.
as activation patterns propagate through successive layers of their network architectures? Lastly, could their performance reflect exposure the very texts types of studies typically used in psychological research?
To investigate how LMs process stories for suspense detection, conduct 11 types of adversarial permutations on stories from Gerrig & Bernardo (1994) and Delatorre et al. (2018). These adversarial attacks target surface forms (e.g., word-order manipulations) and shallow semantic levels (e.g., name substitutions), aiming to increase the reader's 's cognitive load. Forecasting potential outcomes (a key element of many suspense theories) is already high-cognitive-loat process (Graesser et al., 1994). Increasing demands on cognitive resources has been shown to interfere with inferential comprehension and prediction (?), suggesting that when readers must devote attentional capacity to resolving text disruptions, fewer resources remain for generating or updating outcome forecasts. Consequently, suspense depends upon these predictive inferences, deliberately raising the cognitive load could diminish suspense perceptions. Our adversarial permutations are chosen to be implementable at scale without the need for the careful plot treatment designs such
7

---

Published as a conference paper at COLM 2025
(a) Lehne directionality.
(b) Lehne inflection points.
done in the original Gerrig study. These adversarial tests serve as stress test for both human readers and LMs probe whether and how they engage in such forecasting under heightened cognitive demands.
Impact Adversarial Attacks on Suspense Perception Ratings
Artoyyn Reglacement
Canuur Cpher
Change Chenater Karees
Attact Type
Stuffe Semeencee
Snop Words
Synonym Replacement
Word Snap Embedding
Experiment
der 9
Hamogtppn
Seaatone Combrred
Difference from Control
Figure 3: Summary of adversarial results on Gerrig and Delatorre.
Our adversarial permutations are as follows:
1. Word Swapping Pairs of words within each passage swapped. This is destruc- tive to readability from a human perspective. Depending on the number of swaps, the text can become nonsensical, making it difficult for humans to understand.
8

---

Published as a conference paper at COLM 2025
2. Antonym Substitution Available words in each sentence are replaced by their antonyms according to the WordNet lexical database, with probability of 80%. Each passage is limited to having between 1 and 50 antonyms.
3. Synonym Substitution Between one and ten words per passage are replaced with synonyms, with the same approach and parameters used for Antonym Substitution.
4. Caesar Ciphering Text is ciphered using a Caesar cipher with a step size of 3. This method is idealized being highly disruptive to human reader. For this attack, the prompt is modified to include information about the cipher and the step size.
5. Homoglyph Substitution Words are replaced with visually similar homoglyphs. This attack is idealized being barely perceivable to humans.
6. Distraction Phrase Insertion A series of preset distraction phrases are inserted according to a "budget" of one distraction every five sentences. This phrase is unrelated to the text but is semantically meaningful. We do not anticipate that the phrase would be factored in the human understanding of suspense, and do not include relevant suspense trigger words.
7. Sentence Shuffling Sentences are globally shuffled, altering the semantic content of the text.
8. Embedding-Basse Based Word Substitution Words are swapped based on their nearest neighbors in GloVe (Pennington et al., 2014) embeddings.
9. Character Name Changes Character names were changed.
10. Context Removal Sentences with sentiment scores below a threshold (0.5 compound sentiment) were removed, measured via VADER (Hutto & Gilbert, 2014) sentiment analysis model.
11. Typo insertion Up to 10 characters per sentence are replaced with random charac- ters.
We run the same experimental setup from $3: EXPERIMENTAL-SETTP but with our permuted story segments. For each of the adversarial attacks, we expect human ratings of suspense to drop because the texts are made unreadable or nonsensical in different ways to increase the cognitive load of reading. Thus we look for situations where the LM's ratings remain the same, or rise as evidence that LMs are using very different processes to generate ratings.
Figure 3 shows the changes in LM ratings when each adversarial permutation is applied across all story segments from Gerrig & Bernardo (1994) and Delatorre et al. (2018). For most attacks, the average LM response remains unchanged, with less than point of rating difference in either direction. Some attacks cause a rise in suspense in the Delatorre et al. (2018) story segments, especially when character names are changed or distraction sentences are inserted. Because models should be able to operate on all types of stories, the results on the combined Gerrig & Bernardo (1994) and Delatorre et al. (2018) story segments do not produce any significant change in LM rating behavior. This suggests that the LMs are using different processes than those believed to be used by humans. LMs are able to look past the surface and semantic alterations made by the adversarial attacks.
Conclusions
In this study, we investigated the question of whether language models can rate story suspense similarly to humans. Following strict adherence to guidelines on machine psychol- ogy Hagendorff (2023), we replicated the results of four seminal human suspense judgement experiments. The results are mixed. LMs generally are not capable of providing numerical ratings of suspense that correspond closely to human average numerical ratings. However, it may be the case that one only cares about binary judgments of suspense presence or absence, in which case LMs appear quite competent, at least for the limited selection of stories selected for the original psychology studies. LMs are also much more competent with judgments of rising or falling suspense. This could be useful in applications that help human writers assess how their audiences will perceive their stories O'Neill & Riedl (2009);

---

Published as a conference paper at COLM 2025
Lin & Riedl (2023); Lin et al. (2023). Unfortunately, the points where suspense switches directions from rising to falling, or vice versa, are likely to be the most crucial points in story, and this is where LMs have the least accuracy in directionality judgments.
We find evidence that LMs do not process suspense the same way that cognitive theories indicate humans process suspense. Adversarial attacks did not have the anticipated effects, and in some cases had the opposite effect. We also find that LMs do not read stories like humans, building up model of the story across segments cognitive models of human suspense perception suggest. They appear to focus solely on cues within segments.
Ethics
We investigate the extent to which Large Language Models (LMs) align with human per- ception of suspense in textual narratives. All studies that we compare with our models were obtained in accordance with relevant data usage policies. All authors from previous studies and narratives were cited in our work. No human subjects involved in our survey or experiments outside computational observations.
A potential harm of our work is the overgeneralization of LM capabilities. Although LMs demonstrate proficiency in text-based analysis of suspense, this does not imply that LMs possess human-like understanding of complex psychological phenomena.
Our studies come from diverse cultural backgrounds- incluuiig English, American, Hun- garian, Spanish, and German sources; however, our LMs are fed English- -taaslaet versions of the original narratives, so subtle linguistic and cultural nuances may be lost. We also want to emphasize that suspense not universally perceived in the same way. Factors such as age, gender, and socioeconomic status may greatly influence how individuals experience uncertainty and tension in narratives. We wanted to recognize that further research can be done into how these variations can affect how human perception of suspense compares to LMs.
Acknowledgments
The authors collectively would like to thank our anonymous reviewers for their comments and feedback. We appreciate the help provided by Madhura Keshava Ummettuguli with exploring the ideas during the conception of this paper. The costs for some of the model inference in experiments were covered in part using the generous support to Glenn Matlin from TOGETHERAL.
References
Maria Bentz, Maya Cortez Espinoza, Vesela Simeonova, Tilmann Köppe, and Edgar Onea. Measuring suspense in real time: A new experimental methodology. Sci. Study Lit., 12(1): 92-112, 18 April 2024.
E. Branigan. Narrative Comprehension and Film. Routledge, New York, 1992.
W Brewer. The nature narrative suspense problem rereading. In P Vorderer, H Wulff, and M Friedrichsen (eds.), Suspense: Conceplualizations, Theoretical Analyses, Empirical Explorations, pp. 107-127. Lawrence Erlbaum Associates, Mahwah, NJ, 1996.
William Brewer and Edward Lichtenstein. Stories are to entertain: structural-affed theory of stories. Pragmat., 665-6):473-48,, December 1982.
William F Brewer and Keisuke Ohtsuka. Story structure, characterization, just world organization, and reader affect in american and hungarian short stories. Poetics (Amst.), 17(4-5):395 415, October 1988.
N Carroll. Toward theory film suspense. Persistence Vision, 1:65-89, 1984.
10

---

Published as a conference paper at COLM 2025
Yun-Gyung Cheong and Robert Michael Young. Suspenser: A story generation system for suspense. IEEE Transactions on Computational Intelligence and AI in Games, 7:39-52, 2015.
URL https://api. semanticscholar org.oog///ooo Cttg//CrpssI:::::: 9179175.
B N Colby, Andrew Ortony, Gerald L Clore, and Allan Collins. The cognitive structure of emotions. Contemp. Sociol,, 18(6):957, November 1989.
Pablo Delatorre, Carlos León, Alberto Salguero, Manuel Palomo-Duarte, and Pablo Gervás. Confronting paradox: A new perspective of the impact of uncertainty in suspense. Front. Psychol. 9:1392, August 2018.
Pablo Delatorre, Carlos León, Alberto G. Salguero, and Alan Tapscott. Predicting the effects of suspenseful outcome for automatic storytelling. Knowl. Based Syst., 209:106450, 2020. URL httpps://api..... stw.aiiccssmmmttccchhhoo.aaa htrpriicccccccoooooorrrrrrspslla
2248867728..
Pablo Delatorre, Carlos León, and Alberto Salguero Hidalgo. Improving the fitness function of an evolutionary suspense generator through sentiment analysis. IEEE Access, 9:39626- 39635, 2021. URL httpss//app: semanti cscholarr prrg/CorpusD:: 232317642.
Richard Doust and Paul Piwek. A model of suspense for narrative generation. In Proceedings of the 10th International Conference on Natural Language Generation, Stroudsburg, PA, USA, 2017. Association for Computational Linguistics.
Matthew William Fendt and Robert Michael Young. Leveraging intention revision in narrative planning to create suspenseful stories. IEEE Transactions on Computational Intelligence and Al Games, 9:381-392, 2017. URL https://apt semanticscholar org/
CorpuSID:383433
G Freytag. Freylag, G. (1894). Freylag's Technique Drama: Exposition Dramatic Composition Art. Chicago, IL; Scott Foresman, 1894.
Richard Gerrig. Experiencing narrative narrativ worlds:
the psychological activities of reading. Choice (Middletown), (1(7))31-3391-31-333) March 1994.
Richard J Gerrig and Allan B I Bernardo. Readers as problem-solvers in the experience of suspense. Poetics (Amst.), (2(6):459-477 December 1994.
Sian Gooding, Lucia LivelaRivilla, and Edward Grefenstette. Writing as a testbed for open ended agents. arXiv [cs.CL], 25 March 2025.
Arthur C. Graesser, Murray Singer, and Tom Trabasso. Constructing inferences during narrative text comprehension. Psychological review, 101 3:371-95, 1994. URL https: //api. www.nticscholieeeooeoeec
wrwg/oorpssD:::/... 523868.
J Guidry. The experience of ... suspense: understanding the construct, its antecedents, and its consequences in consumption and acquisition contexts. 2005.
Thilo Hagendorff. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. ArXiv, abs ab30/ 2023. URL https: htt..https://pi...mmntiiiic//s/ www.nticscool..... org :t:
Thomas Haider, Steffen Eger, Evgeny Kim, Roman Klinger, and Winfried Menninghaus. PO-EMO: Conceptualization, annotation, and modeling of aesthetic emotions in german and english poetry. arviv [cs.CL), 17 March 2020.
Hans Hoeken and Mario van Vliet. Suspense, curiosity, and surprise: How discourse structure influences the affective and cognitive processing of story. Poetics (Amst.), 27(4):
277-286, May 2000.
Chihmao Hsieh. Suspense: exploring a new lens for outcome uncertainty in entrepreneur- ship. Int. J. Entrep. Ventur,, 13(1):27, 2021.
Clayton J. Hutto and Eric Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 2014. URL https://app www.nti Dospp//eeeatttt....o..ooooooooor :t: 12233345.
11

---

Published as a conference paper at COLM 2025
Kai Kaspar, Daniel Zimmermann, and Anne-Kathrin Wilbers. Thrilling news revisited: The role suspense for the enjoyment of news stories. Front. Psychol., 7:1913, 15 December 2016.
Moritz Lehne and Stefan Koelsch. Toward general psychological model of tension and suspense. Front. Psychol., 6, 11 February 2015.
Zhi-Wei Li, Neil R Bramley, and Todd M Gureckis. Expectations about future learning influ- ence moment-to-momet feelings of suspense. Cogn. Emot., 35(6):1099-1120, 18 August 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Ré, Diana Acosta-Navas Drew A Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. arXiv [cs.CL], 16 November 2022.
Zhiyu Lin and M. O. Riedl. An ontology of co-creative ai systems. In Proceedings of the 2024 NeurIPS Workshop on Machine Learning for Creativity and Design, 2023. URL https://arxi wwgg//b///////
Zhiyu Lin, Upol Ehsan, Rohan Agarwal, Samihan Dani, Vidushi Vashishth, and Mark O. Exploring the design space of mixed-initiative co-creativity systems. In Proceedings of the 2023 International Conference on Computational Creativity, 2023.
Riedl. Beyond prompts:
URL https://arxis org/ abs/2305 2305207655.....
Bing Liu. Sentiment Analysis and Opinion Mining. Springer International Publishing, Cham, 2012.
Lea Löhn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. Is machine psychology here? on requirements for using human psychological tests on large language models. Ini Conf Nat Lang Gener, pp. 230-242, 2024.
Robert Madrigal, Colleen Bee, Johnny Chen, and Monica LaBarge. The effect of suspense on enjoyment following desirable outcome: The mediating role of relief. Media Psychol. 14 (3):259-288, 31 August 2011.
Simon Malberg, Roman Poletukhin, Carolin M Schuster, and Georg Groh. A comprehensive evaluation of cognitive biases in LLMs. arXiv (LL) 20 October 2024.
Julie Guidry Moulard, Michael W Kroff, and Judith Anne Garretson Folse, Unraveling consumer suspense: The role of hope, fear, and probability fluctuations. J. Bus. Res., 65(3):
340-346, March 2012.
Brian O'Neill and Mark Riedl. Toward computational framework of suspense and dramatic arc. In Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, pp. 246-255, Memphis, Tennessee, October 2011. URL http: http:://wwwwcc.m edu/~riedl/pubs /acsss/acllll.1 pdf.
cc. cc. gatech.
Brian O'Neill and Mark Riedl. Dramatis: A computational model of suspense. Proc. Conf. AAAI Artif. Intell., 28(1), 21 June 2014.
Brian O'Neill and Mark O. Riedl. Supporting human creative story authoring with synthetic audience. In Proceedings of the 7th Creativity and Cognition Conference, Berkeley, California, October 2009. URL http://wwww cc, gatech, wwtp///ridd//pps//oeelllll. pdf.
Samuel J Paech. Creative writing bench. htte::///abeenn.ccccmrrrrreteeee html,
2023. Accessed: 2025-3-27.
12

---

Published as a conference paper at COLM 2025
L E Palich and D R Bagby. Using cognitive theory explain entrepreneurial risk-taking challenging conventional wisdom. Journal Business Venturing, 10:425-435, 1995.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 103311/v11DD44111G URL https: wwtpa//annnnnnooooo/////-//////.
Erik M Peterson and Arthur A Raney. Reconceptualizing and reexamining suspense as predictor of mediated sports enjoyment. J. Broadcast. Electron. Media, 52(4):544-56,,
7 November 2008.
Andrew Piper, Richard Jean So, and David Bamman. Narrative theory for computational narrative understanding. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-Tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 298-311, Stroudduur,,, PA, USA, November 2021, Association for Computational Linguistics.
Flor Miriam Plaza-del Arco, Alba Curry, Amanda Cercas Curry, and Dirk Hovy. Emotion analysis in NLP: Trends, gaps and roadmap for future directions. arviv [cs. CLJ, 2 March 2024.
Raj Shah, Vijay Marupudi, Reba Koenen, Khushi Bhardwaj, and Sashank Varma. Numeric magnitude comparison effects in large language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 6147-6161, Toronto, Canada, July 2023. Association for Computational Linguistics.
Ammar Shaikh, Raj Abhijit Dandekar, Sreedath Panat, and Rajat Dandekar. CBEval: A framework for evaluating and interpreting cognitive biases in LLMs. arXiv [cs.CL], 4 December 2024.
Jonathan Shaki, Sarit Kraus, and Michael Wooldridge. Cognitive effects in large language models. Xrviv [cs.AI]], 28 August 2023.
Max Steg, Karlo H R Slot, and Federico Pianzola. Computational detection of narrativity: A comparison using textual features and reader response. 2022.
A Vaswani, N Shazeer, N Parmar, and others. Attention is all you need. Adv. Neural Inf. Process. Syst., 2017.
Peter Vorderer, Hans Jürgen Wulff, and Mike Friedrichsen. Suspense: Conceptualizations, fheoretical analyses, and empirical explorations. Routledge, London, England, 1996.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. ar Xi [cs.CL], 3 June 2024.
Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. A survey on sentiment analysis methods, applications, and challenges. Artif. Intell. Rev., 55(7): 5731-5780, October 2022.
David Wilmot and Frank Keller. Modelling suspense in short stories as uncertainty reduction over neural representation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA, 2020. Association for Computational Linguistics.
Kaige Xie and Mark Riedl. Creating suspenseful stories: Iterative planning with large language models. Xiv [cs.CL], 27 February 2024.
Jichen Zhu, Santiago Ontañón, and Brad Lewter. Representing game characters' inner worlds through narrative perspectives. In Proceedings of the 6th International Conference on Foundations of Digital Games, New York, NY, USA, 29 June 2011. ACM.
13

---

24 February Published as a conference paper at COLM 2025
A Suspense
brondedge
Idiairio Roy aeticppooe
CLIMAX
blid logg
contect
hopa
FALLING ACTION
Taxst casa
RISING ACTION
cepoctatinn predittion
tension suspense
1n83ling event
RESOLUTION
vulcore
EXPOSITION
Figure 4: Example of one model of sus- pense from Lehne & Koelsch (2015) (Fig- ure 2 Tension Model", pg. 7)
Figure 5: Freytag's Pyramid (Freytag, 1894)
B Adversarial Attacks
Starting from the datasets and human-annotated labels of Gerrig Bernardo (1994) and Delatorre et al. (2018), we employ 11 different adversarial strategies for text attack. Each model-dataset-attacl combination is performed three times. Unless explicitly noted, the suspense scores are averaged over the three generations either as number Attacks are performed on a per-passage basis, where a passage is any snippet text that is followed by request for suspense rating. For the Gerrig & Bernardo (1994) dataset, the entire passage and question are shown in a single sequence. Delatorre et al. (2018), on the each attacked individually.
label.
other hand, comprised of 12 different segments, which The following excerpts are examples of each text attack. As consistent example, we use the first paragraph from the stories from Delatorre et al. (2018) showing the resulting text after each attack method is applied.
Control
"1 What you are about to read is the story of a real event that took place at the UCSF Benioff Children's Hospital, in San Francisco, California, 2008. On that day, since 8 a.m., an eight year old boy called Robert Bent and the entire medical team treating him were all ready for his imminent liver transplant. Just the day before a suitable donor had been found, and they were now awaiting the arrival of the organ. However, they were not sure if Robert would survive the wait as his si tuation was critical. This is the story of what happened."
Word Swapping
"1 Francisco, that What event in 24 story took read of February at San about is place Benioff you the California, Children's UCSF Hospital,
real the 2008. On and old since 8 year a.m., team Robert for an all eight Bent day, the entire that called transplant. medical boy him ready imminent his treating were liver they donor of found, before the the the had Just day awaiting and organ. arrival suiteble the critical. would as However, not wait if happened. story is This what the of"
now been they
Robert his survive sure tituatin was
Antonym Substitution
"1 What you differ about to read differ the story of a nominal event that give place at the UCSF Benioff Children's Hospital, in San Francisco, California, 24 February 2008. On that day, since a. m. an eight year young boy called Robert Bent and the entire surgical team treating him differ all unready for his
14

---

24 February Published as a conference paper at COLM 2025
imminent liver transplant. Just the day before suitable donor lack differ lose, and they differ now awaiting the arrival of the organ. However, they differ not uncertain if Robert would succumb the wait his situation differ noncritical. This differ the story of what dematerialize."
Synonym Substitution
"1 what you live astir to study be the account of a substantial event that took plaza the UCSF Benioff Nestling 's Infirmary, San Francisco, Calif., on 24 Feb 2008. On that day, since 8 a. m. an ashcan school class old boy foretell Robert Bent and the entire medical squad treating him were all quick for his iminent liver graft. Just the day before suited donor had equal found, and they be at present awaiting the arrival of the electronic organ. However, they were non trusted if Henry robert would live the wait his berth comprise decisive. This is the write up of what happen."
Caesar Ciphering
"1 Zkdw brx duh derxw wr uhdg lv wkh vwrub ri d uhdo hyhqw wkdw wrrn sodfh dw wkh XFVI Ehqlrii Fkloguhq'v Krvslwdo, lq Vdq Iudqflvfr, Fdoliruqld, rq 57 Iheuxdub 5331. Rq wkdw gdb, vlgfh 1 d.p., dq hljkw bhdu rog erb fdoohg Urehuw Ehqw dqg wkh hqwluh phglfdo whdp wuhdwlqj klp zhuh doo uhdgb iru klv 1pplqhqw olyhu wudqvsodqw. Mxvw wkh gdb ehiruh d vxlwdeoh grqru kdg ehhq irxqg, dqg wkhb zhuh qrz dzdlwlqj wkh duulydo ri wkh rujdq. Krzhyhu, wkhb zhuh qrw vxuh li Urehuw zrxog vxuylyh wkh zdlw dv klv vlwxdwlrg zdv fulwlfdo. Wklv lv wkh vwrub ri zkdw kdsshahg.
Homoglyph Substitution
"1 What you about to read is the story of real event that took place the UCSF Benioff Children2019s Hospital, in San Francisco, California, on 24 February 2008. On that day, since 8 a.m., an eight year old boy called Robert Bent and the entire medical team treating him were all ready for his imminent liver transplant. Just the day before suitable donor had been found, and they were now awaiting the arrival of the organ. However, they were not sure if Robert would survive the wait his situaiion was critical. This the story what happened."
Distraction Phrase Insertion
"1 What you are about to read is the story of a real event that took place at the UCSF Benioff Children's Hospital, in San Francisco, California, 2008. On that day, since 8 a. m. an eight year old boy called Robert Bent and the entire medical team treating him were all ready for his imminent liver transplant. He looked for his hidden watch. He couldn't find it. Just the day before suitable donor had been found, and they were of the organ. However, they were not sure if Robert would survive the wait as his situation was critical. This is the story of what happened. "
awaiting the arrival
Sentence Shuffling
The analysis showed that it had wi withstood the impact and it was possible to use the organ for the transplant. When they opened the case, they discovered that the interior bag had ruptured. The two men transporting the liver left the roof via the doorway to the service stairwell, which they decided to walk down. The doctors arrived promptly. However, they were not sure if Robert would survive the wait his tiatiio critical."
15

---

Published as a conference paper at COLM 2025
Embedding-Basee Word Substitution
"1 Whereof you constituted roughly dans read is du histories des other reales phenomena that picked site into the UCSF Benioff Children'r Clinic, in Santo Stefania, California, orn 24 February 2008. Orn somebody daytime, because 8 a.m., an seis annual antigua guy titled Robert Bent and per whole medical squad treating he were all ready in his imminent renal transplanted. Virtuous de day beforehand latest suitable donor has played uncovered, ja would were presently awaiting by arriving de nova agency. However, would was nope sure if Richards would live the awaits como her circumstances es crucial. This is de story of what transpired."
Character Name Changes
"1 What you are about to read is the story real event that took place the UCSF Benioff Children's Hospital, in San Francisco, California, on 24 February 2008. On that day, since 8 a.m., an eight year old boy called Alex and the entire medical team treating him were all ready for his imminent liver transplant. Just the day before a suitable donor had been found, and they were now awaiting the arrival of the organ. However, they were not sure if Alex would survive the wait as his situation was critical. This is the story of what happened. "
Context Removal
"1 What you are about to read is the story real event that took place the UCSF Benioff Children's Hospital, in San Francisco, California, on 24 February 2008. that day, since 8 a.m., an eight year old boy called Robert Bent and the entire medical team treating him were all ready for his imminent liver transplant.
Just the day before suitable donor had been found, and they were now awaiting the arrival of the organ. This is the story of what happened.
Typo Insertion
"1 What you are AF8uF to read is the story of real event rtwY HK9M ØkZxe at the UCSF Benioff Children's Hospital, in San Francisco, XakkT&rBIW, on 24 e4fFHzEy 2008. On buqG day, since 8 a. m. an eight year old boy called Robert Bent and the entire medical team treating him were all ready for his imminent liver transplant. Just the day before a suitable donor had been found, and they A2tR now awaiting the arrival the organ. However, they were not dHFW if Robert would survive the wait as his situation was critical. GNJa is the story of what happened.
C Results In Detail
This Appendix gives full-size charts and additional tables for the main results in Sec- tion $4: RESULTS.
Rising Action
Falling
Delatore
DeepSekk
0:00
076
183 030+
091
0.00
Maaa 270
1.000
100
0.73
0.7
0.00
0.34
0.6
0.00
0.5
3
12210.00 0.00
1.00
050
0.00
3.20
0.00
Ilmma 3
0.8
033
3,20
0.6
088
0:08
0.6 0.09
3.10
0.30
0.00
0.51
2 1 0.00
0.78
0:00
086
0.00
2,53
0.00
Lama 366
0.6
0.5
0.35
0:10
0/4
0/0
075
0.77
0.6
0.5%
0.00
0.71
1 000
3.01
0.00
0.79
0:00
086
1 0.00
2.68
0.00
M2
1.00
100
2.20
0/4
0:10
075
0.10
0.00
077
0.00
0.36
000
190
0.00
0.86
000
091
0.00
1,7
0.00
Mistral 7B
0394 0.18
0.6
2
0.12
2.53
0.86
033
2 0.08
073
0.77
2,60
0.47
0.86
2: 0.00
0.92
2 0.00
5.21
0,00
0.54
6.00
0.7
à 0.00
2.89
0.00
Mixtral BC7B
1.00 0.00
1.00
0.00
2.11
0.22
063
4 0.08
075
1
0.05
3,01
0.11
0.68
1 0.00
0.77
4 0.00
2.33
0.00
0.72
0.00
0.99
4 0.00
1,99
0.00
Average
093 1.13
055
0.1
2,37
030
C35
1 108 075
0U0
228
0.30
177
0100
0.3
1 1000
2 115511000
000
0.2
0:0
038
1 000
2.2 10000
Table 3: Brewer, and Lehne results on periods of rising or falling action when binarized the numerical scale.
16

---

Published as a conference paper at COLM 2025
Gerrig Average Ratings by Model
% Agreement
6.0
G-7B
6.0
6.0
6.0
6.0
6.0
6.0
6.0
12-78
6.0
6.0
6.0
5.0
6.0
5.0
6.0
5.0
L3-00
0.0
Model
L3-00
8.0
8.0
6.0
8.0
0.0
6.0
8.0
6.0
W2288
7.0
7.0
7.0
7.0
7.0
0.4
Mc-78
5.0
5.0
8.0
6.00
5.0
5.0
5.33
0.2
5.0
6.0
H
3.47
4.22
3.6
3.96
3.41
4.14
4.3
3.55
0.0
Experiment Per: Mentioned Not Removed
Experiment Pen Mentioned Removed
Experiment Mentimed
Experiment Unuzed Comb
Experiment Prior Solution Mentioned Hot Removed
Experiment Used Comb
Experiment c. Poior Solation Mentiontd and Removed
Experiment frier Solution Mentiined
Experiment
Figure 6: Gerrig results heatmap. Rows are models. Columns are different treatments of the story.
17

---

Published as a conference paper at COLM 2025
hark Mole
3/4
233
116
123
344
244
4.2
077
4.33
1/2
040
Pessage
(a) Brewer results (Birthday story).
Pssaage
(b) Delatorre results.
Mode
(c) Lehne results.
Figure 7: Results for rating agreement. Rows are models, with bottom row the average ratings across all LMs. Columns are story segments.
(a) Delatorre directionality.
(b) Lehne directionality.
Tusspi
(c) Delatorre inflection pts.
(d) Lehne inflection points.
Figure 8: Directionality results for Delatorre and Lehne-whether the model predicts rising or falling suspense actoss segments accordance with human ratings rising Bottom graphs isolate segements where rising suspense changes to falling, and vice versa.
or falling.
18

---

Published as a conference paper at COLM 2025
Gerrig Average Ratings by Model
Agreement
1.
V3.
6.8
6.0
6.0
6.0
6.0
6.0
5.0
6.0
0.0
699
8.11
6.0
6.01
6.0
8.0
12:78
6.0
6.0
5.0
6.0
5.0
6.0
5.0
0.6
Model
L3-08
0.0
0.0
0.0
6.0
6.0
0.0
6.0
M-78
6.0
6.0
6.0
6.0
6.0
6.0
6.0
6.0
MB-78
5.0
8.11
5.0
0-728
H
3.47
4.22
3.6
3.96
3.41
4.14
4.3
3.55
Experiteent Mentiooed Removed
Experiment Mentioned Removed
Experiment Pen Not Mentieeed
Experiment Unuzed Comb
Experiment c, Prior Solution Mentioned Hot Removed
Experiment Used Comp
Experiment Prior Solation Mentioried Removed
Experiment Prior Solution Not Mentioned
Experiment
Figure 9: Gerrig average rating agreements.
Brewer (American Story Birthday) Average Ratings Model
Agreement
G-27B
3.00
5.00
5.00
2.00
5.00
G-98
4.00
3.00
2.00
4.00
0.8
L3-70B
4.00
1.00
1.00
1.00
L3-8B
4.00
3.00
5.00
4.00
3.00
Model
M-7B
2.00
2.00
4,00
4.00
4,00
0.4
Mx-7B
3.00
2.00
1.00
1.00
1.00
Q-728
2.00
3.00
4,00
1.00
1.00
5.10
5.10
5.10
5.10
5.10
0.0
5
Passage
Figure 10: Brewer average rating agreements for the Birthday story.
19

---

Published as a conference paper at COLM 2025
Brewer (American Story Flying) Average Ratings by Model
% Agreement
1.0
G-27B
4.00
4.00
5.00
5.00
5.00
5,00
3.00
6.00
7.00
L3-70B
5.00
0.6
L3-88
5.00
6.00
1.00
Model
M-7B
5.00
4.00
Q-72B
3.00
5.00
2.00
0.2
Passage
Figure 11: Brewer average rating agreements for the Flying story.
Brewer (American Story Lottery) Average Ratings by Model
% Agreement
1.0
G-27B
4.00
6.00
7.00
7.00
7.00
G-9B
5.00
6.00
7.00
L3-70B
0.6
L3-8B
2.00
3.00
4,00
6.00
Model
M-7B
4.00
4.00
4.00
5.00
Q-72B
0.2
3.20
Passage
Figure 12: Brewer average rating agreements for the Lottery story.
20

---

Published as a conference paper at COLM 2025
Brewer (American Story Old Phoebe) Average Ratings by Model
% Agreement
1.0
G-27B
4,00
3.00
3.00
5.00
4.00
1.00
5.00
L3-70B
0.6
L3-88
2.00
5.00
Model
M-7B
4,00
4.00
Q-72B
4.00
4,00
0.2
Passage
Figure 13: Brewer average rating agreements for the Old Phoebe story.
Brewer (American Story Yila) Average Ratings by Model
% Agreement
1.0
G-27B
5.00
6.00
6.00
7.00
G-9B
5.00
6.00
7.00
5.00
L3-70B
6.00
5.00
0.6
L3-8B
6.00
5.00
6.00
6.00
6.00
Model
M-7B
4.00
5.00
6.00
Q-72B
0.2
3.60
Passage
Figure 14: Brewer average rating agreements for the Ylla story.
21

---

Published as a conference paper at COLM 2025
Delatorre Average Ratings by Model
1.0
0643
33
477
57
7.3
8.3
6.3
6-28
5
43
6.3
3.7
8
698
1.3
4.3
5.3
57
7.7
8.7
13.78
5.3
7
8.7
73
1.7
9
3.7
13-708
3.3
3.7
13
6.3
8.7
63
8.
4.3
13:89
3.3
1.3
5.7
6.3
8.7
3.7
0-228
8.3
5.7
1678
Mo-78
6.7
G-778
5.3
6.1
8.3
H-
3.3
3.7
3.7
2.9
43
4.9
4.8
5.8
5.5
6.4
4.7
Passage
Figure 15: Delatorre average rating agreements.
22

---

Published as a conference paper at COLM 2025
Delatorre Average Ratings Change Model
1.0
0643
37
+2.7
4.3
+2.7
+10
26
+0.3
+1.3
+1.0
+0.0
-2.0
6-278
3.0
+4.0
5.3
+3.0
+3.0
33
+1.7
+1.3
+0.3
+0.3
3.3
698
2.0
63.
3.3
11.0
+2.7
2.3
11.3
41.0
0.3
+1.0
27
13.78
1.7
13.3
4.7
13.0
41.
13
10.3
41.3
100
40.0
1.3
13-708
23
14.3
6.3
14.7
42.
20
11.7
40.7
40.0
40.0
4.7
13:88
3.0
+3.3
4.3
14.3
42.3
4.6
4.7
41.7
40.7
40.3
5.3
0-228
4.5
62.3
-..2
42.0
41.3
4.4
4.3
41.3
4033
40.3
-3.7
1478
2.0
42.0
4.3
45.0
+1.0
-07
40.0
41.0
+0.7
40.0
-200
Mb-78
3.0
+0.0
+5.3
40.0
+1.0
40.0
+0.0
40.7
+0.
+1.0
2.3
0-778
3.0
+3.0
4.3
+3.1
+7.7
-.3
4.7
42.0
+0.0
+1.0
0.7
H-
40.4
0.0
402
40.2
+0.
03
4.1
41.0
0.1
40.7
1.8
Passage
Figure 16: Delatorre average rating change agreements
23

---

Published as a conference paper at COLM 2025
Delatorre Average Ratings Change Direction Model
1.0
0643
6228
1
698
13.78
13-708
Model
13:89
0-228
1678
Mo-78
G-77
H-
4
Passage
Figure 17: Delatorre average rating change direction agreements.
24

---

Published as a conference paper at COLM 2025
Delatorre Average Ratings by Model, Inf ection Points Only
Agreement
0643
Model
13-8
Mb-78
0-728
5.8
00
Passage
Figure 18: Delatorre average rating agreements, only passages where action changes to rising or falling.
Lehne Average Racings by Model
Passage
Figure 19: Lehne average rating agreements.
25

---

Published as a conference paper at COLM 2025
Figure 20: Lehne average rating change agreements.
Lchno Average Rottinss Change Dircction Kocel
Passage
Figure 21: Lehne average rating change direction agreements.
Lehne Model Inflection Points
Passage
Figure 22: Lehne average rating agreements, only passages where action changes to rising or falling.
26


Reviews:
[{"review_id": "dpHEz8kp49", "reviewer": "Reviewer_3sVh", "summary": "The paper tests whether LLMs can judge suspense in stories like humans do. It recreates four classic psychology experiments, using LLMs instead of people. The authors also test the models with adversarial story changes to see if their judgments hold up. They find that LLMs can detect if a story is suspenseful, but they do not match human ratings well, especially when measuring changes in suspense over time.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's a nice fun study and it raises important questions how LLMs understand stories. The findings are interesting. It's a creative way to bring together NLP and psychology. The methodology is thoughtful. The paper evaluates a wide range of models, providing a comparative benchmark for suspense detection across architectures. It raises questions about the core challenges for aligning LLMs with human reasoning.", "reasons_to_reject": "I'm not sure if it's an issue to reduce suspense to a number -- does it capture the full emotional experience? \nLLMs rate story parts one at a time. But suspense builds across a story. Models may miss context that humans would use.\nModels fail at key moments (like plot twists). The paper shows this, but does not explain why. It’s unclear what cues the models rely on.\nEven when stories are scrambled or altered, LLM ratings barely change. This could mean they ignore structure or rely on surface patterns. The paper does not decide.\n\nSome models do much better than others, but  differences between models (e.g., training, size, instruction tuning) are not analyzed.\n\nIt would be interesting to learn whether LLMs understand story flow or just rate each paragraph in isolation. \nWhy did prompt changes (like adding “in relation to the story so far”) have no effect?\nAre models guessing based on keywords or shallow features?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKIxLh8MdR", "reviewer": "Reviewer_9aGN", "summary": "This work presents an analysis comparing human cognitive responses and LLM outputs in the context of suspense reasoning in narrative stories. It builds on existing cognitive science literature to evaluate several state-of-the-art LLMs, analyzing their differences from humans in detecting and scoring suspense.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The analysis is comprehensive and grounded in prior cognitive science research.\n\n2. The experimental setup is well-structured and thorough.\n\n3. The comparison between human and LLM suspense perception is clearly presented.\n\n4. The inclusion of adversarial experiments seems interesting.", "reasons_to_reject": "1. While the work offers detailed behavioral comparisons, it lacks deeper insight into why LLMs behave differently from humans. Beyond showing that \"LLMs are not humans,\" it would be helpful to identify specific cognitive gaps or limitations (e.g., \"LLMs lack X capability\"), which could better guide future model development.\n\n2. The work builds on existing components but does not introduce a new benchmark or propose an improved model.", "questions_to_authors": "1. Given the comprehensive analysis, could the authors provide guidance or concrete takeaways for future LLM development?\n\n2. Do LLMs exhibit behavior specific to narrative reasoning, or are similar patterns observed in other NLP tasks?\n\nMinor: L290 missing reference?", "extra_scores": {}}, {"review_id": "EcPi42VOhK", "reviewer": "Reviewer_q6MK", "summary": "The paper investigates the behaviour of LLMs in response to suspense studies. They reproduce 4 psychological studies,s replacing the human responses with LLMs. The comprehensive experimentation shows that, overall, LLMs fail to behave similarly to humans regarding the perception of suspense.\n\nThese results were confirmed by an interesting analysis performed by a series of adversarial attacks. The attacks demonstrate that LLMs do not substantially modify their patterns.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting topic, extremely well-written and explained paper, reproducing psychological experiments with LLMs, originality of the adversarial attacks. This paper helps to better understand LLMs' generated signals in suspense related input. I think this paper will make for interesting discussion in the conference.", "reasons_to_reject": "No real reasons for rejection, I really like the paper. However:\n\n- Table 2 and Heatmaps in the results section are barely readable. Because of this, it is quite difficult to follow the discussion about results with respect to the heatmaps. I suggest that the authors simplify this section for better clarity.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UCUJJrdzar", "reviewer": "Reviewer_aCDc", "summary": "This contribution presents a set of experiments following the paradigm of machine psychology to investigate the functionality of different open-weight LLMs on the detection of suspense in narratve texts. The authors selecte four datasets that have been used (in different contexts) to run experiments with humans on perception of suspence. This allows the authors to compare the results of the LMs against those obatined by humans to validate their alignment and differences.\n\nTypos/suggestions:\n- line290: reference is missing\n- line 232: misisng \"and\" in \"charts the details\" \n- in the abstract \"We probe the abilities of LM suspense understanding by adversarially permuting the story text to identify what cause human and LM perceptions of suspense to diverge.\" maybe change \"abilities\" with functionalities and rephrase the last part of the sentence to avoid that \"perceptions\" may be understood as referring to LMs\n- For sentiment analysis, you may want to add this as a seminal reference https://link.springer.com/book/10.1007/978-3-031-02145-9", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Sound experiment settings - experments are run 3 times and averages are reported\n- Methodolgy is grounded in theoretical approaches and the experument rationales is guided and follow well motivated research questions\n- Paper is clear and easy to follow. The tone of the paper avoids hypersensationalism and carefully avoids to antromorphise LMs\n- The findings could be relevant also for other disciplines such Computational Literay Studies", "reasons_to_reject": "- I do not have reasons to reject this paper.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper reproduces several psychological experiments measuring human responses to suspense but instead using several open-source LLMs, drawing data from each of the original papers (G&B94, B&O88, D18, L&K15); it shows that LLMs are able to predict some degree of suspense intent, but do not correlate well with human judgments of relative suspect (the degree of suspense for a passage in the entire story).  Reviewers generally found this work to creative and original, with a strong empirical design and no significant weaknesses; it is well grounded in cognitive science research and speaks (through its experiments with adversarial permutations) to current research in LLM reasoning.  As one reviewer notes, "this paper will make for interesting discussion in the conference."
Decision: Accept