DeltaScore: Fine-Grained Story Evaluation with Perturbations
Authors: Zhuohan Xie
Miao Li
Trevor Cohn
Jey Han Lau
Venue: EMNLP 2023 Findings (2023)
Keywords: story
evaluation
metric
PLM
TL;DR: We propose a new evaluation metric that can be tailored to target each aspect of story quality based on perturbations.

Abstract: Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness.
In this paper, we introduce DeltaScore, a novel methodology that uses perturbation techniques for the evaluation of nuanced story aspects. We posit that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DeltaScore with existing metrics on storytelling datasets from two domains in five fine-grained story aspects:
fluency, coherence, relatedness, logicality, and interestingness. DeltaScore demonstrates strong performance, revealing a surprising finding that one specific perturbation proves highly effective in capturing multiple aspects.
Source code is available on our GitHub repository.

DELTASCORE: Fine- GraiGGr Story Evaluation with Perturbations
Zhuohan Xie
Miao Li
Trevor Cohn
Jey Han Lau
School of Computing and Information Systems, The University of Melbourne
(zhuohanx, mia04) @sttudett.uimmlbbeduu {t.cohn, laujh) @unimelb.edu.au
Abstract
Mike and Jake are best friends!
Mike and Jake are bset friends!
Add typos
Numerous evaluation metries have been devel- oped for natural language generation tasks, but their effectiveness evaluating stories is lim- ited as they are not specifically tailored to sess intricate aspects of storytelling, such as fluency and interestingness. In this paper. we introduce DELTASCORE, novel methodology that uses perturbation techniques for the eval- uation of nuanced story aspects. We posit that the extent to which a story excels in specific aspect (e.g., fluency) correlates with the mag- nitude of its susceptibility to particular pertur- bations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by cal- culating the likelihood difference between pre- and post- pestuurrrrraion states using pre- pre-trained language models. compare DELTASCORE with existing metrics storytelling datasets from two domains in five fine-grained story as- pects: fluency, coherence, relatedness, logical- ity, and interestingness. DELTASCORE demon- strates strong performance, revealing a surpris- ing finding that one specific perturbation proves highly effective in capturing multiple aspects. Source code is available on our GitHub reposi- tory.!
0.85
0.5
0.2
Mike dna Jake aer best friends!
Mike dna Jake aer bset friensd!
Add lypos
0.5
0.4
0.1
(a) Perturbation "Add typos" affects the highly fluent story (top) more than the less fluent one (bottom).
The supermarket has various kinds of goods.
Remove relevant words
The various
of
0.95
0.45
0.5
Remove words
is nice day.
It nice day.
0.55
0.55
(b) Two stories are conditioned on the same title "I always go to the local supermarket", Perturbation "Remove rel- evant words" affects the highly related story (top) more while not affect the unrelated one (bottom).
Figure I: Scenarios where higher quality stories (top) affected more than lower quality ones (bottom) through aspect-specific perturbations (fluency: "Add typos"; relatedness: "Remove relevant words"). Gen- erative likelihood for original/perturbed story is in blue/green circle, and the DELTASCORE value is in orange circle.
1 Introduction
The emergence of large pre-trained language mod- els (PLMs) (Zhao al., 2023) has empowered story generation models to generate plausible nar- ratives (Xie et al., 2021; Tan et al., 2021; Zhang et al., 2022b; Yang et al., 2022). The most ad- vanced models have achieved the ability to pro- duce stories which are not easily distinguishable from human-authored ones (Karpinska al., 2021; Dou et al., 2022; Xie et al., 2023). However, the development of automated evaluation metrics in this domain has not progressed at the same pace (Guan et al., 2021b). Human evaluation, though
considered the gold standard. is hindered by its time-consuming, costly, and non-reproducible na- ture (Sai et al., 2023). Consequently, there is a demand for better automatic methods that can eval- uate the quality of stories.
The prevailing evaluation metrics for story as- sessment have primarily been adapted from other natural language generation (NLG) tasks, such BLEU (Papineni al., 2002) for machine trans- lation, or ROUGE (Lin, 2004) for summarization. Fortunately, recent progress has given rise to the emergence of new metrics explicitly tailored for
NNw at Google DeepMind chttps://ggttee http:///ggtthuu..c///hhhaaaa//// wwtp//hhuohan///elltaaaaooo

---

story evaluation, with a focus on quantifying story coherence (Guan and Huang, 2020; Ghazarian et al., 2021) or capturing human preferences (Chen et al., 2022). Other works have directly utilized the likelihood of a story under a PLM (Vaswani et al., 2017: Han et al., 2022) or its conditional likelihood based on human references other con- textual factors, such as story title (Thompson and Post, 2020; Yuan et al., 2021). Nonetheless, these approaches often yield a singular score that pro- vides an estimate of the overall quality. However, Chhun et al. (2022) argue that the quality of story is comprised of various fine-gres grained aspects, such as fluency and adherence to commonsense, suggest- ing that an overall quality score has limited utility for comprehensive story evaluation.
Similarity metries mainly focus on measuring lexical overlap such as BLEU (Papineni al., 2002), NIST (Doddington, 2002) and ROUGE (Lin, 2004) or semantic similarity with contextual rep- resentations including MoverScore (Zhao et al., 2019) and BERTScore (Zhang et al., 2020) be- tween the machine-generate text and its human reference.
Discriminative metrics typically involve train- ing discriminator model to differentiate be- tween high-quality and low-quality texts, includ- ing UNION (Guan and Huang, 2020), MAN- PLTS (Ghazarian et al., 2021), CTC (Deng et al., 2021), StoryER (Chen et al., 2022), and UNIEVAL (Zhong et al., 2022). Specifically, UNION constructs negative samples of original sto- ries using heuristic rules and trains discriminator to differentiate them. MANPLTS an extension of UNION that constructs improved negative samples by manipulating storylines and generating alternate stories based on these manipulated storylines us- ing a story generation model. StoryER builds a classifier to learn human preference by training it to differentiate highly-upvoted stories from lowly- upvoted ones Reddit. CTC treats the evaluation task as an information alignment task. UNIEVAL frames the evaluation as a question answering task where different questions are asked to assess a par- ticular aspect.
In this paper, we present DELTASCORF, a method that evaluates story quality by measuring the likelihood difference using a PLM between an original story and its perturbed version. The idea is that higher quality stories will exhibit more sig- nificant effects from the perturbation compared to lower quality ones. To provide fine-gre the-grained assess- ment of story quality, we experiment with perturba- tions that target specific aspects. Figure I presents two examples to demonstrate the intuition of our approach: 1) When we introduce random typos to modify the two stories shown in Figure la, we observe that the story with higher fluency is af- fected more by the perturbation; 2) When we mod- ify the two stories in Figure 1b by removing rele- vant words, we observe that the perturbation affects the story that has a closer association with the title to a greater extent. Our empirical analysis demon- strates the superior performance of DELTASCORE compared to existing metrics in evaluating intricate story aspects. Furthermore, our investigation re- of our simplest
Generative metrics usually rely on genera- tive likelihood to determine the quality of the text, including BARTScore (Yuan et al., 2021), T5Score (Qin et al., 2022) and GPTScore (Fu et al.. 2023). Specifically, BARTScore evaluates gener- ated text by calculating its conditional likelihood under BART. GPTScore calculates the likelihood of the story under a PLM with additional prefix to target a particular aspect. T5Score benefits from both worlds by employing both generative train- ing with the standard negative log likelihood loss and discriminative training with contrastive loss where human judgments for generation quality are available.
veals an interesting discovery: perturbation methods, which simply shuffles all the words the story, is very effective in capturing multiple aspects. This points to a possible interpre- tation that the pertubation may be functioning as a normalisation factor to modulate the effects of word frequency and text length when estimating sequence likelihood.
2.2 Natural Text Perturbation
The use of perturbations is conventional tech- nique generate negative samples for both dis- criminative (Guan and Huang, 2020) and genera- tive (Zhong et al., 2022) tasks. Ribeiro al. (2020) propose CheckList, a suite of perturbation tech- niques to evaluate the behavioral performance of
2
Related Work
2.1 Automatic Evaluation Metries
Existing automatic evaluation metrics can be broadly categorized into three paradigms.

---

NLP models. Sai et al. (2021) further delve into applying perturbations assess robustness of NLG evaluation metries, while Karpinska et al. (2022) specifically focus on machine translation evalua- tion. He et al. (2022) also develop perturbation tests to identify blind spots of model-based evalu- ation metrics, Notably, all of these perturbations rely on heuristic rules. In contrast, recent adver- sarial attacks such those proposed by Li al. (2020); Morris et al. (2020) use language models to generate adversarial examples, which can also be considered a form of text perturbation. In our work, we explore perturbation for a different purpose: to evaluate fine-grained story qualities.
In terms of decoder PLMs, we concatenate c and s to form sequence æ (I1,.. Zn+m
C1 u. Cns 81, .... 6m) to compute DELTASCORE:
logp(s|c)= 1 m+
Σ
log plai 0) (3)
This formulation means we feed the full se- quence including the story condition e and story s as imput to the decoder-only PLM, although when computing the story likelihood, we only consider the conditional probabilities for the s tokens.
3.2 Perturbations on Story Aspects
We follow Xie et al. (2023) to assess five funda- mental aspects of story quality: fluency, coherence, relatedness, logicality, and interestingness. To this end, we survey perturbation methods from the liter- ature (Ribeiro al., 2020; Sai et al., 2021; Guan et al., 2021b; He et al., 2022) and attempt to align them to one of these five aspects. For some aspects, we also propose new perturbation methods. We now describe each aspect and its associated pertur- bation methods; A summary of these methods and examples is given in Table 1.
3 DELTASCORE
We now describe the idea of our approach. Given a story condition (e.g., a story title) Cl,... On containing n tokens, model-generated story $1, ...,, 8m containing m tokens, and per- turbed story s' 8
$ -'..- Snn containing m' tokens, DELTASCORE calculates the likelihood difference under language model:
DELTASCORE(S) log p(sc) log p(0)p(())c (1)
Fluency assesses the readability of sentences in the story. Perturbations targeting fluency modify the text at the word or phrase level. We use two perturbation approaches from Ribeiro et al. (2020);
where p(s[c) represents the likelihood of a condi-
tioned on e under a language model. In our experi- ments, we investigate several PLMs with varying architectures ($ 3.1) ) and perturbation techniques that are designed to target specific aspects ($ 3.2).
1) Typo, where we randomly transpose a character with an adjacent one in the text, and 2) Subject-verb disagreement SubjVerrDisy where we modify the verbs in sentence that they no longer agrec with their subjects.
3.1 Two Different Likelihood Calculations
We now explain how we compute p(s|c) with encoder-decoder PLMs (e.g., BART (Lewis et al., 2020) and T5 (Raffel et al., 2020)) and decoder PLMs (e.g., GPT-3 (Brown et al., 2020)). p(s'(c) is computed in the same way and we omit it for brevity.
Coherence assesses the level of connectivity be- tween sentences the story. Perturbations tar- geting coherence modify the text at the sentence level. We use two perturbation approaches from Sai et al. (2021): 1) Jumble, where we randomly shuffle words within the story, and 2) Sentence Re- order (SentReorder), where we randomly shuffle the sentences within the story.
Denoting language model parameters as 8, we compute DELTASCORE as follows for encoder- decoder PLMs:
logp(s/c)= 1 m
Relatedness focuses on the extent which the story is relevant to the given condition (e.g., story title). Perturbations targeting relatedness alter the story to reduce its association with its condition. We propose two new methods: 1) Remove Rel- evant Words (RmRelWords). where we use Chat- GPT to identify words related to the given title and
log Dogp(si/8ce: c,
(2)
m
where denotes timestep in the sequence, and 8ct denotes all tokens before the current timestep. In- tuitively, the story condition is captured by the encoder, and the likelihood of the story s is pro- duced by the decoder.
hhutss//chat.opemai.com

---

Aspect
Perturbation
Original story
Perturbed story
Typo
he went see what the problem is the best student in the classroom.
he went see whta the problem he am the best student in the classroom.
Flu.
SubjVerbiy
Jumble
We play badminton every evening. she did n't intend to buy anything unfortu- nately she has poor impulse control...
badminton every We evening play.
Coh.
unfortunately she has poor impulse control.
SentReorder
she did n't intend to buy anything
RmReIWords
The supermarket has various kinds of goods
The has various kinds of goods
Rel.
StoryReplace
The supermarket has various kinds goods
It nice day to hang out
The boy got the gift he always wanted, he was so happy
The hoy got the gift he always wanted, he was so sad
Antonym
Log.
they took me down to the lake i threw my line out and caught several worms...
they took me to the moon. i threw my line out and caught several stars
Commonsense
felt really angry, talked to my estranged father, and he gave me a gun! But knew violence is not solution here
I felt upset and talked to my father about it He advised me to handle the situation calmly so I decided not to resort to violence.
Int.
BlanderNarrative
Table I: Summary of perturbations that target a story quality aspect: Fluency (Flu.), Coherence (Coh.), Relatedness (Rel), Logicality (Log.), and Interestingness (Int.). For "Relatedness" example stories, they are conditioned on the title "I always go the local supermarket". Underlined perturbations original methods propose.
then remove them from the story, and 2) Story Re- placement (StoryReplace), where we substitute the original story with another story from a different story condition. To select "comparable" story, choose a story with where its likelihood is similar to the original story.?
Dataset
Condition
Story
ROC
[FEMALE] dad
we sat a spot and waited for days ...
took me fishing.
WP
tell me a story where the first line and last line ...
as walked into the house was assailed by smell aging
Table 2: Sampled examples of given story condition and its generated story for each dataset.
Logicality focuses on the extent to which the story complies with commonsense. Perturbations targeting logicality introduce elements into the story that contradict commonsense. We adopt one approach from Guan al. (2021b): Antonym, where we randomly replace the word with its antonym; and propose new approach: Common- sense, where we use ChatGPT to modify some story elements to violate commonsense.
Experiments
4.1 Benchmarks
We use the generated stories and human ratings collected by Xie et al. (2023) on two story datasets: ROCStories (ROC; Mostafazadeh et al. (2016); 5- sentence simple stories) and WritingPrompts (WP; Fan et al. (2018); longer fictional stories written by users on Reddit). The story condition (c) for ROC is the leading sentence; for WP, it is the short paragraph that describes the idea of the story, which is called "prompt". We present two example stories from the two datasets in Table 2.
Interestingness measures the degree of pre- dictability in the progression of events within a story, representing a highly subjective aspect. We propose one approach: BlanderNarrative, where we use ChatGPT to modify story to make the narrative less interesting.
Xic al. (2023) experiment with 6 story genera- tion models that cover large models with prompt- based learning (e.g., GPT-3), smaller fine-tuned models (e.g., BART) and other methods that in-
The ChatGPT instructions for the aforemen- tioned perturbations are detailed in Appendix A. For Typo, Jumbo and Antonym, we can control the degree of perturbation, and this parameter is tuned in 5.1.
'Xie et al. (2023) also collected human judgements for CNN-Dailymail, but we do not use them in our study for two reasons: 1) the stories depict real-world events rather than fictional narratives, and 2) most of the language models we test have been trained on this dataset, and so there are potential circularity issues.
We calculate the likelihood of original story and a candidate story without considering their story conditions. "We use OpenAl API with the model gut-b.5-

---

corporate planning and commonsense (Xu al.. 2020; Guan et al., 2020, 2021a; Tan et al., 2021). They then conduct human evaluation on five as- pects, judged using an ordinal scale from (worst) to 5 (best). Two distinct groups of annotators were recruited, comprising in-house PhD students and crowdworkers. The results obtained from both groups were found to be similar, indicating the ro- bustness and reliability of the annotation process.
Arch.
Model
Size
#Data
Objectives
BART
406M
160GB
Denoising
En-De
FLAN-T5
11B
Denoising
BLOOM
7B
366BT
LM
LLaMA
65B
1.TTT
LM
De
OPT
66B
180BT
LM
GPT-3.5
175B
300BT
I.M
Table Summary of PLMs, classified by their archi- tecture (Arch.) encoder-decoder (En-De) decoder (De). "Size" indicates model parameters. #Data indi- cates the pre-trained data scale ("GB" "gigabyte"; "BT" = "billion tokens"; and "TT" "trillion tokens"). "LM" indicates causal language modeling objective.
The judgment from the first group is used for preliminary exploration of optimal settings, such as assessing the effectiveness perturbation methods and language models ($ 5.1). The judgment of the second group is used for the final comparison of our approach with existing evaluation metrics ($ 5.2).
alignment approach, which is also called "consis- tency" in the original paper. For UNIEVAL, the question answering models are trained on text sum- marization and dialogue generation tasks. We mod- ify the questions to adapt UNIEVAL for evaluating diffcrent aspects of stories as the authors demon- strate the zero-shot transfer capability. Please refer to Appendix B for our questions. For generative metrics, we select BARTScore and GPTScore. We use the reference-free version of BARTScore (i.e., - s), and employ text-davinci-003 from OpenAl as the backbone of GPTScore with spe- cific prompts for different story aspects. Prompts for GPTScore can be found in Appendix C.
4.2 Language Models
We select of representative PLMs to compute DELTASCORE. For encoder-decoder PLMs, we use BART and FLAN-T5 (Chung et al., 2022). For decoder PLMs, we use BLOOM (Scao et al., 2022), LLaMA (Touvron et al., 2023), OPT (Zhang et al., 2022a), and GPT-3.5.7 We use the largest possible variant whenever possible as we found larger mod- els tend to work better in preliminary experiments. We present summary of these models in Table 3.
4.3 Compared Evaluation Metrics
To comprehensively compare DELTASCORE with other existing evaluation metries, we select repre- sentative evaluation metrics from each of the three categories mentioned in $ 2.1.
We summarise all these metrics Table 4, show- ing whether they: require additional training or ground truth reference; are originally introduced for story evaluation; and can measure fine- fine-grained story aspects.
For similarity metrics, we run experiments for BLEU, BERTScore and MoverScore. For dis- criminative metrics, have UNION, MANPLTS, StoryER, CTC and UNIEVAL. Additionally, we include zero-shot GPT-3.5 using simple instruc- tions as prompt to gather judgements for the five specific aspects (Chiang and Lee, 2023). This ap- proach is referred to as GPT3.5Eval; detailed in- structions can be found in Appendix D.
Results
We evaluate Kendall correlation the story level. which involves comparing the predicted metric score versus the aggregated human rating for each story on a specific aspect. We use this as our pri- mary metric due to the non-linear relationship be- tween automatic and human metrics, as well as the ordinal scale employed in human judgments (Kendall, 1938). We explore different settings of our approach in $ 5.1 and present comparison of our best approach with existing evaluation metrics in $ 5.2. Note that we use two different set of judg- ments, as explained in $ 4.1, to avoid tuning and testing on the same test set.
Since UNION, MANPLTS and StoryER are all originally designed for story evaluation, use their released models without fine-tuning for our experiments. For CTC, we use the reference-free
'For both groups. they gather assessments for 140 sto- ries ROC and 100 stories WP, with each story being evaluated by three annotators. Annotations on human-written stories are excluded as they could introduce bias in favor reference-hases metrics. As result, this leaves us with 120 stories for ROC and 80 stories for WP, respectively. We use text-davinci-ONe in our experiments.

---

Objective Metric
FT
B/F
MS
ROC Corrolation
BLEU
X
B
X
30
Similarity
BERTScore
X
B
X
MoverScore
B
x
UNION
F
/
x
10
MANPLTS
x
0.2
0.4
1.0
45
Discriminative
StoryFR
F
x
WP Correlatt lation
Correlation
40
CTC
V
F
x
/
35
UNIEVAL
30
BoRe
F
/
Generative
25
GPTScore
X
F
0.4
N/A
GPT3.5Eval
Perturbation Degree
Typo
Antonym
Table 4: Statistics of compared evaluation metrics. "FT" indicates whether the metric requires additional syn- thetic data fine-tune on. "B/F" indicates whether the metric is reference-based (B) or reference-free (F). "ST" indicates whether the metric originally designed for story evaluation. "MS" indicates whether the metric produces scores consider multiple aspects.
Jumble
w/o perturbation
Figure 2: Impact of perturbation degree with LLaMA on in-house judgements for measuring coherence.
turbation methods and present the results over ROC and WP in Figure 2. In the case of Typo, the degree pertains to the percentage of characters that we opt to swap. Concerning Jumble, we shuffle tokens within certain text span while the span length is controlled by the degree. place the token with its antonym under the specified probability (degree). As before, we use LLaMA as the PLM and focus on evaluating coherence here. Interestingly, Typo appears to be relatively stable and unaffected by the perturbation degree, where else Jumble and Antonym work better with more aggressive perturbation. Based on these results, we the perturbation degree 0.4, 0.9, and 0.8 for Typo, Jumble, and Antonym respectively for both ROC and 8P.8
5.1 Preliminary Exploration
Perturbation Methods We commence by show- casing the comparative performance of various per- turbation methods ($ 3.2) in relation to human judg- ments across the five aspects, as demonstrated in Table For this analysis, we employ LLaMA as the PLM. The notation "w/o perturbation" denotes the calculation of story likelihood directly under LLaMA, without any perturbations applied. Our findings revealed intriguing results. Notably, observed that perturbations specifically designed to target particular aspect did consistently ex- hibit a higher correlation with human judgments for that aspect. Furthermore, our analysis indi- cates that measuring interestingness is particularly challenging, the correlation numbers associated with this aspect are generally lower compared to last and perhaps
for Antonym, we re-
Language Models We next present DELTAS- CORE results using different PLMs in Table 6. We use the top performing methods with the opti- mal degrees determined our previous analysis. Encouragingly, across different PLMs and story aspects, we see that DELTASCORE outperforms vanilla likelihood ("w/oo perturbation") in almost all instances, suggesting that measuring story qual- ity using likelihood difference is generally better approach than using its likelihood directly. Broadly speaking, Jumble is the most consistent perturba- tion method: in ROC is the consistently the best performer, while in WP it is either the best or sec-
the other aspects. Finally, most surprising observation is that small set of perturbation methods, namely Typo, Jumble, and Antonym, exhibit strong performance in evaluating most aspects.
Perturbation Degree In the preceding phase, we carried out the most intense perturbation for Jum- ble and Antonym, in which the perturbation was applied the entire sentence, and random selec- tion of half of the characters for Typo. light their strong performance, we now investigate impact perturbation degree using the these per-
"The results in the previous subsection (Table 5) use these perturbation values.

---

ROC
WP
Target Aspect
Perturbation
Flu.
Coh.
Rel.
Log.
Int.
Flu.
Coh.
Rel.
Log.
Int.
N/A
w/o perturbation
17.3
31.0
22.8
35.2
20.2
26.8
27.0
32,1
34.6
17.8
Typo
14.2
31.9
23.5
36.0
23.2
36.5
42.9
38.1
47.8
35.5
Flueney
SubjVerbDis
5.5
13.9
8.8
13.0
7.8
13.0
15.9
4.7
10.7
6.1
Jumble
17.6
37.1
16.8
34.1
22.5
35.9
36.0
31.9
36.3
28.1
Coherence
SentReorder
4.4
17.7
4.8
20.3
8.9
18.6
27.3
20.5
15.1
25.4
RmRelWords
14.7
30.9
25.6
32.6
21.7
7.1
31.9
35.4
32.8
16.1
Relatedness
StoryReplace
1.8
7.0
16.0
21.3
8.6
28.5
26.6
27.6
38.5
23.8
Antonym
14.4
16.9
11.4
11.6
33.5
35.0
35.0
25.3
Logicality
Commonsense
20.1
27.1
20.7
34.9
21.4
16.0
12.3
17.4
8.8
Interestingness
BlanderNaarative
8.1
19.3
2.1
15.1
3.9
14.7
13.3
8.4
18.1
9.5
Table Story-level Kendall correlation ((r) between DELTASCORE with LLaMA and in-house judgements. We highlight instances where DELTASCORE outperforms vanilla likelihood ("w/o perturbation"),
Correlation on ROC
Correlation on WP
50
30
40
20
10
10
Flu. Coh. Rel. Log. Int.
Flu. Coh. Rel. Log. Int.
DeltaScore (Typo)
BLBU
UNION
CTC
GPTScore
DeltaScore (Jumble)
MANPLTS
UNIEVAL
5PTl
DeltaScore (Antonym)
MoverScore
StoryER
BARTScore
Figure 3: Absolute value of Story-level Kendall correlation (T)) between different metrics and crowdworker ratings. Higher bar indicates better performance. Red bars indicate DELTASCORE. Blue bars indicate similarity based metrics. Green hars indicate discriminative metrics. Purple bars indicate generative metrics. Gray bar indicates GVvl.
ond best performer, depending on the PLM. This observation aligns with the findings presented in Table 5, providing further confirmation that the Jumble perturbation method demonstrates effec- tiveness in measuring various story aspects. When examining the correlation magnitudes for differ- ent story aspects, it is evident that interestingness consistently cxhibits lower values, reaffirming its inherent difficulty in measurement. There are, how- ever, some curious exceptions: in ROC the correla- tion for fluency and relatedness is particularly low. We do not have strong hypothesis of these ob- servations, but will note that the language of ROC stories are somewhat formulaic and possibly dif- ferent to the language of the pre-training data. For relatedness, the story condition ROC the first sentence, and is a rather artificial condition to set
the "topic" for story generation.
An unsurprising observation is that larger mod- els tend to exhibit stronger correlations, with GPT- 3.5 and OPT performing the best among the PLMs. BLOOM and FLAN-TS fall in the middle range, while BART shows the lowest correlation scores. Upon comparing GPT-3.5 and OPT, we observe a slight advantage for OPT despite its smaller model size and pre-training data. This finding suggests that beyond a certain scale, the benefits of further scaling may become less significant.
5.2 Comparison with Other Metrics
We next compare DELTASCORE with other evalua- tion metrics Figure 3. Note that in this compari- son, we utilize OPT as the chosen PLM. consider- ing its superior performance, along with the same

---

ROC
WP
Metric
Flu.
Coh.
Rel.
Log.
Int.
Flu.
Coh.
Rel.
Log.
Inf.
DELTASCORE (with BA
Rarrgge
406M)
w/o perturbation
4.5
14.6
11.6
14.5
2.3
15.0
14.5
22.4
24.5
10.6
Jumble
11.4
21.9
14.6
19.5
13.1
30.0
21.6
27.3
33.6
21.2
Typo
6.3
17.7
19.8
17.5
4.7
24.5
26.9
30.2
36.6
24.3
Antonym
5.7
8.3
7.3
5.4
6.9
28.0
26.4
37.6
31.8
25.7
DELTASCORE (with FLAN-TS XXL 11B)
w/o perturbation
16.2
19.4
23.1
19.9
14.1
.4
20.8
4.8
Jumble
24.5
20.2
27.9
20.6
31.7
28.3
34.4
21.8
Typo
11.3
21.8
28.0
14.7
29.0
25.7
33.1
31.8
18.0
Antonym
10.7
12.9
10.1
7.9
33.1
34.0
30.6
34.2
29.2
DELTASCORE (with BLOOM 7B)
w/o perturbation
12.4
24.1
13.3
25.5
10.2
24.8
21.3
27.5
29.2
12.6
Jumble
25.0
35.5
14.2
33.9
25.4
34.2
34.8
30.7
33.6
23.1
Typo
14.3
31.6
12.2
30.3
14.5
38.4
40.9
38.7
44.7
29.5
Antonym
10.2
12.0
9.4
10.3
12.2
27.4
31.5
32.9
30.6
DELTASCORE (with L.LaMA 65B)
w/o perturbation
17.3
31.0
22.8
35.2
20.2
26.8
27.0
32.1
34.6
17.8
Jumble
19.7
36.9
15.8
35.9
23.0
36.8
35.3
32.9
35.5
27.7
Typo
14.2
31,9
23.5
36.0
23.2
34.3
42.2
35.2
44.0
31.8
Antonym
13.3
15.9
10.6
14.2
11.0
31.3
37.9
35.0
36.2
31.6
DELTASCORE (with OPT
66B)
w/o perturbation
17.4
30.6
20.2
32.6
15.8
27.5
24.5
32,2
31.0
15.7
Jumble
27.4
44.2
21.6
41.4
30.0
37.9
38.7
35 .4
39.5
32.0
Typo
17.9
39.6
21.3
38.7
22.8
39.0
41.9
38,0
45.6
30.4
Antonym
15.8
19.0
13.8
14.9
12.9
34.0
40.3
36.0
39.4
36.7
DELTASCORE (with
GPT-3.5
175B)
w/o perturbation
21.3
31.2
18.7
28.6
18.2
36.2
32.9
36.0
37.1
23.3
Jumble
18.7
34.0
23.3
38.5
29.7
35.0
36.6
36.1
37.7
33.5
Typo
17.0
35.1
11.7
32.2
26.3
41.7
46.7
37.5
45.0
37.3
Antonym
19.1
21.5
12.9
20.5
13.9
36.1
41.2
41.3
39.6
38.2
Table 6: Absolute value of Story-level Kendall correlation ([r) between different metrics and in-house judgements. We bold the best scores for each aspect and highlight instances where DELTASCORE improves over vanilla likelihood ("w/o perturbation").
top-performing perturbation methods. The results of our evaluation very promising: DELTAS- CORE consistently outperforms all competitor met- rics across all story aspects. Jumble stands as the most effective perturbation method among the three. The similarity metrics generally has the low- est performance, highlighting the inadequacy of reference- baseed metrics for story evaluation, which aligns with previous research findings (Guan and Huang, 2020; Xie et al., 2023). Among the dis- criminative metrics, CTC and UNIEVAL show rel- atively strong competitiveness, although they still fall behind DELTASCORF. The performance of generative scores inconsistent. GPTScore shows strong performance in evaluating logicality and in- terestingness, especially in ROC, where it performs similarly to DELTASCORE. However, its effective- ness is limited in other scenarios. More detailed scores be found in Table 8.
Discussion and Conclusion
Initially, our aim was to investigate various types of perturbations for assessing fine-grained aspects of storytelling. But seeing that performance of each metric does not vary much across different aspects in Figure it suggests these aspects may be somewhat inter-correlated. Also, our findings revealed that one of the simplest perturbation meth- ods, namely Jumble, exceptionally effective measuring most aspects. One hypothesis could be that Jumble is functioning as a normalisation factor to modulate word frequency and sentence length effects when estimating sequence quality. This finding aligns with prior study that used sequence probabilities for measuring sentence acceptability (Lau al., 2020). They found that it is important normalise the probabilities and introduced various normalization techniques to mitigate the impact word frequency and sentence length. Jumble can

---

be interpreted an alternative normalisation tech- nique. Given this insight, may also mean that DELTASCORE has broader application beyond the evaluation of story quality. For instance, it could be used to score sentences in machine translation and summarization.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang. Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better Ilm-based evaluators through multi-agent debate. CoRR, abs/2308.070 0..200..
Hong Chen, Duc Vo, Himoy Takamura, Yusuke Miyao, and Hideki Nakayama. 2022. StoryER: Automatic story evaluation via ranking, rating and reasoning. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
In conclusion, we introduce DELTASCORE, novel approach to assess fine-grained story aspects by comparing the likelihood difference between the original story and a perturbed version using a pre- trained language model. Surprisingly, we discov- ered that a small set of perturbation methods excel in measuring the majority of story aspects. Further- more, our findings demonstrate that DELTASCORE shows stronger correlations with human judgments compared to a range of existing metrics across two different story domains.
Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story gen- eration. In Proceedings of the 29th International Conference on Computational Linguisties, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 5794-5836. International Committee on Computational Linguistics.
David Cheng- Han Chiang and Hung-yi Lec. 2023, Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guisties (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 15607-15631. Asso- ciation for Computational Linguistics.
Limitations
Our study only investigates a limited range of per- turbations, and we acknowledge that there may be other forms of perturbations that could work better. The field of evaluation metrics is rapidly evolving, with numerous contemporary evaluation metrics in- troduced recently, such as G-Eval (Liu et al., 2023) and ChatEval (Chan et al., 2023), which were not incorporated into the comparative evaluation met- rics within this study.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao. Yanping Huang, Andrew M. Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finen finetuned language models. CoRR, abs/2210. 11416.
Acknowledgements
We extend our thanks to the reviewers for their valuable feedback, which has greatly contributed the improvement of this work. Zhuohan Xie is supported by Melbourne Research Scholarship, and would like to expresses his sincere appreciation to the program.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing. and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are fhw-shot learners. Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020. December 6-12, 2020, virtual.
George Doddington. 2002. Automatic evaluation of ma- chine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second Interna- tional Conference on Human Language Technology Research, HLT '02, page 138-145 San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski Noah A. Smith, and Yejin Choi. 2022. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pages 7250-7274, Dublin, Ireland. Association for Computational Linguistics.

---

Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
lia Tsvetkov, 2022. On the blind spots of model- based evaluation metrics for text generation. CoRR. abs/2212. 0../2212.10022 10020.
Marzena Karpinska, Nader Akoury, and Mohit Tyyer. 2021. The perils of using Mechanical Turk to evalu- ate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. CoRR, abs/2302.04166.
Sarik Ghazarian, Zixi Liu, Akash S M, Ralph Weischedel, Aram Galstyan, and Nanyun Peng. 2021. Plot-guided adversarial example construction for evaluating open-domain story generation. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4334-4344, Online. Association for Computational Linguistics.
Marzena Karpinska, Nishant Raj, Katherine Thai, Yix- iao Song, Ankita Gupta, and Mohit Tyyer. 2022. DEMETR: Diagnosing evaluation metries for trans- lation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9540- 11600-966...00 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
M. G. Kendall. 1938. A new measure of rank correla- tion. Biometrika, 30:81-93.
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A Knowledge-enhanced pre- training model for commonsense story generation. Transactions of the Association for Compitational Linguistics, 8:93-108.
Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020. How fu- riously can colorless green ideas sleep? sentence acceptability in context. Transactions of the Associa- tion for Computational Linguistics, 8:296-310
Jian Guan and Minlie Huang. 2020. UNION: An Un- referenced Metric for Evaluating Open-ended Story Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP),, pages 9157-9166, Online. Association for Computational L Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising eqquence-to-sqquence pre-training for natural language generation, translation, and com- prehension. In Proceeding of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguisties.
Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wen- biao Ding, and Minlic Huang. 2021a. Long text generation by modeling sentence-level and discourse- level coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). ). pages 6379-6393, Online. Association for Computational Linguistics.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversar- ial attack against BERT using BERT. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193 6202, Online. Association for Computational Linguistics.
Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wen- biao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021b. OpenMEVV: A benchmark for eval- uating open-ended story generation metries. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 11th Inter- national Joint Conference Natural Language Pro- cessing (Volume 1: Long Papers), pages 6394 6407. Online. Association for Computational Linguistics.
Chin-Yew Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment. CoRR, ab6/2303.16634.
Rujun Han, Hong Chen, Yufei Tian, and Nanyun Peng. 2022. Go back in time: Generating flashbacks in stories with event temporal prompts. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1450-1470, Seattle, United States. Association for Computational Linguistics.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby. Di Jin, and Yanjun Qi. 2020. TextAttack: A frame- work for adversarial attacks, data augmentation, and adversarial training in NLP. Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126. Online. Association for Computa- tional Linguistics.
Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Ku- mar, Kyunghyun Cho, James R. Glass, and Yu-

---

Nasrin Mostafazadeh, Lucy Vanderwende, Wen-tau tau Yih, Pushmeet Kohli, and James Allen. 2016. Story cloze evaluator: Vector space representation evaluation by predicting what happens next. In Proceedings of the 1st Workshop on Evaluating Vector-Space Repre- sentations for NLP, pages 24-29, Berlin, Germany. Association for Computational Linguistics.
Brian Thompson and Matt Post. 2020. Automatic ma- chine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing ( (EMNLP), pages 90-121, Online. Association for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, Abss/2302.19777
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008.
Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022. T5score: Discriminative fine- tuning of generative evaluation metries. CoRR, abs/2212.05726..
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn, Res., 11:140:1-140:67.
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. The next chapter: A study of large language models in storytelling. In Proceedings of the 16th Inter- national Natural Language Generation Conference, pages 323-351, Prague, Czechia Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics.
Zhuohan Xie, Jey Han Lau, and Trevor Cohn. 2021, Ex- ploring story generation with multi task objectives in variational autoencoders. In Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association, pages 97-106, Online. Aus- tralasian Language Technology Association.
Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021. Perturbation CheckLists for evaluating NLG evaluation metrics. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7219-7234, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung. Anima Anandkumar, and Bryan Catanzaro. 2020. MEEGATRON CNNTR:: Control- lable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2831-2844 Online. Association for Computational Linguistics.
Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2023. A survey of evaluation metries used for NLG systems. ACM Comput. Surv., 55(2):26:1-26:3.
Kevin Yang. Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 4399-4479, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick. Suzana Ili'e, Daniel Hesslow, Roman Castagn's e, Alexandra Sasha Luccioni, Franc- cois Yvon, and et al Matthias Gallé. 2022. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2211.05100
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Informa- tion Processing Systems 2021, NewrlPS 2021, De- cember 6-14, 2021, virtual, pages 27263-27277
Bowen Tan, Zichao Yang, Maruan AI-Shedivat, Eric Xing, and Zhiting Hu. 2021. Progressive generation of long text with pretrained language models. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3313-4324 Online. Association for Computational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe. Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin,

---

Todor Mihaylov, Myle Ott, Sam Shleifer. Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022a. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01068..
A Perturbation Prompts
We use following prompts for perturbations when we apply API with GPT-3.5-turbo.
RelevantWords: Find all words in the given story that is relevant to the given title. Please only print words in the given story, and sep- arate them by " "title": (title), "story": {story)
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinherger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Commonsense: Revise the following story such that certain elements does not make sense. The revision should be minimal, e.g.. by changing a few words. "story": (story)
Zhexin Zhang. Jiaxin Wen, Jian Guan, and Minlie Huang. 2022h. Persona-guided planning for con- trolling the protagonist's persona in story generation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies. pages 3361,, Seattle, Seattle, United States. Association for Computational Linguistics.
BlanderNarrative: Revise the following story to make it less interesting (c.g., expected end- ing, no plot twist). The revision should be minimal. "story": (story)
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. et al. 2023. survey of large language models. arXiv preprint arXiv:2303.18223.
We present several examples of perturbed stories by GPt--.5-turbo Table We observe that Blan- derNarrative does not significantly alter the origi- nal story. This observation is in line with previous findings that BlanderNarrative does not effectively impact interestingness in Table 5. We speculate that this outcome may be attributed to the inherent simplicity of most stories, which limits the extent to which GPT-3.5-turbo can modify them to reduce their level of interest.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Lin- guistics.
B UNIEVAL Questions
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
We ask the following questions for each aspect. Note that we try to use the narrative/oocabularyy as close to the original questions Zhong et al. (2022) use in their efforts as possible.
Fluency: Is this fluent utterance?
Coherence: Is this a coherent utterance?
Relatedness: Is this claim consistent with the document?
Is this utterance consistent with the commonsense?
Logicality:
Interestingness:
Is this an interesting utter-
ance?
C GPTScore Prompts
We use the following prompts for each aspect. Note that we try to use the narrative/vocauulary as close to the original prompts (Fu et al., 2023) use in their efforts as possible.

---

Perturbation RmRe|Words
Title
Original Story
Perturbed Story
the man dered extra cheese.
the loved cheese and always dered extra his favourite was pizza with extra cheese his favourite pizza place and ordered his usual. when the pizza arrived he that there was angry that he threw the pizza across the room.
loved and always his favourite one day he went to his favourite place and ordered his usual. when the pizza arrived he that there the man across the room.
day he went
angry that he the pizza
cheese the
RmReIWords
the buy
the boy cried because he was sad and scared his parents were getting di- vorce and he did n't know what was going to happen to him he was go- ing to have leave his home and his friends and move to new house his mother said that he could choose where he wanted to live. but his father said that he had live with him. the boy did n't know what to do
because was and were getling and he was going happen. he going to have to his and his friends and to a. his mother said that he could he to but his father said that he had to him. the boy did n't know what to.
cried.
Commonsense
[FEMALE] dad took me fishing
they took me down to the lake i threw my line out and caught several worms, i turned in one worm and caught catfish... i told my dad and he took me home and raised it for dinner.
they took me down to the sky threw my book out and caught several words. turned in one word and caught space- ship i told my mom and she took me home and raised for dinner.
Commonsense
[FEMALE] was at the mall
she was walking to the food court when she saw a man who looked lost she went up to him and asked if he needed help. the man told her he was looking for his wife and daughter. [FEMALE] took him to the food court and pointed them out to him.
she was flying to the food court when she saw a bird who looked lost she went to it and asked if it needed help the bird told her it was looking for its wife and daughter [FEMALE] took it to the food court and pointed them out to it
BlanderNarrative
[FFMALF] went out to cat with her friends.
she ordered a hurger and fries, when the food arrived, she was really disap- pointed the burger was tiny and the
she ordered a burger and fries. when the food arrived, it was just as she expected the burger was small and fries were lukewarm.
fries were cold and soggy
BlanderNarrative
[MALE] is with his friends at bar.
[MALE] drinks beer and cats few [MALE] feels very sick , [MALE] embarrassed that he drank so much.
he decides to buy
decides buy a beer . [MALE] drinks beer and [MALE] feels sick.. [MALE] regrets drinking too much
few more
Table 7: We show some examples of perturbed stories where we use GPT-3355ttubb for perturbation.
Fluency: Generate fluent story for the given title: (title), and story: (story)
Fluency
The goal of this task is to rate story frag-
ment.
Note: Please take the time fully read and under- stand the story fragment.
Coherence: Generate a coherent story for the given title: [title), and story: (story)
Story fragment:
Relatedness: Generate story related to the given title: (title), and story: (story)
{ (Storry)) }
How fluent is the text of the story fragment? (on a scale of 1-5, with being the lowest)
Logicality: Generate story that adhere to commonsense for the given title: (title), and
The goal of this task is to rate story Coherence
story: [story]
fragment.
Interestingness: Generate interesting story for the given title: [title,, and story: (story)
Note: Please take the time to fully read and under- stand the story fragment.
Story fragment:
D GPT3.5Eval Instructions
{ (Story))
We use the following instructions for each aspect, following Chiang and Lee (2023).
How coherent is the text of the story fragment? (on a scale of 1-5, with being the lowest)

---

The goal of this task is to rate story fragment.
Relatedness
Note: Please take the time to fully read and under- stand the story fragment.
Story title:
[(Tile))
Story fragment:
([Story))
How related is the text of the story fragment the title? (on scale of 1-5, with I being the lowest)
Logicality The goal of this task is to rate story fragment.
Note: Please take the time to fully read and under- stand the story fragment.
Story fragment:
[[toory])
How logically correct is the text of the story frag- ment? (on a scale of 1-5, with being the lowest)
Interestingness The goal of this task is to rate story fragment.
Note: Please take the time to fully read and under- stand the story fragment.
Story fragment:
[[Story))
How interesting the text of the story fragment?
(on a scale 1-5, with being the lowest)

---

ROC
WP
Metric
Flu.
Coh.
Rel.
Log.
Int.
Flu.
Coh.
Rel.
Log.
Int.
Similarity Based Metries
BLEU
3.4
4.4
0.8
4.6
0.4
13.4
11.4
9.6
11.2
7.2
Score
3.5
5.0
14.0
5.7
7.3
31.8
26.7
24.0
28.5
24.6
MoverScore
3.6
6.5
15.7
8.0
11.2
16.4
17.2
20.1
20.7
23.7
Discriminative Metries
UNION
12.8
17.4
18.1
22.9
MANPLTS
32.8
23.2
14.7
12.4
4.6
StorEER
4.0
14.1
17.9
CTC
22.9
27.3
14.3
11.1
8.3
45.9
51.6
40.3
53.1
UNIEVAL
32.2
31.7
23.7
20.0
18.8
39.3
41.3
38.6
50.7
39.1
Generative Metries
BARTScore
24.9
11.6
10.9
11.3
14.6
32.2
25.6
31.0
30.2
23.0
GPTScore
21.2
20.9
21.3
23.9
22.0
29.3
32.6
30.8
35.3
28.9
5Pvl
23.9
33.3
10.5
11.8
4.8
21.1
34.6
19.2
12.8
1.2
DELTASCORE
Typo
35.9
32.0
28.6
21.8
22.1
44.8
45.8
36.8
50.1
36.6
Jumble
43.2
42.4
27 4
25.6
23.9
56.1
55.5
45.9
60.9
48.5
Antonym 16.5
21.9
8.4
11.9
9.3
55.2
51.8
36.9
56.0
45.9
Table 8: Absolute value of Story-level Kendall correlation (/7)) between different metrics and crowdworker ratings. We bold the best scores each aspect.


Reviews:
[{"review_id": "iZHG2iYzE4", "reviewer": "Reviewer_5YuF", "summary": "", "questions": "1. see the above reasons to reject\n2. it seems that the likelihood regarding the frequency of typos are non-linear. any empirical validations? \n3. any other works also used perturbation for evaluation purposes in NLP field in general?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your review, we have following responses regarding the issues/questions you point out.\n\n**Responses To Issues:**\n\n*1. lacks of intuition of the connection between why perturbation-based evaluation could be useful and the model working mechanism or human evaluation. Although the effectiveness is validated empirically. It would be helpful to see more discussion related to why this perturbation-based method is useful.*\n\nOur idea is mostly based on the intuition that higher quality text will be affected more by perturbations than lower quality text (shown in Figure 1) and our empirical results validate this. This perturbation-based mechanism also allows a straightforward way to inject inductive bias into evaluation metrics, namely that we expect some perturbations to have a more marked effect on a high versus low-quality texts.\n\n*2. it would be interesting to include a baseline that is zero shot prompting gpt3.5 asking the model about the 5 aspects and compare with human annotations.*\n\nThank you for this suggestion. We have done some experiments for zero-shot prompting with simple instructions to ask GPT-3.5 to evaluate the stories on five aspects directly, on crowdsourced data which is described in Table 6. The Kendall correlation results are as follows.\nFor ROC, we have correlations on fluency: 23.90, on coherence: 33.27, on relatedness: 10.49, on logicality: 11.82, on interestingess: 4.78\nFor WP, we have correlations on fluency: 21.10, on coherence: 34.55, on relatedness: 19.20, on logicality: 12.81, on interestingess: 1.22\nThis zero-shot prompting approach shows better performance in terms of fluency and coherence than recently proposed BERTScore and BARTScore, but it is still behind DeltaScore in all aspects.\nWe will include this new set of results as another baseline in the revision.\n\n**Responses To Questions:**\n\n*1. see the above reasons to reject*\n\nSee above responses.\n\n*2. it seems that the likelihood regarding the frequency of typos are non-linear. any empirical validations?*\n\nAssuming that you meant the relation between the correlation and the perturbation degree in Figure 2, then yes the relation appears to be non-linear. Intuitively, it may be because of the subword tokenisation. Introducing too many typos will disrupt subword tokenisation too much and result in unreliable probability estimates, hence the sweet spot between 0.4 and 0.6.\n\n*3. any other works also used perturbation for evaluation purposes in NLP field in general?*\n\nTo the best of our knowledge, we were the first to use perturbations directly for evaluating generated texts by the time of writing this paper; however, there are other works such as Ribeiro et al., 2020 and Sai et al., 2021 use perturbations to assess the robustness of evaluation metrics. We discussed this in Sec 2.2 (line 144-149).\nPerturbations are also used in training LLMs, that is, masking, dropping and substituting tokens are commonly used, e.g., in T5 / BART style encoder-decoder approaches where the LLM is trained to **denoise** the input.\n\n**Responses To Presentation Improvements:**\n\n*1. line 258 instructions typo*\n\nThank you for pointing this out, we will revise it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- comprehensive experiments and compared with a set of baselines\n", "reasons_to_reject": "- lacks of intuition of the connection between why perturbation-based evaluation could be  useful and the model working mechanism or human evaluation. Although the effectiveness is validated empirically. It would be helpful to see more discussion related to why this perturbation-based method is useful. \n- it would be interesting to include a baseline that is zero shot prompting gpt3.5 asking the model about the 5 aspects and compare with human annotations. \n", "questions_to_authors": "1. see the above reasons to reject\n2. it seems that the likelihood regarding the frequency of typos are non-linear. any empirical validations? \n3. any other works also used perturbation for evaluation purposes in NLP field in general?", "paper_topic_and_main_contributions": "- introduced DELTASCORE, a  method that employs perturbation techniques for the evaluation of nuanced story aspects.\n- the central proposition is that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos).\n- measuring the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained LMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 258 instructions typo"}, "extra_text": {}}, {"review_id": "lhH5GLG0jO", "reviewer": "Reviewer_pCUx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your review, we have following responses regarding the issues/questions you point out.\n\n**Responses To Issues:**\n\n*1. While the method works across all aspects for a subset of perturbations, the initial goal of finding a targeted method for individual aspects remains open. The method seems to work empirically but it is unclear why.*\n\nWe agree that it is a curious finding. In section 6 we discussed and hypothesised that jumble might be functioning as a normalisation factor to regulate word frequency and sentence length for estimating quality, thus it provides great performance with DeltaScore.\nIt is true that finding a targeted method for individual aspects remain open as it is trikcy to find approaches to target on one aspect while keeping the others the same, and this paves way for future theoretical work/analyses.\n\n*2. On lines L375-L386, talking about the perturbation degrees, it is not entirely clear what those numbers mean. A practitioner can likely guess what they are supposed to mean but it would be better to be explicit.*\n\nWe use perturbation approaches provided by the checklist (Ribeiro et al., 2020) and openmeva papers (Guan et al., 2021).\nFor typo, we choose certain number of characters to swap, number = story length * perturbation degree; for jumble, we choose a span of a random length to shuffle tokens, length = story length * perturbation degree; for antonym, we change the token to its antonym using wordnet under the certain probability (perturbation degree).\n\n*3. Similarly, it is not clear what perturbation parameters were used in Table 5.*\n\nFor Table 5, we were selecting methods before selecting the best performing degrees, the degree for typo, jumble and antonym are 0.5, 1.0, and 1.0, respectively.\nWe will include these details into the revision.\n\n**Responses To Presentation Improvements:**\n\n*1. L217: \"substitute\" -> I'd suggest saying \"transpose\" or \"exchange\" as \"substitute\" could be interpreted as a doubling of a letter*\n\nWe will change the word in the revision, thank you for the suggestion.\n\n*2. I think it would be helpful to include a table of dataset statistics such as number of annotated stories and number of raters for the data of Xie et al. (2023).*\n\nWe agree and will include this in the revision.\n\n*3. I think Fig. 3 would be a lot better as a table! As is, it is very difficult to glean more than a very high-level idea of the comparison.*\n\nThanks for the suggestion. We will add the table in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall I found the work to be very well written and presented. The arguments flow well and choices are well argued.\n\nThe method itself is well described and seems relatively simple to use and robust with respect to the choice of backbone PLM (see Table 6).\n\nI like that the evaluation procedure uses separate sets of human ratings for the preliminary study of parameter choices and the full comparison across PLM and with other metrics.\n\nThe initial goal seemed to be to have specific targeted perturbations for each aspect, which resulted in a mixed outcome (Table 5). I appreciate the attempt to explain the surprising performance of \"jumble\" across all aspects in Section 6.", "reasons_to_reject": "While the method works across all aspects for a subset of perturbations, the initial goal of finding a targeted method for individual aspects remains open. The method seems to work empirically but it is unclear why.\n\nOn lines L375-L386, talking about the perturbation degrees, it is not entirely clear what those numbers mean. A practitioner can likely guess what they are supposed to mean but it would be better to be explicit.\n\nSimilarly, it is not clear what perturbation parameters were used in Table 5.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This works proposes DeltaScore, an unsupervised metric for story generation. It is based on measuring difference in likelihood between a generated story and a specifically perturbed version of the same story under a generic pre-trained language model (PLM). The work proposes multiple perturbation methods targeting 5 aspects of evaluation: fluency, coherence, relatedness, logicality, and interestingness. \nThey evaluate DeltaScore on two story generation datasets and compute correlation to human judgments based on annotations from Xie et a. (2023). While most of the proposed perturbations do not consistently improve correlations for the specific aspect they target, the authors identify 3 perturbation methods that improve all aspects. The DeltaScore methods was tested with multiple PLM backbones and compared to   various existing evaluation metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L217: \"substitute\" -> I'd suggest saying \"transpose\" or \"exchange\" as \"substitute\" could be interpreted as a doubling of a letter\n\nI think it would be helpful to include a table of dataset statistics such as number of annotated stories and number of raters for the data of Xie et al. (2023).\n\nI think Fig. 3 would be a lot better as a table! As is, it is very difficult to glean more than a very high-level idea of the comparison."}}, {"review_id": "jZEBe0YEtc", "reviewer": "Reviewer_cnBq", "summary": "", "questions": "A. Did you find any bias in the perturbation results from ChatGPT? For example, how ChatGPT selected related words or stories for the Relatedness domain or revised commonsense output?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your review, we have following responses regarding the issues/questions you point out.\n\n**Responses To Issues:**\n\n_1. The quality of perturbation outputs was not validated. For example, the Blander Narrative method was shown as the least effective method, but we cannot be sure if that was due to the ChatGPT's output quality for \"making the narrative less interesting.\"_\n\nWe did some analyses in preliminary experiments to engineer/tune the instructions (to make sure the approach is optimal), but we agree that this can be validated more rigorously. We will perform error analyses on the generated output and include them in the revision.\n\n_2. Perturbation methods in Fluency would need further clarifications. For example, how did you select a character/word to alter?_\n\nWe use perturbation approaches provided by checklist (Ribeiro et al., 2020) and openmeva (Guan et al., 2021).\nFor typo, we choose certain number of characters to swap, number = story length * perturbation degree; for jumble, we choose a span of random length to shuffle tokens, length = story length * perturbation degree; for antonym, we change the token to its antonym using wordnet under the certain probability (perturbation degree).\nWe will include these details in the revision.\n\n**Responses To Questions:**\n\n*1. Did you find any bias in the perturbation results from ChatGPT? For example, how ChatGPT selected related words or stories for the Relatedness domain or revised commonsense output?*\n\nFor commonsense, it usually changes some entities to the unrelated ones to break the logic, such as \"she put the batter in the oven and waited for it to bake .\" -> \"she put the hat in the tennis court and waited for it to bake .\".\nWe did not notice any obvious patterns in terms of word selection, but agree it is possible that there could be some kinds of bias (e.g., cultural or domain or others) with a deeper analysis. We will include random example generations in the main paper (to help reader have a grasp on the overall quality) and release the full output for the community to enable future research in this direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough analysis: The paper tested their method across five different domains, using two perturbation methods each, including four newly introduced methods from the related studies. The results extensively compared their method with multiple existing methods for NLG evaluations. The analysis also covered the results from various encoder-decoder and decoder-only PLMs.\n- The methods presented in the paper were simple but effective. \n- Good presentation: Figures and Tables were effective. Reading the manuscript was enjoyable.\n", "reasons_to_reject": "I'd like to see further clarification on the quality of the perturbed/augmented data. \n- The quality of perturbation outputs was not validated. For example, the Blander Narrative method was shown as the least effective method, but we cannot be sure if that was due to the ChatGPT's output quality for \"making the narrative less interesting.\" \n- Perturbation methods in Fluency would need further clarifications. For example, how did you select a character/word to alter?", "questions_to_authors": "A. Did you find any bias in the perturbation results from ChatGPT? For example, how ChatGPT selected related words or stories for the Relatedness domain or revised commonsense output?", "paper_topic_and_main_contributions": "This paper presented a new evaluation method for model-generated narratives by comparing the likelihood of the next tokens from the original and the perturbed stories. The authors showed that their methods correlate better with human judgments than other conventional NLG evaluation metrics across five domains. The evaluation results were able to cover fine-grain and relatively simple to use.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new evaluation method, DeltaScore, for evaluating nuanced aspects of story generation models. The authors propose using perturbation techniques to assess the quality of a story's specific aspect (e.g., fluency), suggesting that the susceptibility of a story to certain perturbations (e.g., the introduction of typos) correlates with the story's excellence in that aspect. The quality of an aspect is measured by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models (LMs).

The reviewers generally appreciated the comprehensive experiments and comparisons with baseline methodologies. They also acknowledged that the paper was well-written and presented, with effective use of figures and tables. The simplicity and effectiveness of the proposed method were also noted as strengths.

However, some concerns were raised about the lack of intuition regarding the connection between perturbation-based evaluation and the model working mechanism or human evaluation. Reviewers suggested that a more thorough discussion about why this method is useful would be beneficial. The authors were also asked about the possibility of bias in the perturbation results from ChatGPT, and how it selected related words or stories for the Relatedness domain or revised commonsense output. One of the reviewers also noted that the quality of perturbation outputs was not validated.

In response to these concerns, the authors provided thorough rebuttals. They discussed their intuition behind the perturbation-based method and presented additional experimental results to support their claims. They also clarified the mechanisms of their perturbation methods and promised to include more details in their revisions. The authors acknowledged the need for rigorous validation of perturbation outputs and committed to performing error analyses in the future.

Overall, the reviewers found the study to be generally strong and sound, providing sufficient support for its claims. They were excited about the paper and believed it could deepen the understanding of certain phenomena or lower the barriers to an existing research direction. However, they also acknowledged that there were areas for improvement and additional work needed, particularly in terms of validating the quality of perturbation outputs and clarifying the perturbation methods used.
Decision: Accept-Findings