A Causal Approach for Counterfactual Reasoning in Narratives (2024)

Abstract: Counterfactual reasoning in narratives requires predicting how alternative conditions, contrary to what actually happened, might have resulted in different outcomes. One major challenge is to maintain the causality between the counterfactual condition and the generated counterfactual outcome. In this paper, we propose a basic VAE module for counterfactual reasoning in narratives. We further introduce a pre-trained classifier and external event commonsense to mitigate the posterior collapse problem in the VAE approach, and improve the causality between the counterfactual condition and the generated counterfactual outcome. We evaluate our method on two public benchmarks. Experiments show that our method is effective. Code is available at https://github.com/mufeiteng/CausalCRN.

heading: 1 Introduction
text: Counterfactual reasoning in narratives (CRN) is commonly known as predicting how alternative events, contrary to what actually happened, might have resulted in different outcomes (Qin et al., 2019; Ashida and Sugawara, 2022). Specifically, given the observed narrative S = (c, x, y), where c, x, and y denote the context, condition, and outcome, respectively, CRN considers how y′ would be if keeping the context c unchanged while perturbing x to a similar but different x′. Figure 1 presents a case of CRN.
Even though it is considered a crucial component of intelligent systems (Pearl, 2009; Pearl and Mackenzie, 2018), only a few resources have been devoted to CRN. Some of the works (Hao et al., 2021; Chen et al., 2022; Li et al., 2023b) design dataset-specific heuristic methods, but they are actually abusing unique patterns, i.e., the feature of minimum editing, in the dataset, which limits the generality of their methods. Other works (Qin et al., 2019; Zhou et al., 2022) take advantage of the
progress of pre-trained language models (PLMs), and fine-tune PLMs for CRN, i.e., learning the conditional distribution p(y′|c,x′,S). Despite the success of simulating real examples, the conditional distribution is notorious for being susceptible to exploiting artifacts of the dataset, instead of learning to robustly reason about counterfactuals (Qin et al., 2019). For example, the models often directly copy the original y or learn to paraphrase y without acknowledging the counterfactual condition (Qin et al., 2019; Hao et al., 2021). As a result, the predicted counterfactual outcome y′ usually conflicts with the counterfactual condition x′.
Generally, CRN relies on the ability to find causality in narratives (Chen et al., 2022), i.e., y′ should express a clear causal relation to x′ to make it clear how the perturbation makes the observed outcome change. This problem naturally fits to be formulated with a causal mechanism (Pearl et al., 2016), which requires us to infer the background knowledge that is compatible with (c, x′, y′). However, this is non-trivial as it involves estimating the posterior of the background knowledge. Luckily, with the variational technique (Kingma and Welling, 2013), we are able to use the background compatible with the observed S to approximate the posterior distribution. In fact, the variational process provides an approximation of the background of (c, x′, y′), but it may face the problem of posterior collapse (Razavi et al., 2019). As a result, the generated y′ may not be the precise effect of
6556
x′, and the resulting model may be sub-optimal. To mitigate this problem, we further propose two intuitive strategies, which introduce a pre-trained classifier and commonsense causality, to enhance the causality between (c, x′) and the generated y′.
In this work, we propose a causal approach for CRN. We utilize the variational process to approximate the implicit background of counterfactual scenarios. In addition, we devise two strategies to alleviate the posterior collapse problem in this variational process. First, inspired by research on natural language inference (Kang et al., 2018; Dziri et al., 2019), we want to ensure that the generated y′ entails its true condition x′. In other words, the model should correctly learn the influence of the condition on the outcome. Therefore, we introduce a pretrained classifier that estimates the likelihood of a text y entails an input (c, x). We use the Gumbelsoftmax technique (Jang et al., 2016; Hu and Li, 2021) to enable gradient back-propagation. Second, we exploit COMeT (Hwang et al., 2021) to retrieve diverse event causality tailored for (c, x′), which allows for deducing plausible event sequences and provides an explicit background for the unobserved counterfactual outcome y′.
To summarize, we formulate CRN in a variational framework and introduce event causality and a pre-trained classifier to further improve the causality between x′ and the generated y′. Our method is a general approach that is applicable to multiple tasks. The experiment proves the effectiveness of our method. We also study the practicality of the generated counterfactual narratives via a data augmentation experiment.

heading: 2 Related Work
text: Causality for NLP The research of causality aims to explore the causal relationships in the data (Yao et al., 2021). Recently, there has been a strong interest in utilizing causal inference to enhance current natural language understanding and generation. These works are mainly studied in event detect (Chen et al., 2021), text classification (Mu and Li, 2023), relation extraction (Liu et al., 2021a), etc. Another line of research attempts to equip the current text generation with counterfactual reasoning ability. These works involve fields such as dialogue generation (Ou et al., 2022), machine translation (Liu et al., 2021b), style transferring (Hu and Li, 2021), etc. Yet there have been few works that apply causal perspective to counterfactual reason-
ing in narratives. We adapt the ideas of the above works and propose additional strategies to improve the causality between the counterfactual condition and the generated counterfactual outcome.
Counterfactual Story Generation Counterfactual story generation aims to revise an original story ending guided by a modified condition (Qin et al., 2019). Previous works (Chen et al., 2022; Li et al., 2023b) usually utilize a two-stage approach. Generally, in the first stage, each token in the original story ending is determined if it requires modification. In the second stage, the identified words are modified to align with the story logic under the counterfactual condition. However, it is difficult to migrate this dataset-specific framework to other datasets (Ashida and Sugawara, 2022). Instead, motivated by counterfactual reasoning (Pearl and Mackenzie, 2018), we propose a general framework for counterfactual reasoning in narratives.
Knowledge-Enhanced Narrative Generation Narrative generation requires models to produce fluent and coherent stories under predefined conditions. Many studies inject structured knowledge into the generation process. For example, Ji et al. (2020) introduces explicit knowledge from ConceptNet, and Mu and Li (2022) introduces structural event causality to improve narrative generation. These works prove that external knowledge helps to enhance the coherence between the input and output text. Motivated by these works, we introduce commonsense causality tailored for the counterfactual condition to improve the causality between (c, x′) and the generated y′.

heading: 3 Methods

heading: 3.1 Problem Setting with Causal Mechanism
text: Given a narrative S = (c, x, y), we perturb x into a counterfactual condition x′ and want to predict the new outcome y′. To solve this problem, we need to speculate on the background knowledge compatible with (c, x′, y′), which allows us
to predict the precise effect of x′. This problem is naturally suitable to be expressed with a causal mechanism. Figure 2 shows the structural causal model (SCM) (Pearl, 2009) that describes the generation process of narratives. Here the latent variable z denotes the unobserved background knowledge. When people write stories, they usually generate narrative events one by one based on their knowledge. This is reflected in the graph as z affects all of (c,x,y). The similar setting can be seen in (Chen et al., 2022), in which they unify the unobserved knowledge and the context into one confounder variable. On the contrary, we model the unobserved knowledge independently to directly infer the information of z.
The SCM thus defines a joint distribution: p(y,x, c, z) = p(y|x, c, z)p(x, c|z)p(z), (1)
where p(z) is a standard Gaussian distribution following common practices. Similarly, conditioned on the observed S, the joint distribution of the counterfactual scenario is defined as:
p(y′,x′, c, z|S) = p(y′|x′,c, z,S)p(x′, c|z,S)p(z|S),
(2)
where p(y′|x′, c, z,S) is the decoder model requiring us to infer z from all (S, c, x′, y′) data. However, the inference of z involves estimating the posterior distribution of the knowledge, i.e., p(z|c,x′,y′,S). We next introduce our basic variational process to approximate the distribution.

heading: 3.2 The Basic Variational Objective

heading: 3.2.1 Variational Inference
text: Our basic objective follows the common VAE approach (Kingma and Welling, 2013). By introducing the approximate network q(z|c,x′,y′,S), a lower bound of the model’s marginal log-likelihood (that marginalizes out z) is:
log p(y′|c,x′,S) = log ∫
z
p(y′, z|c,x′,S)
≥ ELBO = Ez∼q(z|c,x′,y′,S) log p(y′, z|c,x′,S) q(z|c,x′,y′,S) .
(3)
For simplicity, we denote q(z|c,x′,y′,S) as q(z|·). Then, according to Equation 2, we have:
ELBO
= Ez∼q(z|·)[log p(y′, z, c,x′|S)
q(z|·) − log p(c,x ′|S)]
≈ Ez∼q(z|·)[log p(y′|z, c,x′,S) + log p(c,x′|z,S)] − KL[q(z|·)||p(z|S)],
(4)
where p(c,x′|S) is a constant for the given dataset and independent of the parameterized model.
Hence, given the labeled set D which contains all (S, c, x′, y′) examples, our basic objective is:
LVAE = − 1|D| ∑
D Ez∼q(z|·)[log p(y
′|z, c,x′,S)
+ λx log p(c,x ′|z,S) + λkKL[q(z|·)||p(z|S)].
(5)
λx and λk are hyper-parameters. We use the cyclic schedule (Li et al., 2020) to anneal λk from 0 to 1 to avoid excessive regularization of the KL term.

heading: 3.2.2 p(y′|z, c,x′,S) vs. p(y′|c,x′,S)
text: Current generative models follow the autoregressive paradigm, but suffer from exposure bias. Note that (c, x, y) and (c, x′, y′) have similar content. When performing inference, given the input (S, c, x′), p(y′|c,x′,S) has no information about the gold y′, so it may paraphrase y. Different from p(y′|c,x′,S), we encode y′ into q(z|·), and use the KL term to bridge the gap between q(z|·) and p(z|S). When performing inference, we sample z ∼ p(z|S) and feed it into p(y′|z, c,x′,S). This can somewhat alleviate the issue of exposure bias and mitigate the problem of paraphrasing y.

heading: 3.2.3 Model Implementation
text: We use PLMs, e.g., BART (Lewis et al., 2019), as backbone to implement p(y′|z, c,x′,S). We first encode the input part (S, c, x′) into the context vectors HC = BARTEncoder(S, c, x′), where HC ∈ Rl×d, l is the total length of [S; c, x′], d is the hidden size. To fuse z ∼ q(z|·) into PLMs, as suggested in (Li et al., 2020), we concatenate z with HC , and pass it into the decoder for autoregressive learning. The hidden state of t-th time step of the target sequence hyt is computed by:
hyt = BARTDecoder(Y<t, [HC ; z]). (6)
The word distribution of t-th time-step over the standard vocabulary V is:
P (yt|Y<t) = softmaxV (Wvhyt + b). (7) To implement q(z|·) and p(z|S), we approximate them to Gaussian distributions. We use the pre-trained BARTEncoder to initialize different text encoders, which are used to encode S and (c, x′, y′, S). Following several linear layers, we obtain the mean and log-variance of two distributions, which are used to calculate the KL loss. To implement P (c,x′|z,S), we adopt the in-batch contrastive learning. For the positive example (c, x′, z, S), we collect different x̄′ from the minibatch and regard (c, x̄′, z, S) as negative examples. This actually matches the z ∼ q(z|·) with the counterfactual x′, and differitiates it from negatives.
Then the representations of examples are projected into scalar values for binary classification.
Training with the above base objective alone can lead to posterior collapse, i.e., the KL term tends to be zero. As a result, the decoder will ignore the information from q(z|·), and the generated text is not the precise result of x′. We next introduce our two strategies, which introduce the pre-trained classifier and external event causality to improve the causality between (c, x′) and the generated y′.

heading: 3.3 Introducing the Pre-trained Classifier
text: Intuitively, we expect that the generated y′ truly entails its condition x′. To achieve this goal, we pre-train a classifier f([c, x, y]) that estimates the likelihood of the input (c, x) entailed by the output y. Motivated by (Chen et al., 2022), we use the training set of the used datasets to obtain positive and negative examples. For example, given the example (c, x, y, x′, y′), (c, x′) should entail by y′ but contradict with y, and (c, x) should entail by y but contradict with y′. That is, (c, x, y) and (c, x′, y′) are positive, and (c, x′, y) and (c, x, y′) are negative. We initialize f(·) with BARTEncoder to keep the embedding space the same as the generator. Once the classifier is pre-trained, it is frozen in the later process of training our generator.
Then, we train the generator so that its predicted outcome entails the corresponding condition with a high likelihood measured by the classifier:
LCla =− 1|D| ∑
D Ez∼q(z|·),ỹ∼p(y|z,c,x,S′)[
log f([c, x, ỹ]) + log(1− f([c, x′, ỹ]))], (8)
where S′ = (c, x′, y′) and p(y|z, c,x,S′) is the mirror of p(y′|z, c,x′,S). Here, we consider p(y|z, c,x,S′) rather than p(y′|z, c,x′,S) because it has been optimized in Equation 4. Equation 8 can be explained in more detail as following. That is, we first use our generator to generate examples, then we use the frozen classifier to calculate the coherence of the examples, and then we update the parameters of the generator to maximize the coherence. In fact, we use the classifier to restrict the generated ỹ entails its true condition x but contradicts with x′. As in (Hu et al., 2017; Hu and Li, 2021), we use Gumbel-softmax to enable gradient backpropagation for the discrete text.

heading: 3.4 Utilizing External Event Causalities
text: The variational process provides an implicit background for unobserved counterfactual outcomes,
this further motivates us to utilize external event causalities which allows for introducing diverse event commonsense and providing an explicit background for generating counterfactual outcomes.

heading: 3.4.1 Retrieving Event Causality
text: We use COMeT (Hwang et al., 2021) as the event knowledge base. We first feed the zero-hop events (c, x′) into COMeT to generate one-hop events with corresponding relations. The one-hop events are then fed into COMeT to generate two-hop events. We leave the details in Appendix A. We next organize the retrieved knowledge into an event graph G = (V,E) where V denotes the node set and E denotes the edge set. Each node e ∈ V is an event which is a word sequence. Each edge in E is a tuple (eh, r, et) containing a head event eh, a relation r, and a tail event et. Then, we perform reasoning on G to select guided events, which are the possible effects of (c, x′). We use the selected events as guidance for generating y′.

heading: 3.4.2 Selecting Guided Events
text: Motivated by (Ji et al., 2020; Mu and Li, 2022), we perform multi-hop reasoning on G to select important event nodes. We iteratively compute the relevance scores of multi-hop events with respect to (c, x′), as shown in Figure 3(a). In each iteration, we parallelly calculate the scores of events in the same hop. For the tail event et, the score s(et) is calculated by polymerizing information from its neighbors Net including pairs of (eh, r):
s(et) = 1 |Net | ∑
(eh,r)∈Net
(s(eh) +R(eh, r, et)). (9)
At the beginning, zero-hop events, i.e., (c, x′), are assigned a score of 1, e.g., s(c) = s(x′) = 1, while other events are assigned a score of 0. R(·) is the relevance of the edge (eh, r, et) with respect to the (c, x′), which is calculated by:
R(eh, r, et) = σ(h T (c,x′)Wk · [heh ;hr;het ]), (10) where Wk ∈ Rd×3d, [·; ·] denotes the concatenation, h(c,x′) ∈ Rd is the embedding of (c, x′), heh ,hr,het are the embeddings of eh, r, et.
We select the top-k events according to their scores: Ek = topki(s(ei)). k is set to 4 after searching on the dev set. To fuse the guided Ek into the generation, we concatenate (S, c, x′) with Ek, and pass them to BARTEncoder to obtain knowledge-enhanced context vectors HC̃ = BARTEncoder(S, c, x′, Ek). Then, we concatenate HC̃ with z ∼ q(z|·), and feed it into the
decoder for autoregressive learning, i.e., y′ ∼ p(y′|z, c,x′,S,Ek), as shown in Figure 3(b).

heading: 3.5 Training and Inference
text: Similar to (Mu and Li, 2022), we add an additional object to guide the event selection. We maximize the probability of selecting positive events by:
LE = ∑
D |D|
∑
i
−li log p(ei)−(1−li) log(1−p(ei)), (11)
where p(ei) = σ(s(ei)) is the probability that the event ei is selected. li is the label of ei which is subject to the overlap between ei and the gold y′. The details are in Appendix A. The final object is:
L = LV AE + αLCla + βLE , (12)
where α and β are hyper-parameters. When inference, given input (S, c, x′), we first sample z ∼ p(z|S), then we select guided event Ek according to (c, x′), at last we generate the counterfactual outcome y′ ∼ p(y′|z, c,x′,S,Ek).

heading: 4 Experiment

heading: 4.1 Datasets
text: We evaluate our method on two datasets. (1) TimeTravel (Qin et al., 2019) is a counterfactual story rewriting dataset. It is built on the ROCStories (Mostafazadeh et al., 2016) corpus, which consists of a large set of five-sentence stories S = s1:5. s1 is set as the context c, s2 is set as the condition x, and s3:5 sets up the outcome y. In TimeTravel, the initial condition x is rewritten by humans into a counterfactual condition x′, and then annotators perform minimal edits to the original ending y to create the counterfactual outcome y′. In TimeTravel, the major challenge is the trade-off between generating natural stories and modifying the original y with minimal edits.
(2) PossibleStories (Ashida and Sugawara, 2022), also built on the ROCStories corpus, considers the problem that: for the same context, if the current situation is different, the final consequence
may be different. It is originally a multiple-choice dataset, where each example consists of the original context c, the original ending y, the counterfactual question x′, and candidate options including the counterfactual ending y′. To adapt it to text generation, we set the original condition x as a simple text “what’s the most likely story ending?", then we generate y′ according to (c, x, y, x′).
The statistics of two datasets are in Appendix B.

heading: 4.2 Baselines
text: We produce the following kinds of baselines: • Prompting large chat models, e.g., Chat-
GLM2(6B) (Zeng et al., 2022), Llama2Chat(7B) (Touvron et al., 2023), ChatGPT (OpenAI. , 2023). We use one-shot prompting for experiments, the used prompts are in Appendix C. • Supervised fine-tuning. We fine-tune several pre-trained language models, including GPT2(base) (Radford et al., 2019), T5(base) (Raffel et al., 2020), BART(base) (Lewis et al., 2019), and Llama2(7B) (Touvron et al., 2023). We use QLoRA (Dettmers et al., 2023) to adapt Llama2(7B) on a single 3090 GPU. For TimeTravel, we additionally compare our
method with some task-specific methods: • DELOREAN (Qin et al., 2020) and EDUCAT
(Chen et al., 2022) which regard the task as a controllable generation problem, and unsupervised edit the original y to the counterfactual y. • CLICK (Li et al., 2023a), a two-stage method, first detects which words in the original ending need to be modified, and then implements the modification.

heading: 4.2.1 Implementation Details
text: We use the train set of the two datasets to train the classifier. We use the AdamW optimizer and set lr to 5e-6. We select checkpoint according to F1 on the dev set. The best checkpoint achieves the F1 scores of 66.1 and 70.1 in the test set of two
datasets. When training the counterfactual generator, we use the AdamW optimizer and set lr to 5e-5. We linearly decrease lr to zero with a 10% warmup ratio. We search for the best hyper-parameters according to ENTScore on the dev set of each dataset. The searched parameters are in Appendix D. When inference, we adopt the multinomial sampling strategy to generate y′, and we repeat for 5 times to calculate the average performance.

heading: 4.3 Automatic Evaluation
text: Metrics For Timetravel, we follow the previous works and use BLEU (Papineni et al., 2002), BertScore (Zhang et al., 2019), ENTScore (Chen et al., 2022), and HMean = 2·BLEU·ENTScoreBLEU+ENTScore (Chen et al., 2022) as metrics. BLEU and BertScore evaluate the similarity between the generated y′ and the ground truth. ENTScore evaluates the coherence between (c, x′) and the generated y′. For PossibleStories, we use BLEU, BertScore, and ENTScore as metrics.

heading: 4.3.1 Our Method vs. Baselines
text: The automatic evaluation result is shown in Table 1 and 2. We can see BART generally performs better than GPT2 and T5, therefore we use BART as the backbone. In addition, we observe that: • In Table 1, unsupervised editing-based methods
have poor performances, indicating that this kind
PossibleStories
Methods BLEU BertScore ENTScore
Prompting Chat Models ChatGLM2(6B) 1.9 48.4 38.8 Llama2Chat(7B) 3.0 49.9 43.8 ChatGPT 5.0 53.5 48.5
PLMs-based Finetuning GPT2 6.0(0.7) 49.4(0.3) 37.3(0.4) T5 5.7(0.3) 49.2(0.3) 35.8(0.7) BART 13.2(0.5) 53.8(0.2) 42.9(1.0) Llama2(7B) 16.3(1.1) 54.4(0.6) 45.1(0.9) Ours 16.1(0.2) 56.2(0.1) 46.9(1.0)
Ablation Experiment
w/o Clas 15.7(0.4) 55.6(0.3) 45.6(0.7) w/o Event 15.5(0.4) 55.8(0.2) 46.0(0.5) w/ VAE 15.5(0.5) 55.9(0.3) 45.0(0.4)

heading: 4.3.2 Ablation Study
text: Settings To investigate the effectiveness of different components, we devise the following ablated variants to compare with our full model. (1) “w/o Event" means we do not use event causality. (2) “w/o Cla" means we remove the pre-trained clas-
sifier. (3) “w/ VAE" means we ablate both event causality and the pre-trained classifier. In this case, this variant degenerates into the basic VAE module.
Result The ablation study result is shown in Table 1 and 2. We have the following observations. • Compared with BART, “w/ VAE" achieves a
2.1/2.1 gain in ENTScore on both datasets. This demonstrates the effectiveness of the variational process. The possible reason is that the variational process learns an approximation of the latent z, which provides an implicit background for generating counterfactual outcomes. • Compared with “w/ VAE", “w/o Clas" and “w/o Event" achieve higher ENTScore on both datasets, indicating the two strategies contribute to improving the causality between x′ and the generated y′. This makes sense because (1) external event causality provides a causal background for generating y′ and (2) the classifier will punish the unqualified generation. The best result is achieved when combining two strategies, these show that two strategies complement each other. • “w/o Clas" performs better than “w/o Event" on both datasets. This shows that the classifier is more important than event knowledge. The possible reason lies in two aspects. (1) Though retrieved event causality may contain useful information for generation, unrelated and noisy knowledge may also be retrieved. (2) The classifier directly measures the causality between the condition and the generated outcome, and penalizes incoherent generation.

heading: 4.4 Manual Evaluation
text: Setting For TimeTravel, we follow (Chen et al., 2022) and use Minimal-Edits and Coherence as manual evaluation metrics. Coherence denotes the logical consistency between the counterfactual context (c, x′) and generated y′. Minimal-Edits denotes the extent of minimal revision between the original y and the generated y′. For PossibleStories,
we use Similarity and Coherence as metrics, where Similarity evaluates the similarity between the generated y′ and the ground truth. We carry out pairwise comparisons between our method with some baselines, including Llama2Chat, ChatGPT, and two ablated models “w/o Event" and “w/o Clas". We randomly sample 100 cases from the two test sets for each pair of models, respectively. Three annotators are recruited to make a preference among Win, Tie, and Lose given the input and two outputs generated by our model and a baseline respectively. The annotators are research students from the field of commonsense text generation to make sure they have a fair judgment of used metrics.
Result The result is shown in Table 3. Compared with the two ablated variants, our full method shows an increase in Coherence, but a decrease in Minimal-Edits. This is because both of the two strategies prevent copying the original ending y. On TimeTravel, our full model performs better in Minimal-Edits, but not as well in Coherence as the chat models. This is consistent with automatic evaluation. We calculate Fleiss’s kappa reliability as the inter-rater agreement. For TimeTravel, the agreement of Minimal-Edits and Coherence is 0.43 and 0.56. For PossibleStories, the agreement of Similarity and Coherence is 0.50 and 0.52.

heading: 4.5 Further Discussion

heading: 4.5.1 Analyzing the VAE Module by Manipulating z
text: Since the strength of our VAE module lies in its ability to approximate the posterior distribution, we are interested in whether the encoded z benefits counterfactual narrative generation. We conduct a pilot study on TimeTravel. We first replace z ∼ p(z|S) with a random noise znoise ∼ N (0, 1), and feed znoise into the VAE decoder p(y′|z, c,x′,S) for generation. We get a 43.8 ENTScore, which is significantly worse than the result of BART. This is reasonable because the random noise disrupts the model structure and brings about a significant negative impact. Next, we make a linearly interpolation z̄ = α ·zprior+(1−α) ·zposterior, where zprior ∼
p(z|S) and zposterior ∼ q(z|c,x′,y′,S), and feed z̄ for generation. The result is shown in Figure 4. The larger the proportion of zposterior, that is, the more posterior information about the gold y′, the better the result is achieved. When using zposterior for generation, we get a 56.5 ENTScore, but the value is still not satisfactory enough. The possible reason is that too much information is lost during the process of encoding the sequence into a vector z, making it hard for the model to reconstruct the gold y′. According to above results, we speculate that the more effective solution to this problem is to introduce more diverse and more large-scale data. Therefore, we conduct the following data augmentation experiment.

heading: 4.5.2 Generating Counterfactual Stories for Data Augmentation
text: (Qin et al., 2019) provides an additional data partition that only has counterfactual conditions x′ but no counterfactual outcomes. This partition contains about 97k examples. We use this partition to study the practicality of the generated counterfactual stories via a data augmentation experiment. Specifically, we use our ablated variant “w/o Event" as the generator since its performance is not significantly worse than our full model, and there is no need for external knowledge, making it easy to use. For each (S, c, x′), we use “w/o Event" to generate 60 candidates and keep the one with the highest ENTScore as the pseudo counterfactual outcome, denoted as ỹ′. Finally, we obtain the pseudo set DP = {(S, c, x′, ỹ′)}. We test this set for both generation and classification tasks.
Testing for the Generation Task First, We only use DP to directly fine-tune BART and Llama2, i.e., learning p(y′|c,x′,S). The result is shown in Figure 5(a). When training with about 32k pseudo examples, BART achieves a 52.0 ENTScore, which is obtained using the labeled set D. When using more pseudo examples for training, the result con-
tinuously improves. We have a similar observation from the result of Llama2(7B). Because finetuning Llama2(7B) is time-consuming, e.g., it takes about 1.5 hours to train an epoch with 30k samples, we use a maximum of 50k samples for fine-tuning. Next, we mix the labeled set D and a different number of pseudo examples to fine-tune BART. The result is shown in Figure 5(b). When mixing D and all pseudo examples, BART obtained a 70.1 ENTScore, which is better than ChatGLM2 and closer to Llama2Chat. However, as the number of pseudo examples increases, BLEU continues to decline, but overall the decline is acceptable.
Testing for the Classification Task Motivated by (Chen et al., 2022), we construct a binary classification task to test the quality of pseudo examples, which is the same as training the classifier. We explore three types of training examples to train RoBERTa-large (Liu et al., 2019) and then validate on the test set of TimeTravel: (1) the labeled set D, (2) the pseudo set DP in which we randomly sample a different number of pseudo examples, and (3) the mixed set in which we mix the label set D and a different number of pseudo examples. The F1 result on the test set of TimeTravel is shown in Figure 6. When training with more pseudo examples, F1 achieves a stable improvement. The F1 under the mixed set is better than that under the labeled set, indicating that the pseudo set is an effective supplement to the labeled set.
Overall, these results demonstrate the practicality of the generated pseudo examples, which further proves the effectiveness of our method.

heading: 4.5.3 Case Study and Error Analysis
text: Table 4 presents a case study. The counterfactual ending generated by ChatGPT is coherent, but it differs significantly from the original ending. Our model changes the “land" in the original ending to “fossil" and generates a coherent counterfactual ending with minimal-edits. Since the context has changed from “oil" to “fossils", the counterfactual endings containing fossil-related information better
reflect the causal relationship between the perturbation and story endings. However, we find that the issue of paraphrasing y still exists in our method, as shown in Appendix E, Table 5. But this issue less-likely occurs in large chat models. We speculate that there are two reasons: (1) the problem of exposure bias cannot be completely eliminated; (2) The used model is small, and the scale and diversity of data are insufficient. More examples can be seen in Appendix E, Table 9.

heading: 5 Conclusion
text: In this work, we formulate counterfactual reasoning in narratives in a VAE framework. In addition, we introduce a pre-train classifier and external event causality to further improve the causality between the counterfactual condition and the generated counterfactual outcome. The experiment proves the effectiveness of our method. We also conduct a data augmentation experiment to verify the practicality of our method.

heading: Acknowledgements
text: We thank the anonymous reviewers for their encouraging feedback. This work is supported by Research Grants Council of Hong Kong(PolyU/15207920, PolyU/15207821, PolyU/15213323) and National Natural Science Foundation of China (62076212).
Limitations
We construct our method based on small-scale datasets and the small model, e.g., BART, therefore
our method cannot outperform large language models. But it is inherently unfair to directly compare small models with LLMs, as large models are obtained with massive resources, e.g., data, hardware, funding, etc. Although large models have strong capabilities, their disadvantage is that once trained, it is especially difficult to adjust them. Therefore, LLMs may not be able to meet task-specific constraints that have not been seen during training. However, the problem is that different tasks may have additional requirements. For example, TimeTravel requires the generated y′ to exhibit MinimalEdits characteristics. However, the output of chat models conflicts with the requirement, indicating that chat models still face the problem of precisely following instructions for generation.
In the experiment, fine-tuned Llama2 performs better than fine-tuned BART. This indicates that constructing our method upon larger models may have a better performance. In addition, the generated counterfactual stories are beneficial for counterfactual narrative reasoning. This foreshadows the future direction, that is, we can transform x into different x′ through different perturbations, thus generating diverse counterfactual stories for data augmentation. In a bootstrapping manner, more and more qualified examples can be created for training neural models. Different from predicting counterfactual outcomes, it is easy to perturb x into x′, and there have been a lot of related research works. We leave this in the future work.
In addition, we demonstrate our method using five-sentences RocStories datasets, but it does not mean that our method is not applicable for stories of variable length. Our SCM contains three variables: the context c, the treatment x, and the effect y. Actually, c, x, and y may contain one or several narrative events. That is, given a narrative story with variable length, we can divide it into three parts (c, x, y) to adapt our approach.
Ethical Considerations
This paper mainly focuses on narrative reasoning. It describes event relationships in human daily life, and does not involve sensitive, biased, or harmful content. The datasets used in this work does not involve any sensitive data, but only crowd-sourced datasets released in previous works, including RocStories (Mostafazadeh et al., 2016), TimeTravel (Qin et al., 2019), and PossibleStories (Ashida and Sugawara, 2022). We believe that our research
work meets the ethics of ACL.

heading: A Details for Constructing Event Causality Graph
text: To obtain event causality knowledge for counterfactual narrative generation, we use COMeT as event knowledge base. COMeT is a transformer model trained on ATOMIC (Sap et al., 2019) that generates nine kinds of inferences of events in natural language. In our work, we select 8 relations that are causal related for retrieving event knowledge. They are xNeed, xIntent, Causes, HasSubEvent, oEffect, oWant, xEffect, xWant. Given a query event, we retrieve 5 knowledge tuples for each relation. We first feed the central events (c, x′) into COMeT to generate one-hop events with corresponding relations. The one-hop events are then fed into COMeT to generate two-hop events. After that, we obtain many event paths. Several heuristic rules are applied to filter low-quality paths. For example, if an event in a path contains less than 2 words or more than 8 words, the path will be discarded. Finally, we transform the retained event chains into an event graph that simulates the evolution tendency of events centered on (c, x′), and provides a explicit background for the unseen counterfactual outcome. We heuristically use the word-overlap ratio to label event nodes. That is, an event is labeled as positive if 60% of event tokens are contained by the gold y′. The event labels are used as supervision for selecting guided events.

heading: B Statistics of the Used Datasets
text: The statistics of the used datasets are shown in Table 6.

heading: C The Prompts for Different Tasks
text: Table 8 presents the prompts we used for utilizing chat models.

heading: D Hyper-Parameters Used in This Work
text: The searched hyper-parameters on two datasets are shown in Table 7.

heading: E Supplementary Cases
text: Table 5 shows a case for error analysis, in which our method still copies the original y. We count the statistics on the test set of TimeTravel. The number of our model directly copies the original is approximately 8% of the test set. Table 9 presents some cases with the generated counterfactual outcome by different models. The case #1 is from the test set of TimeTravel, and the case #2 is from the test set of PossibleStories.

heading: Future Work
text: This foreshadows the future direction, that is, we can transform x into different x′ through different perturbations, thus generating diverse counterfactual stories for data augmentation. In a bootstrapping manner, more and more qualified examples can be created for training neural models. Different from predicting counterfactual outcomes, it is easy to perturb x into x′, and there have been a lot of related research works. We leave this in the future work. In addition, we demonstrate our method using five-sentences RocStories datasets, but it does not mean that our method is not applicable for stories of variable length. Our SCM contains three variables: the context c, the treatment x, and the effect y. Actually, c, x, and y may contain one or several narrative events. That is, given a narrative story with variable length, we can divide it into three parts (c, x, y) to adapt our approach. Ethical Considerations
This paper mainly focuses on narrative reasoning. It describes event relationships in human daily life, and does not involve sensitive, biased, or harmful content. The datasets used in this work does not involve any sensitive data, but only crowd-sourced datasets released in previous works, including RocStories (Mostafazadeh et al. , 2016), TimeTravel (Qin et al. , 2019), and PossibleStories (Ashida and Sugawara, 2022)