COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective
Year: 2023

Abstract:
Detecting commonsense causal relations (causation) between events has long been an essential yet challenging task. Given that events are complicated, an event may have different causes under various contexts. Thus, exploiting context plays an essential role in detecting causal relations. Meanwhile, previous works about commonsense causation only consider two events and ignore their context, simplifying the task formulation. This paper proposes a new task to detect commonsense causation between two events in an event sequence (i.e., context), called contextualized commonsense causal reasoning. We also design a zero-shot framework: COLA (Contextualized Commonsense Causality Reasoner) to solve the task from the causal inference perspective. This framework obtains rich incidental supervision from temporality and balances covariates from multiple timestamps to remove confounding effects. Our extensive experiments show that COLA1 can detect commonsense causality more accurately than baselines.

heading: 1 Introduction
text: Commonsense Causal Reasoning (CCR) aims at identifying plausible causes and effects of events in natural language that are typically reasonable by an average person (Zhang et al., 2022c). To solve the task, existing efforts devoted by the community mainly rely on language models wholeheartedly with supervised learning approaches (Staliūnaitė et al., 2021; Sap et al., 2019; Tamborrino et al., 2020; He et al., 2020; Raffel et al., 2020). Those ingenious engineering works have brought significant progress in recent years. However, recent studies (Kavumba et al., 2019; Han and Wang, 2021) found that pure engineering designs are inadequate to seize commonsense causation, as language mod-
1The code and data are available at https://github. com/HKUST-KnowComp/COLA.
els tend to reach higher scores by exploiting superficial artifacts in data.
Recently, Zhang et al. (2022c) first studied grasping commonsense causation from the causal inference perspective, by drawing analogies between observational studies (Cochran and Chambers, 1965; Rosenbaum, 2002) and natural languages (Zhang and Zhang, 2021). The proposed framework ROCK achieves good potential for the zero-shot CCR task (e.g., COPA by Gordon et al. (2012)). However, Zhang et al. (2022c) only focuses on the commonsense causation between a pair of events without specifying context. Given that events are complex (Chen et al., 2021), an event may have different causes under different contexts (Mostafazadeh et al., 2020). Thus, it is necessary to utilize context when detecting commonsense causation, such as other events related to given ones. Missing a clear and specific context simplifies commonsense causal knowledge and hinders models from detecting commonsense causal relations more accurately. For example, as shown in Figure 1, the frequent cause of “Emma made a steak in the kitchen.” is “Emma felt hungry.” However, the cause also could
5253
be “Emma was doing her job” if “Emma is a chef at a steakhouse.” Without the context of Emma’s job, models cannot distinguish those two causes and may return to the frequent one.
To involve context when detecting commonsense causation, we propose a new task to detect causes between two events in an event sequence, called Contextualized Commonsense Causal Reasoning (Contextualized CCR). In this task, models are asked to detect commonsense causal relations between two given events enclosed in an event sequence. Other events in the event sequence can provide a clear and specific definition of the current context, helping models to capture commonsense causation more accurately. In fact, we find that contextualized CCR is a non-trivial task. Directly applying the framework ROCK (Zhang et al., 2022c) cannot achieve competitive performance on the contextualized CCR since it cannot integrate context information.
We propose the framework COLA, which incorporates contextual information from an event sequence, to solve the Contextualized CCR. Our framework adopts the potential-outcomes framework (Rubin, 1974; Rosenbaum, 2002; Rubin, 2005) to estimate the causal estimand ∆ defined as a type of “average treatment effect” (ATE), which measures the change in the likelihood of Ej’s occurrence when intervening Ei (denoted by ¬Ei) as
∆ = P (Ei ≺ Ej)− P (¬Ei ≺ Ej) , (1) where P(·) can be estimated with a pre-trained language model, such as a masked language model (Devlin et al., 2018). The magnitude of average treatment effect informs the strength of Ei’s effect on Ej , and its sign indicates the direction of the effect. For instance, ∆ ≈ 1 means Ej becomes more prone to occur due to the occurrences of Ei. In an ideal world (e.g., Ei and Ej on any study unit occur completely randomly), a plugging-in estimator in Equation (1) suffices for detecting commonsense causation. Nevertheless, spurious correlations introduced by pervasive confounding co-occurrences need to be eliminated for an unbiased estimation of the causal estimand. This can be done by balancing events that precede Ei, or covariates. To incorporate context, we design a mechanism to sample diversified covariates from multiple timestamps and use temporal propensity (Zhang et al., 2022c) for balancing.
We annotated commonsense causal relations between two events (~1.3k examples) within event
sequences from ROCStories (Mostafazadeh et al., 2016) to benchmark our proposed contextualized CCR task. We conduct extensive experiments with multiple pre-trained language models, showing that COLA can detect cause-and-effect relations more accurately than competitive baselines by a large margin. Our experiments also show that temporality is essential in our framework but not sufficient to detect commonsense causation without covariates being appropriately balanced.

heading: 2 Background and Related Works
text: Understanding events and relations between them have long been a challenging NLP task (Chen et al., 2021). The community has dedicated many works to studying various event-centric tasks, including event relation reasoning (Ning et al., 2018; Zhou et al., 2021; Wang et al., 2020), event extraction (Huang et al., 2018; Lai et al., 2020; Zhang et al., 2022b; Lin et al., 2023), event-centric KG construction (Zhang et al., 2020b, 2022a), and many others (Chambers and Jurafsky, 2008; Chen et al., 2020; Jin et al., 2022; Wang et al., 2022b). Among them, there are a few lines of work that are most related to our work:
Commonsense Causal Reasoning Since our work is about Contextualized CCR, we first discuss related works about commonsense causal reasoning. Existing commonsense causal reasoning approaches are typically categorized under the general topic of commonsense reasoning (Rashkin et al., 2018; Sap et al., 2020). Most previous works depend on language models. Remarkable progress in CCR mainly comes from dataset augmentation, training procedure design, and external knowledge (Staliūnaitė et al., 2021; Sap et al., 2019; Shwartz et al., 2020; Tamborrino et al., 2020; Iter et al., 2020). Studies (Kavumba et al., 2019; Han and Wang, 2021) show that language models exploit superficial artifacts to achieve suspicious high performance.
Causal event detection (Mirza and Tonelli, 2014; Mirza et al., 2014) forms another line of work pertinent to CCR. The task aims to detect causal relations in documents, where various methods are proposed (Chang and Choi, 2005; Do et al., 2011; Ning et al., 2019). However, those works consider verbal (e.g., “attack”) or nominal predicates (e.g., “explosion”) as events, oversimplifying the relation detection task. In this paper, we study events expressed in free text, facing a more challenging
setup but being closer to real applications.
Narrative-related Tasks Since Contextualized CCR is primarily about chains of events, our work is inspired by earlier research that deals with narratives and scripts (Chambers and Jurafsky, 2008; Granroth-Wilding and Clark, 2016; Mostafazadeh et al., 2016; Bhagavatula et al., 2019; Zhang et al., 2020a). In contrast, our work aims to identify causal relations in a chain of events.
Methodologies of Causal Inference Zhang and Zhang (2021) provided the first study to solve the CCR task from the causal inference perspective. The human population studies have scrutinized extensively the causal inference, which identifies causal relations from ubiquitous associations, including biomedical research, agriculture, epidemiology, and economics (Fisher, 1958; Imbens and Rubin, 2015; Giannarakis et al., 2022; Rosenbaum, 2002; Cochran and Chambers, 1965), where researchers usually use the potential-outcomes framework (Splawa-Neyman et al., 1990; Rubin, 1974; Holland, 1986), graphical and structural equation models (Robins, 1986; Pearl, 1995; Heckman, 2005), and Granger causality (Granger, 1969).
Recent studies have drawn causal inferences on textual data with the help of powerful pre-trained language models (Kang et al., 2017; Keith et al., 2020; Feder et al., 2022). Concurrently, causal inference can improve the robustness and fairness of NLP models (Feder et al., 2022) or boost performance on downstream tasks (Ghosal et al., 2021; Zheng et al., 2022; Alabdulkarim et al., 2021; Wang et al., 2022a).

heading: 3 Problem Formulation
text: Notation We use sans− serif fonts to represent an event, such as Ei in Figure 2, where the subscript i means the i-th event in a sequence. Simultaneously, uppercase serif letters denote indicators of whether the corresponding event occurs: Ei = 1 {Ei occurs}, and a lowercase serif letter means the realizations of this indicator: ei,j = 1 {Ei occurs to the j-th study unit}. We introduce the point process to more clearly describe the order of a pair of events: E(t) with t ∈ {0, 1} (e.g., past versus present), so that Ei(0) and Ej(1) means that Ei happens before Ej . We also use Ei ≺ Ej to indicate that Ei occurs before Ej for simplicity. We write P(Ei ≺ Ej) = P(Ei(0), Ej(1)).
Task Description We articulate the Contextualized CCR as a tweaked form of the binary classification problem. Specifically, we provide models with an event sequence of n events: E1,E2, . . . ,En. Models need to find the top-k events in this sequence that more plausibly have commonsense causal relations with the last event En, where k indicates the number of positive labels in the ground truth. Then, models need to predict the commonsense causal relation between each event pair (Ei, En) as a positive/negative one. The strength of causal relations between Ei and En can be expressed with average treatment effect as:
∆i = P (Ei ≺ En)− P (¬Ei ≺ En) . (2)
4 Theoretical Mechanism of COLA
As discussed in Section 1, we articulate the Contextualized CCR problem as the estimation of the causal estimand ∆, which we model as the change of temporal likelihood with contexts controlled. We adopt the potential-outcomes framework to design COLA to eliminate potential confounding effects due to co-occurrences of events when estimating the causal estimand from data. In this section, we first clarify all essential concepts in the theoretical mechanism of COLA one by one, including study unit, intervention, covariate, and propensity, by drawing analogies between the underlying causal mechanisms in natural languages with that in human population research. We then describe the implementation of each component in Section 5.

heading: 4.1 The Analogy and Study Unit
text: Motivated by Zhang et al. (2022c), we draw the analogy between human subjects and semantic meanings through the following process: assuming that every human subject kept a textbook recording each event (s)he has experienced, we can then treat each textbook (in natural language) as a study unit and infer the temporal relationships between events from it. In this analogy, we clearly understand the study unit in semantic meanings.
Then, we can formulate contextualized CCR with concepts from the potential-outcome framework. Given two events Ei and En from an event sequence E1,E2, . . . ,En, where we assume that Eij represents the event that the j-th study unit experienced at the timestamp i when Ei is supposed to occur. Then for each unit j, we can define the treatment assignment as Eij = 1 {Eij = Ei}, realizations of covariates as xj = (xjl)Nl=1 for
xjl = 1 {Xl ≺ Eij}, and two potential-outcomes as {
r0,ij = 1 {Eij,Ei=0 ≺ En} , r1,ij = 1 {Eij,Ei=1 ≺ En} .
(3)
When the j-th unit receives the treatment assignment Eij , the hypothetical scenario is denoted by Eij,Ei=1−Eij , which describes what if the assignment were flipped. Clearly, we can only observe either r0,ij or r1,ij , but not both of them. We can rewrite the causal estimand ∆i in Equation (2) exactly as an average treatment effect by averaging over the unit index:
∆i = Ej [r1,ij − r0,ij ] ≡ P(Ei ≺ En)− P(¬Ei ≺ En).
(4)
The above formulation naturally embodies the temporal nature of covariates (Rubin, 2005), which, by definition, are pretreatments that precede treatments.

heading: 4.2 Intervention Beyond Negation
text: In human population studies, the generally accepted stable unit treatment value assumption (Rubin, 1980) ensures only one type of non-treatment (usually negation) for each study unit. As events are complicated, we would interpret intervention (manipulation) of semantic meanings in a broader sense. Take Ei “Emma felt hungry” from Figure 2 as an example. While “Emma didn’t feel hungry” is the most direct intervention, it is nonetheless too restrictive: Emma may have felt happy; maybe Alice is the one who felt hungry, instead of Emma. Consequently, interventions in our framework are interpreted much broader as any event that could result in a plausible counterfactual of the outcome. We use A to represent all possible interventions of an event Ei.

heading: 4.3 Balancing Covariates and Comparable Study Units
text: We have discussed that the plugging-in estimator in Equation (1) suffers from biases due to potential confounders. One mitigation strategy is properly balancing the covariates (Rubin, 2005), namely events that occur before Ei, which ensures that covariates of untreated study units are comparable to those of treated ones. Consider the vaccine trial as an example; one needs to ensure that the health conditions (covariates) in the control group (not administered vaccines) are comparable to the treated group (administered vaccines). As such, we rewrite the causal estimand ∆ in Equation (2) as expectations conditional on the covariates x among comparable study units:
Ex [P(Ei ≺ En|x)− P(¬Ei ≺ En|x)] , (5)
provided that the treatment assignment is strongly ignorable with respect to potential outcomes (i.e., r1,ij and r0,ij) according to the strong ignorability assumption.
The strong ignorability assumption is essential in causal inference. Simply, it means that given a set of covariates, the treatment assignment among study units can be viewed as “random” (or “ignorable”) with respect to potential outcomes (see, e.g., Rosenbaum (2002); Rubin (2005) for textbook treatments and Zhang et al. (2022c) for more discussions on this assumption in CCR).

heading: 4.4 Matching Temporal Propensities
text: Directly estimating Equation (5) may face the issue of data sparsity since we may sample multiple covariates, and combinations of their values grow exponentially. There are various methods to mitigate this issue in balancing covariates, including assignment modeling, outcome modeling, and doubly-
robust estimations (Rubin, 2005), among which we base our method on propensity score matching. It is a simple and effective method that is widely used in observational studies (Rosenbaum and Rubin, 1983), which matches the propensity scores of study units to balance covariates. The propensity score is defined as p(x) = P(Ei(1) = 1|x(0)), which represents the probability of Ei taking place conditioning on covariates x. Since it is unclear how to pack an unordered set of covariates (events) into a sequential input, Zhang et al. (2022c) proposed to use a relaxed notion of temporal propensity vector, defined as the vector of probabilities of Ei happening conditional on each covariate x ∈ x: q(x) = q(x;Ei) = (P(Ei(1) = 1|x(0)))x∈x . (6) Hence, we can rewrite the conditional expectation in Equation (5) in the form of matching temporal propensity vectors for some fixed threshold ϵ, given below: { ∆̂i = f(Ei,En)− 1|A′| ∑ A∈A′ f(A,En),
A′ := { A ∈ A : 1|X | ∥q(x;A)− q(x;Ei)∥2 ≤ ϵ } ,
(7) where f(Ei,En) is an estimate for P(Ei ≺ En) produced by a language model.
5 The COLA Framework
After establishing the theoretical mechanism for our framework COLA, we describe the implementation of each component of COLA in this section. Generally, since events are in free-text form in our task, pre-trained language models play a central role in our framework. Given that LMs are pre-trained on an enormous amount of textual data (Gao et al., 2020; Raffel et al., 2020), it is sensible to suppose that those LMs would emulate the responses of an average reasonable person.
Specifically, our framework COLA takes two events Ei and En from a sequence E1,E2, . . . ,En as input. As shown in Figure 2, our framework COLA contains four steps: (1) a multistamp covariate sampler samples a set X of covariates. (2) an intervention generator generates a set A of interventions. (3) A score estimator builds temporal propensity vectors and selects a matched subset A′ out of A, by estimating the temporality with a temporal predictor. (4) Eventually, the same score estimator computes ∆̂i according to Equation (7).
Multistamp Covariate Sampler Our multistamp covariate sampler is based on GPT-J
(6b) (Wang and Komatsuzaki, 2021). For an input event E, we add “Before that,” at the end to build “E Before that,” as the prompt template. For the i-th event in a sequence, we first sample a covariate set Xi, which contains events before Ei. To diversify covariates, we also sample events before E1,E2, . . . ,Ei−1 separately, forming X1,X2, . . . ,Xi−1. Those events are also before Ei and can serve as covariates due to the transitivity of the temporal relation2. We evenly merge covariates before each timestamp to construct the final covariate set:
X = ∪il=1Xl. (8)
Union vs. Intersection While the aforementioned method takes the union of events sampled according to the left-side context of Ei, another intuitive approach is to take the intersection of events sampled according to multiple timestamps in the right-side context. In this way, we can collect covariates that happen before all of Ei,Ei+1, . . . ,En, that is, X = ∩nl=iXl. We discuss the experimental results of these two methods in Section 7.3 and found taking union works better, which will be the default in the remaining sections.
Intervention Generator This component generates a set A of interventions as discussed in Section 4.2. There are a variety of related works about generating interventions (counterfactuals) of an event (Gardner et al., 2020; Qin et al., 2019; Ribeiro et al., 2020) and we choose PolyJuice (Wu et al., 2021) in our framework owing to its non-task-specific training objective. PolyJuice generates interventions by masking some phrases individually and filling in masks with a fine-tuned GPT2. Then, we apply the semantic role labeling (SRL) tool provided by AllenNLP (Gardner et al., 2018) to extract the verb V and two arguments ARG0 and ARG1 as phrases to be manipulated (see Appendix A.2 for more details).
Temporal Predictor We prompt a masked language model to estimate the temporal relation scores between two given events Ei and En. The prompt template “Ei <MASK> En” predicts scores fb(Ei,En) and fa(Ei,En) for the output tokens before and after. Similarly, we can obtain a reversed estimation by inputting “En <MASK> Ei.” Final temporal score f averages scores from
2In a sequence of temporally ordered events if A ≺ B and B ≺ C, then A ≺ C.
both directions: f(Ei,En) = 12(fb(Ei,En) + fa(En,Ei))
Our temporal predictor needs to be fine-tuned on a temporal relation corpus. Directly applying a pretrained LM encounter the problem of low coverage, where the tokens before and after cannot be found in the top-k prompted tokens (even k = 30). Thus, we fine-tuned masked language models to predict the masked connectives in a prompt learning setting. Intuitively, temporal relations exist between each pair of adjacent events in a chronologically ordered event sequence. Assuming an event sequence contains two adjacent events Ei,Ei+1, we then can create an example Ei before Ei+1 and an symmetric example Ei+1 after Ei. We also construct negative samples by replacing Ei or Ei+1 with a randomly sampled event from other sequences. Those negative examples can teach models when no temporal relation exists. While we mask before or after in a positive example for models to predict, a special token [none] should be prompted for negative examples. Then, the cross-entropy loss is used to optimize the temporal predictor. We call this fine-tuning process temporal fine-tuning. More details about the temporal relation dataset and fine-tuning process are shown in Appendix A.1.
Score Estimator With the temporal predictor, we can estimate P(X ≺ A) for all covariates X ∈ X and interventions A ∈ A: P(X(0), A(1)) = P(X ≺ A) = f(X,A). We also need to estimate P(X(0)) to compute conditional probabilities P(A(1) = 1|X(0)) in temporal propensity vectors q(x;A). As all covariates X are events preceding Ei sampled by GPT-J, there is an implicit conditioning on Ei. Thus, we can approximately get P(X(0)) ≈ f(X,Ei) (see Appendix B for more details). Then, temporal propensity vectors are computed as
q(x;A) =
( P(X(0), A(1))
P(X(0))
)
X∈X . (9)
Finally, the score estimator computes ∆̂i in Equation (7). We also test normalization methods in Appendix E and observe that some of those normalization methods can benefit our framework.

heading: 6 Experiment
text: We conduct extensive experiments and compare COLA with a wide selection of baselines.

heading: 6.1 Dataset
text: Since our work is the first attempt to study Contextualized CCR, we carried out human annotation on Amazon Mechanical Turk. We randomly sampled event sequences from ROCStories (Mostafazadeh et al., 2016), where each sequence contains five chronologically ordered events. Workers are asked to annotate whether an event causes the last event in a sequence. There are two qualification tests to choose workers to maintain rigorous quality control. See more details in Appendix D.
Eventually, we collected a dataset containing 1,360 event pairs, called Choice of Plausible Event in Sequence (COPES). We equally divide them into a validation set and a testing set.

heading: 6.2 Evaluation Metric
text: We calculate accuracy, F1-score, and Macro F1score between predicted labels and ground truth labels, to automatically evaluate all models on our dataset. Notice that our task definition provides the number of positive events in a sequence, so that recall, precision, and F1-score are the same.

heading: 6.3 Baseline Methods
text: We compare our framework to three baselines:
CLM Perplexity An intuitive solution to the contextualized CCR task would be computing perplexity scores for each pair of events with a causal language model (CLM). An event pair (Ei, Ej) within a sequence E1, E2, . . . , En (n is sequence length) is converted into full-text input with the prompt template: “If E1, E2, . . . , En, Ej because Ei”. The causal language models we tested are GPT2, GPT2-medium/large/XL (Radford et al., 2019), and GPT-J (Wang and Komatsuzaki, 2021).
Cloze Prompt Score This baseline proposed by Tamborrino et al. (2020) concatenates two events (Ei, Ej) into full-text input. Then, it masks and tries to recover each token with a masked language model. It averages log-likelihood over every token as the final score of two events. The prompt used is the same as CLM Perplexity. Multiple masked language models are tested: BERT-base/large (Devlin et al., 2018), RoBERTa-base/large (Liu et al., 2019), DeBERTabase/large (He et al., 2020).
ROCK This baseline is a causal inference framework (Zhang et al., 2022c) that draws analogies between human subjects and natural language. We
test different language models for the temporal predictor: BERT-base/large, RoBERTa-base/large, and DeBERTa-base/large.

heading: 7 Main Evaluation
text: We provide results in Table 1 for baselines and COLA with the temporal predictor based on different language models. In general, COLA can detect commonsense causal relations more accurately, outperforming all baseline models by a large margin. Our framework COLA based on DeBERTa-large and DeBERTa-base (also BERT-large) achieves the best performance on the validation and testing set, respectively. Also, changing language models of the temporal predictor in COLA only involves a small fluctuation in performance, showing that our framework is robust to underlying language models.
Another observation is that CLM Perplexity and ClozePromptScore can achieve performance higher than the random result. This manifests that pre-trained language models with commonly used pre-training objectives can capture commonsense causal relations to some extent.

heading: 7.1 Ablation Study
text: In this section, we conduct four ablation experiments to demonstrate that our causal inferencemotivated framework can mitigate spurious correlations between events and boost performance.
Temporal Propensity Matching The first three ablation experiments prove the effectiveness of temporal propensity matching. Here, we separately remove three modules in our framework: (i) We drop the multistamp covariate sampler and sample covariates only based on the last timestamp (⋄ w/o Multi Step). This experiment verifies the benefit of utilizing context to detect commonsense causality. (ii) We remove all interventions (⋄ w/o Inter) and use temporal precedence as causation: ∆̂i = P(Ei ≺ En), equivalent to ϵ = 0 in Equation (7). (iii) Covariates are removed (⋄ w/o Cov) so that interventions are not adjusted. This unadjusted score keeps all sampled interventions, equivalent to ϵ = 1 in Equation (7).
From the results in Table 2, we observe that balanced estimand ∆̂i achieve better performances, showing that multiple timestamp sampling, treat-
ment effect, and balancing covariates play essential roles in detecting commonsense causation accurately. Removing any of the three modules will result in sheer drops in all metrics. These experiments imply that temporal relation is vital in Contextualized CCR, but it is still insufficient due to spurious correlations. Thus, we need to measure average treatment effect with balancing covariates.
Temporal Predictor We also ablate the temporal predictor (⋄ w/o Temp) to verify the effectiveness of temporal fine-tuning (in Section 5). Here, we use pre-trained language models and increase k to 30 to mitigate the problem of low coverage.
As shown in Table 2, a directly pre-trained language model without fine-tuning cannot perform well. We conclude that pre-trained language mod-
els do not have sufficient “temporal awareness” and temporal fine-tuning is necessary for our framework.

heading: 7.2 Rules-of-thumb for Choosing ϵ:
text: The hyperparameter ϵ controls the number of interventions when balancing covariates in Equation (7). In Figure 3, we can observe that a recommended range for ϵ is ϵ ∈ [0, 0.1]. We also list the optimal ϵ in Appendix A.3. From Table 6, ϵ should be fairly small within [0.001, 0.015]. Though the best solution relies on how to implement the components in COLA and data distribution, our analysis can provide a good starting point. We also study the effect of changing another important hyperparameter: covariate set size N = |X |, in Appendix C.

heading: 7.3 Union and Intersection
text: Our framework COLA samples covariates from multiple timestamps and take union on them to get the final covariate set. We also introduce another method to sample covariates preceding all timestamps after and including Ei (“intersection”) in Section 5. Here, we conduct experiments to discuss the differences between these two methods. For the “intersection” method, we also use
GPT-J to sample covariates with “There are temporally ordered events [Ei, Ei+1, . . . , En]. Before all events,” being the prompt template.
As shown in Table 3, the “union” method gets better performance since it can diversify covariate sets. It samples covariates conditioned on each event before Ei separately. Meantime, each covariate of the “intersection” method is only conditioned on the same context Ei, Ei+1, . . . , En. We compute the self-BLEU (Zhu et al., 2018) to evaluate the diversity of the generated covariates. The self-BLEU of the “intersection” method is 66.40% while that of the “union” method is 41.34%, quantitatively showing that our method can diversify the covariate set.

heading: 7.4 Comparison with ChatGPT
text: Large language models have shown strong performance on extensive NLP tasks (Radford et al., 2019; Ouyang et al., 2022). Thus, we compare our framework with ChatGPT3, the latest large language model trained using Reinforcement Learning from Human Feedback (RLHF). To adopt ChatGPT to our task, we design the prompt template: “Given a story as a chain of events E1, E2, . . . , En, which k event(s) in the first n− 1 events more plausibly cause the last event?” where k is the number of positive events in the ground truth. We randomly sample 120 examples within 30 event sequences from our dataset COPES and manually read predicted labels from ChatGPT’s answers. We also test ChatGPT without providing the number k, i.e., removing k from the prompt.
The experimental result in the selected 120 examples is shown in Table 4. From the table, we can find that providing ChatGPT with the number k does not lead to too much change in all metrics. (More discussion about this in Appendix F) Also, our framework COLA achieves better zeroshot performance than ChatGPT with much fewer parameters.
3https://openai.com/blog/chatgpt/

heading: 8 Conclusion
text: In this paper, we design a new task to consider the context when detecting commonsense causal relations. We also crowd-sourced a dataset with strict quality control to benchmark the task. Our COLA framework is motivated by the principles of causal inference and attempts to leverage context information. The extensive evaluation demonstrates the effectiveness of COLA for detecting commonsense causal relations.

heading: 9 Acknowledgement
text: The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520 and 16205322) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to HKMAAC and CUSBLT. This paper was also supported by the Tencent Rhinobird Focused Research Program. We also thank the support from NVIDIA AI Technology Center (NVAITC) and the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).
Limitations
Our COLA framework is mainly based on pretrained language models, such as GPT-J and PolyJuice. We only utilize the simplest prompt to sample covariates and interventions. Further efforts such as prompt engineering are expected to boost the performance of COLA further. Also, pre-trained language models can be biased by implicit events and reporting biases in their training data. Such biases lead the framework to omit some critical covariates and interventions, hindering our framework from achieving more accurate detection of commonsense causal relations. How to account for this problem of language model remains to be studied. Last but not least, the temporal propensity used in our framework is only an approximation of the exact propensity (Zhang et al., 2022c) since it is unclear how to pack an unordered set of covariates into a sequential input for language models. Further studies are needed to design the exact propensity and improve performance.
Ethics Statement
This work presents COPES, a free and open dataset for the research community to study the contextualized CCR problem. Examples in COPES are collected from ROCStories (Mostafazadeh et al., 2016), a free and open dataset about anonymized chains of events. Each chain logically follows everyday topics and does not involve privacy problems about any specific entities (e.g., a person or company). We carried out human annotation on Amazon Mechanical Turk. Annotators are fairly paid 1.2 USD for each HIT, which fulfills the minimum wage requirement.

heading: B Probability Approximation
text: Our approximation of the probability of a covariate P(X(0)) in Section 5 is a consequence of the efficiency trade-off. Notice that even though a covariate X is an event preceding Ei sampled by GPTJ, it can occur before other events. Strictly, we must enumerate all possible events E, compute P(X ≺ E), and marginalize out E. Here, we assume P(E(1)|X(0)) ≈ 1 to compute P(X(0)) efficiently. We approximately have:
P(X ≺ E) = P(X(0), E(1)) = P(X(0))P(E(1)|X(0)) ≈ P(X(0))
(10)

heading: C Number of Covariates
text: We show that balancing covariates are essential in our framework COLA. Meanwhile, the number
of covariates also has an impact on performance. Here we test different covariate set size N = |X | spanning from 10 to 40, shown in Figure 4 with various ϵ values.
We observe that gradually increasing the covariate set size N can enhance the performance of our framework when the ϵ is within [0.001, 0.01] as we recommend in Section 7.2. On the other hand, adding more covariates may introduce more noise if we test with a larger ϵ. For example, when ϵ equals 0.02 and 0.025, increasing covariates only brings about fluctuations in accuracy.

heading: D Dataset Annotation
text: Since our work is the first attempt to study Contextualized CCR, we carried out human annotation on Amazon Mechanical Turk to construct a new dataset. In this appendix, we discuss the annotation process of the dataset COPES in our paper. We randomly sampled event sequences from ROCStories (Mostafazadeh et al., 2016), where each sequence contains five chronologically ordered events. Workers are provided with event sequences and are asked to annotate whether an event causes the last event in a sequence4. Each human intelligence task (HIT) includes ten potential (cause, effect) event pairs (with corresponding event sequences), and each pair is labeled by seven workers. We take the majority vote among seven votes as the final result for each pair.
We conduct two qualification tests to choose workers to maintain rigorous quality control. First, we invited annotators who meet the following conditions to take our qualification examinations: 1) an approval rate of at least 95% and 2) at least a thousand approved HITs. In the second round, a qualification question set including both effortless and tricky examples is collected by experts, namely the authors of this paper, who have a clear understanding of Contextualized CCR. The experts annotate 160 event pairs sampled from ROCStories. An annotator needs to answer a HIT involving ten questions from the qualification set, and the answers are compared with the experts’ answers. An annotator should correctly answer 8 out of 10 questions to pass the second round test. While 307 annotators participated in the second round qualification test, only 29 (9.45%) were selected as our main round annotators.
Eventually, we collected the dataset containing 4There may be multiple causes of an event.
human-annotated labels for 1,360 pairs from 340 event sequences (1360 = 340× 4), called Choice of Plausible Event in Sequence (COPES). The IAA score is 61.13% calculated using pairwise agreement proportion, and Fleiss’s κ (Fleiss, 1971) is 0.52. We equally divide them into a validation set and a testing set (e.g., each contains 680 examples). We also provide a screenshot of our annotation platform in Figure 5.
D.1 Dataset Statistics Here we provide the breakdown of numbers of causal relations k in all 340 event sequences in Table 7. Also, 476 event pairs are positive (with causal relation), and 884 are negative (without causal relation).

heading: E Normalization
text: In this section, we enumerate some methods to normalize estimand ∆̂ and temporal propensity vectors in Equation (7).
Direct Matching (D): Instead of condition probability in Equation (9), we can directly use the vectors of temporal relation scores (f(A,X))X∈X as propensity vectors.
Score Simplification (S): We fine-tune temporal predictors to prompt before, after, and [none], which might be difficult for smaller Masked LMs, like BERT-base. When constructing propensity vectors, we simplify the task to consider only before, after by normalizing scores:
f(X,A) = fb(X,A)+fa(A,X)
fb(X,A)+fa(X,A)+fb(A,X)+fa(A,X)
Propensity Covariate Normalization (Q): We also try to normalize temporal relation scores on the covariate set X before building propensity vectors: P(X) = f(X,Ei)/ ∑ X′∈X f(X
′,Ei) and P(X,A) = f(X,A)/ ∑ X′∈X f(X ′,A).
Co-occurrence Normalization (C): The finetuned temporal predictor may sometimes still faces
the problem of low coverage, causing estimates f(Ei,En) and f(A,En) inaccurate in Equation (7). We set them to (f(Ei,En) + f(En,Ei))/2 and (f(A,En) + f(En,A))/2, respectively.
Estimand Normalization (E): In this method, estimates f(Ei,En) and f(A,En) in estimand ∆̂ in Equation (7) are normalized by f(Ei,En) + f(En,Ei) and f(A,En) + f(En,A), respectively.
We also conduct comprehensive analyses about removing these normalizations. The results are shown in Tables 8 to 10. We present drops in every metric when removing each normalization method. Some normalization methods cannot improve performance, and we delete those rows from tables. We observe that some of these normalization methods, such as Estimand Normalization (E), can benefit our framework on Contextualized CCR.

heading: F ChatGPT
text: In this section, we show and discuss the result of ChatGPT on COPES. Given the insignificant differences in performance between the prompts with and without the number k in Section 7.4, we focus on the answers of the prompt with k. We show three examples, respectively corresponding to three cases:
1. ChatGPT’s answer is correct (with a good explanation)
2. ChatGPT’s answer is different from the ground truth answer, but the correctness is debatable with the explanation from ChatGPT
3. ChatGPT’s answer is incorrect.
Example 1. Which event mainly/directly causes the last event? E1: Charlie went to food truck event at the beach. E2: He said they had a ton of trucks there. E3: He tried food from a few trucks. E4: He said the food was great. E5: I am going to try to go next year. ——————————
Gold answer: E4 ChatGPT answer (short): E4 ChatGPT answer (original): The event "He said the food was great." directly causes the last event "I am going to try
to go next year." as it expresses the protagonist’s positive
experience and desire to attend again in the future.
Example 2. Which event mainly/directly causes the last event? E1: Ericka enjoyed going to restaurants, but she liked to save money too.
E2: She found a coupon for her favorite restaurant. E3: She went there for lunch and ordered a burger. E4: When the bill came, she was nervous about the coupon. E5: The restaurant accepted the coupon and Ericka saved a lot of money!
——————————
Gold answer: E2,E3 ChatGPT answer (short): E2,E4 ChatGPT answer (original): The second and fourth events mainly/directly cause the last event. The second
event ("She found a coupon for her favorite restaurant.")
sets up the possibility of her saving money, and the fourth
event ("When the bill came, she was nervous about the
coupon.") establishes that the coupon is being used in
this specific instance. Together, these two events lead to
the final event ("The restaurant accepted the coupon and
ericka saved a lot of money!") where she is able to save money.
Example 3. Which event mainly/directly causes the last event? E1: We decided to get married on the beach. E2: Our family and friends met us there. E3: It was a perfectly sunny summer day. E4: Best of all, no one had to wear shoes!. E5: We ended our day with a dip in the cool ocean. ——————————
Gold answer: E1 ChatGPT answer (short): E4 ChatGPT answer (original): The fourth event "Best of all, no one had to wear shoes!" mainly/directly causes the last
event "We ended our day with a dip in the cool ocean." as it
describes the condition that allowed the last event to happen:
people were barefoot.
In most cases, ChatGPT clearly explains its choices. For example, in the Example 1, it points out the positive emotion “... food was great.” as the underlying fact that causes the last event “... going to try to go next year.” (i.e., decision). Similarly in the Example 2, even though ChatGPT’s answer is different from the ground truth answer, ChatGPT mentions two key points “... The second event ... sets up the possibility of ...” (i.e., potential) and “.. the forth event ... establishes that the coupon is being used ...” (i.e., action) that together causes the last event (i.e., effect). The reasoning paradigm implicit in its explanation, e.g emotion → decision and potential + action → effect in the abovementioned two examples, implies the inherent reasoning capability of ChatGPT. Nonetheless, there are
cases where ChatGPT fails to give a persuasive answer. However, as a foundation model instead of a task model, ChatGPT’s performance is fair enough.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the Section Limitations, we discuss the limitations of our work.

heading: 3 A2. Did you discuss any potential risks of your work?
text: In the Section Ethics Statement, we discuss the ethical considerations, including privacy problems.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We annotate new datasets based on a previous dataset (Section 6, Appendix B, and Appendix D) and a new model (Section 4 and Section 5).

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We cite the previous dataset a few times in Section 6, Appendix B, Appendix D.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The artifacts we use is open sourced without a license.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. No intended use is stated for the existing artifact(s) in our paper.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The artifact we use is already anonymized.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The Section Ethics Statement.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 6, Appendix B, and Appendix D.
C 3 Did you run computational experiments? Sections 6 and 7.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 6 and Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix B.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 6 and Appendix B.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 6 and Appendix D.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix D

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 6, Section Ethics Statement, and Appendix D.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 6 and Appendix D.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. For anonymity.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. For anonymity.

heading: Future Work
text: " as it expresses the protagonist’s positive
experience and desire to attend again in the future. Example 2. Which event mainly/directly causes the last event? E1: Ericka enjoyed going to restaurants, but she liked to save money too. E2: She found a coupon for her favorite restaurant. E3: She went there for lunch and ordered a burger. E4: When the bill came, she was nervous about the coupon. E5: The restaurant accepted the coupon and Ericka saved a lot of money!
——————————
Gold answer: E2,E3 ChatGPT answer (short): E2,E4 ChatGPT answer (original): The second and fourth events mainly/directly cause the last event. The second
event ("She found a coupon for her favorite restaurant. ")
sets up the possibility of her saving money, and the fourth
event ("When the bill came, she was nervous about the
coupon. ") establishes that the coupon is being used in
this specific instance. Together, these two events lead to
the final event ("The restaurant accepted the coupon and
ericka saved a lot of money!") where she is able to save money. Example 3. Which event mainly/directly causes the last event? E1: We decided to get married on the beach. E2: Our family and friends met us there. E3: It was a perfectly sunny summer day. E4: Best of all, no one had to wear shoes!. E5: We ended our day with a dip in the cool ocean. ——————————
Gold answer: E1 ChatGPT answer (short): E4 ChatGPT answer (original): The fourth event "Best of all, no one had to wear shoes!" mainly/directly causes the last
event "We ended our day with a dip in the cool ocean. " as it
describes the condition that allowed the last event to happen:
people were barefoot. In most cases, ChatGPT clearly explains its choices. For example, in the Example 1, it points out the positive emotion “. food was great. ” as the underlying fact that causes the last event “. going to try to go next year. ” (i. e. , decision). Similarly in the Example 2, even though ChatGPT’s answer is different from the ground truth answer, ChatGPT mentions two key points “. The second event. sets up the possibility of. ” (i. e. , potential) and “. the forth event. establishes that the coupon is being used. ” (i. e. , action) that together causes the last event (i. e. , effect). The reasoning paradigm implicit in its explanation, e. g emotion → decision and potential + action → effect in the abovementioned two examples, implies the inherent reasoning capability of ChatGPT. Nonetheless, there are
cases where ChatGPT fails to give a persuasive answer. However, as a foundation model instead of a task model, ChatGPT’s performance is fair enough. ACL 2023 Responsible NLP Checklist