COMET-M: Reasoning about Multiple Events in Complex Sentences
Authors: Sahithya Ravi
Raymond T. Ng
Vered Shwartz
Venue: EMNLP 2023 Findings (2023)
Keywords: Reasoning about events
Complex contexts
Discourse
TL;DR: Multi Event inference

Abstract: Understanding the speaker’s intended meaning often involves drawing commonsense inferences to reason about what is not stated explicitly. In multi-event sentences, it requires understanding the relationships between events based on contextual knowledge. We propose COMET-M (Multi-Event), an event-centric commonsense model capable of generating commonsense inferences for a target event within a complex sentence. COMET-M builds upon COMET (Bosselut et al., 2019), which excels at generating event-centric inferences for simple sentences, but struggles with the complexity of multi-event sentences prevalent in natural text. To overcome this limitation, we curate a Multi-Event Inference (MEI) dataset of 35K human-written inferences. We train COMET-M on the human-written inferences and also create baselines using automatically labeled examples. Experimental results demonstrate the significant performance improvement of COMET-M over COMET in generating multi-event inferences. Moreover, COMET-M successfully produces distinct inferences for each target event, taking the complete context into consideration. COMET-M holds promise for downstream tasks involving natural text such as coreference resolution, dialogue, and story understanding.

COMET-M: Reasoning about Multiple Events in Complex Sentences
Sahithya Ravi Raymond Ng' Vered Shwartz!?
University of British Columbia Vector Institute for AI (sahiravi, rng, vshwartz]@cs wbc.
Abstract
In this work, focus on deriving event-centric commonsense inferences from sentences with mul- tiple events, prevalent in written and spoken lan- guage. Consider an ambiguous news headline such as "Stevie wonder announces he" 'Il be having kid- ney surgery during London Concert"l. Deriving inferences from such sentences is challenging due to the intricate causal and temporal relationships between events. For example, the announcement occurs during the concert, but the surgery happens later. Moreover, is crucial to stay true to the con- text when making inferences about specific events. Existing commonsense knowledge models such as COMET (Bosselut et al., 2019) can generate commonsense inferences along dimensions such causes, effects and the mental states of the event participants. For example, given the sentence "Ste- vie Wonder will be having surgery", COMET pre- diets "he sick" as a reason. However, since COMET was trained on simple sentences with a single predicate, it falls short on deriving infer- ences from multi-event sentences, as we show in Fig. 1. COMET's inferences conflate the different events, for example, predicting "he goes to the hos- pital" both event that happened before (the surgery) and after (the announcement). Mapping the inferences back to the events that they refer to is not trivial. In addition, the assumption that the two events happened at the same time leads to incorrect inferences such as "he has to cancel the concert". This hurts COMET's applicability in tasks involving complex sentences.
Understanding the speaker's intended mean- ing often involves drawing commonsense in- ferences to reason about what is not stated explicitly. In multi-event sentences, it re- quires understanding the relationships between events based on contextual knowledge. We propose COMET-M (Multi-Event), an event- centric commonsense model capable of gen- erating commonsense inferences for target event within complex sentence. COMET-M builds upon COMET (Bosselut et al., 2019), which excels at generating event-centric in- ferences for simple sentences, but struggles with the complexity of multi-event sentences prevalent in natural text. overcome this limitation, curate Multi-Event Inference (MEI) dataset of 35K human-written inferences. We train COMET-M on the human- written in- ferences and also create baselines using auto- matically labeled examples. Experimental re- sults demonstrate the significant performance improvement of COMET-M over COMET in generating multi-event inferences. Moreover, COMET-M successfully produces distinct in- ferences for each target event, taking the com- plete context into consideration. COMET-M holds promise for downstream tasks involving natural text such as coreference resolution, dia- logue, and story understanding.
1 Introduction
Human understanding of narratives involves build- ing mental representations the described events (Pettijohn and Radvansky, 2016). We make com- monsense inferences about the unstated but plau- sible causes, effects, and mental states of the par- ticipants. This fundamental ability is crucial for NLP systems for human-level performance tasks such question answering, summarization, and story understanding. Additionally, reasoning about events can help improve the coherence and consis- tency of generated text, making it more natural and human-like.
To address this challenge, we introduce the task of Multi-Event Inference (MEI) which involves generating distinct inferences for each target event within a complex sentence. We collected the Multi- Event-Inference (MEI) dataset consisting of 35k (sentence, target event, dimension, inference) tu- ples. We use MEI to continue training the latest

---

He needed to to the hospital
He wanted to be healthy
Stevie Wonder announces he'll be having kidney surgery during London
Rifect
concert
COMET- ATOMIC
Hindracoe
He didn't
want want ruin the concert
Bodore
performs the concert, goes hospital
Aftr
performs the concert, goes hospital
Haghhorquittee
Stevia needs know about the surgery
wants
statement
Stevie Wonder he'll be having kidney surgery cuuinng London concert.
The fans disappointed
COMET-M
Hindrazoe
Stevie Wonder does not attend the concert
Stevie Wonder
the event
Aftr
Stevie Wonder thanks the fans for their support
Stevie Wonder very sick
Stevie wonder has kidney disease
Stevie Wonder ennounces he'll be having kidney surgery during London concert.
Stavia Wonder recovers from the surgery
COMET-M
Stavia Wonder has money for the surgery
Stovio Wander goes to the hospital
Afoez
Stevie wonder recovers from surgery
Figure I: Top: COMET's inferences mix up the events announces (inferences in blue) and having [a surgery] (inferences in pink). Inferences in black are incorrect. Bottom: COMET-M improves upon COMET by generating inferences for the target events, announces and having. Each set of inferences clearly refers single event, e.g. he prepares for the event before the announcement, and rushed to the hospital before the surgery.
version of COMET (Hwang al., 2021), creat- ing COMET-M, a new multi-event commonsense model. COMET-M can generate distinct inferences for each target event in the sentence, while taking into account the entire context. As shown at the bottom of Fig. I, when the announces event is the target, our model predicts that he would later thank his fans for their support. Conversely, for the hav- ing event, predicts that he will then recover from the surgery.
and reading comprehension (Rashkin et al., 2018). Earlier approaches represented event types terms of their participants and subevents, in a structured representation referred to as a "script" (Schank and Abelson, 1975) or "narrative chain" (Chambers and Jurafsky, 2008). For example, person gets hired at company, works for it, and then quits, gets fired, or retires. Script knowledge for common events is typically mined from text (Chambers and Juraf- sky, 2008; Pichotta and Mooney, 2014; Rudinger et al., 2015; Weber al., 2020). Knowledge about common events could then be leveraged order to represent and make predictions about new events.
COMET-M improved upon COMET in the MEI task in terms of both automatic and human evalua- tion. Additionally, we found that baselines trained synthetic multi-event inferences generated from LMs improve the original COMET in some aspects, but may not be sufficient to reach the same level of performance.
Later efforts leveraged crowdsourcing to collect large-scale knowledge graphs of event-centrie com- monsense (Sap al., 2019; Mostafazadeh et al., 2020). The ATOMIC knowledge graph (Sap al., 2019) defines events along nine dimensions per- taining to causes, effects, and the mental states of the event participants. It consists of 880k (event, re- lation, inference) triplets collected through crowd- sourcing. The latest version of ATOMIC was ex- tended to 1.1M triplets and additional relations (Hwang et al., 2021). Although ATOMIC is large- scale, it cannot cover the complete range of event- centric commonsense knowledge. Additionally, aligning a given context with the knowledge graph can be challenging, due to lexical variability and because inferences are context-sensitive
COMET-M has the potential to benefit discourse tasks such as summarization. dialogues and story understanding. Further, MEI can be used to create tasks of varying difficulty for future research.
2 Related Work
Reasoning about events is essential for a wide range of NLP tasks such as summarization (Pighin et al., 2014), natural language generation (Chen et al.,
2021), dialogue systems (Ghosal et al., 2021, 2022),
"Code, models and data are available here.

---

To that end, Bosselut al. (2019) developed COMET (COMmonsensE Transformers). COMET is based on pre-trained Language Models (LMs) such as GPT (Radford et al., 2018) or BART (Lewis et al., 2020), which are further fine-tuned on struc- tured knowledge from ATOMIC. As a result, it can dynamically generate event-centric common- sense inferences along ATOMIC's dimensions for new contexts. This hybrid approach is success- ful because it combines the structure and accuracy of numan-writted commonsense facts with the vast commonsense knowledge that LMs learn from their massive pre- pre-training corpora (Petroni et al., 2019; Davison et al.. 2019), COMET is widely used and has been applied to many downstream tasks such dialogue and question answering (Kearns et al.. 2020; Majumder et al., 2020; Sabour et al., 2021; Kim et al., 2022; Ravi al., 2023a).
Retation
Question
HasPrerequisite
What are typecally the prerequisites even!
isllefore
What typically happens immediatelly before event?
iskfter
What ty typically happens immediately after the event?
cause
Causes
What could be the effeet of the event?
HinderedBy
What can hinder the event?
Table 1: Event-centric relations in COMET-M and the corresponding question templates used for crowdsoure- ing.
an openly accessible, generic model that can gen- erate inferences along multiple dimensions and is applicable across multiple domains.
Multi-Event- -nffen-- Dataset
In order to create the Multi EeenE--veenneeeee (MEI) dataset, we obtained annotations of target events within complex sentences, through crowdsoureing (Sec 3.2). As the souree of the events, consid- ered datasets from different domains (Sec 3.1).
Several subsequent variants COMET have been released recent years. COMET-distill (West et al., 2022b) is version of COMET that is trained on ATOMIC-style events obtained by knowledge distillation from GPT-3 (Brown et al., 2020), while VisualCOMET (Park et al., 2020) generates infer- ences about the causes and effects of events de- picted in an image. Finally, COMET-ATOMIC 2020 (Hwang et al., 2021) is the most recent ver- sion of COMET, which was trained on more data and larger underlying LM.
3.1 Events
Within each dataset domain, we extract events from the longest sentences belonging to the 50 most frequent topics.
News formal domain. We use 500 of the gold- labeled event mentions across different topics from the ECB+ dataset (Cybulska and Vossen, 2014), which is a collection of news articles annotated with entity and event coreferences.
The variant of COMET most relevant to our work is ParaCOMET (Gabriel et al., 2021b), paragraph-level COMET model. The key differ- ences between COMET-M and ParaCOMET are as follows. First, COMET-M is trained on vari- ous genres for broader applicability, whereas Para- COMET was trained on the ROCStories corpus (Mostafazadch et al., 2016) in the fiction genre. Second, COMET-M is trained using human-written inferences, unlike ParaCOMET's silver-standard supervision. Our evaluation in Sec 6 demonstrates that silver-standard baselines lag behind the gold- standard baseline in the MEI task on important aspects. Further, COMET-M focuses on generat- ing inferences for multiple events within a single sentence, while ParaCOMET focuses on simple sentences in multi-sentence paragraphs.
Dialogue. We sample 200 events from the DREAM dataset (Sun et al., 2019), which includemuuliple-chiie questions that were ex- tracted from 6,444 dialogues. For this dataset and the following ones, spacy's rule based matching recognize event mentions by matching contiguous verbs. (Honnibal al., 2020).
Narratives.
The SamSum dataset (Gliwa et al.. 2019) is composed of 16k chat dialogues with man- ually annotated summaries. We the one-line summaries instead of dialogues add naturally written sentences to our dataset. addition, we use WikiPlots collection of plot summaries for more than 42,000 books and movies. We collect annotations for 400 events from Samsum and 200 events from WikiPlots.
In another recent work, Ravi al. (2023b) im- proved an existing event coreference system by incorporating GPT-3 generated implicit events that likely occurred before or after each event mention. Differently from Ravi et al. (2023b), COMET-M is

---

Instructions:
Read the context sentence write least two inferences for each question.
Context: Bryant Dalton was shot and spent several weeks medical facility.
Question 1:
What are typically the prerequisites the event spent?
Question 2:
What typically happens immediately before the event spent?
Question 3:
What typically happens immediately after the event spent?
Figure 2: An example Human Intelligence Task (HIT) Amazon Mechanical Turk.
Blogs. The Webis TLDR Corpus (Völske et al., 2017) corpus consists of 5,848,330 posts from Red- dit, with an average length of 270 words. We sam- ple 300 events.
Train 60%
Dev
Test
10 %
30
Inferences
21454
3576
10726
Target predicates
919
156
467
Complex contexts
557
145
358
Overall, we collected 1,600 events. To measure the diversity these events, we calculated the per- cent of unique bigrams, which was 81%, indicating substantial diversity. In addition, we examined the distribution of the number of events in each sen- tence across the datasets. We found that 62% of the sentences have two events, 16% contain three events, 13% include four events, 6% feature five events, and 3% include only one event. Finally, to assess the complexity of the events, we sampled 200 multi-event sentences and manually analyzed the temporal relationships between the every pair of subsequent events E; and E in the sentence. We found the common relationship to be: Eg causes E (35% of the cases), Eş happens before E,+1 (24%), Eg happens after Es+1 (18%). E, is the ef- fect of F (16%), and F, a prerequisite of E (6%). These results underscore the presence of tem- poral interdependencies among multiple events in the sentences,
Table 2: Statistics of the train, development, and test splits of Multi-Event Inference (MEI) dataset (Sec 3).
example HIT.* In the second set, we asked about the plausible causes, effects, and hindrances of the event (rows 4-6 in Table 1). In both HIT types, we acquired least two inferences for each relation, and collected inferences from two workers per HIT. On average, we obtained inferences per relation per instance. Overall, we collected 35,418 infer- ences. We manually reviewed the annotations and removed 2-3% inferences of poor quality, such as incomplete or irrelevant sentences and events an- notated containing offensive hate speech. We randomly split the data into train (60%), test (30%) and validation (10%) splits, such that the events do not overlap between splits. We show the statisties of events, contexts and inferences in each split in Table 2.
3.2 Inferences
Model
We focus on 6 event-centric relations from COMET, presented in Table 1. We collected commonsense inferences for the events we extracted in Sec 3.1 along these relations. The annotation task was conducted on Amazon Mechanical Turk (MTurk). To ensure the annotation quality, we required that workers have completed 5,000 prior tasks with a minimum acceptance rate of 98%. We limited the worker population native English speakers (US, UK, Canada, Australia, NewZealand) and required workers to pass a qualification test similar to the annotation task. We paid 20 cents for each example or Human Intelligence Task (HIT).
The goal of Multi-Event Inference (MEI) is gen- erate commonsense inferences for a target event within complex sentence. Formally, given a con- text sentence C that includes event mentions F1, E2, .... Eg., we want to generate a set of com- monsense inferences for each event E, across the 6 inferential relations shown in Table 1. As shown in Figure 1, it is crucial that the inferences generated for each event F, are consistent with the overall context C and specific to the event Ej-
We build COMET-M based on an existing ver- sion of COMET (Hwang et al., 2021) based on BART (Lewis et al., 2020). The motivation is that COMET already captures an abundance of event-centric commonsense knowledge. By contin- uing to train COMET on the Multi-Event Inference
We divided the relations into sets. In the first set, we asked workers to generate inferences about plausible things that happened immediately before or after the target event, and the prerequisites of the event (rows 1-3 Table l). Figure shows an
*See Appendix A for the full template.

---

(MEI) dataset Sec 3, we can leverage this knowl- edge while further adapting the model to generate event-specific commonsense inferences. The train- training
COMET and train it on multi-event inferences. The difference is that rather than training on the gold- standard MEI as in the case of COMET-M, we train the models on silver-standard multi-event in- ferences generated automatically, as detailed be- low.
ing is done with a sequence-to-Se objective with the following input and output for- mat:
C. Rr [GEN] T;
where the imput is composed of C, the context con- taining event F. enclosed between <TGT> tokens, and R, special token denoting the relation. [GEN] is a special delimiter token that indicates the start the tail of given relation for a given head entity. Our format is the same as that used in COMET except for the <TGT> special token added as new special token to indicate the event of interest. For maintain
This bascline takes complex sentences and automatically splits them into simple sentences in the format COMET is familiar with. Specifically, we take the complex sentences from our training set (Sec 3), and split them into SVO triplets using MinlE (Gashteovski al., 2017), Open Information Extraction (OIE) system. As op- posed other OIE systems, MinlE removes overly specific modifiers, yielding simple and short sub- jects, objects, and predicates. We use the safest mode that eliminates the least amount of unnec- essary context. MinIE returns multiple SVO for each complex sentence, with the predicate corre- sponding to target events. We feed these SVOs into COMET generate inferences along the var- ious dimensions. The MinlE SVOs along with the events (predicates) and their corresponding COMET inferences serve as silver standard super- vision for our task. We fine-tune COMET on this data in the same manner as COMET-M (Sec 4).
COMET-M-Split.
consistency with COMET training, the same relation names. The output T is the target inference in free text.
During inference, we prompt the model in the same format as the training heads, C₁ R; [GEN]
5 Experimental Setup
5.1 Baselines
We evaluate two types of baselines on the Multi- Event Inference (MEI) test set. The first type con- sists of off-the-shelf models that are used without any modifications and are not trained on multi- event sentences (Sec 5.1.1). The second type of baselines comprises supervised models that fine-tuned on multi-event inferences generated au- tomatically, rather than relying on gold standard annotations (Sec 5.1.2).
We adopt the same strat- egy as COMET-M-Split and develop a technique to selectively choose event-specific inferences that are best compatible with the context of the com- plex sentence. For a given context sentence C, and relation r. we first generate the full inferences {4, In) from COMET. Then, choose the inferences for a target event Ey from this full set I, we look at the inferences generated by the MINIE-based split sentence S, for the same rela- tion f {K. 13......... We pick the inferences from 1 that have cosine similarity score greater than 0.7 with any inference in I. Since COMET conflates inferences of multiple events, this can be seen as a way to select event event-specific inferences that are consistent with the context. We then fine- tune COMET on these selected inferences in the same way as COMET-M. This strategy is similar to the one adopted in ParaCOMET (Gabriel et al., 2021a).5
COMET-M-Overlap.
5.1.1 Off-the-Shelf Models
We generate inferences from the following models using the same input format described in Sec 4.
We use a pre-trained BART model that was not specifically trained to generate common- sense inferences.
BART.
COMET. We use the off-the-shelf COMET model from Hwang et al. (2021), which was trained on ATOMIC-2020 but not on multi-event infer- ences.
5.1.2 Supervised Models
We train several baselines with the same setup de- scribed in Sec 4, i.e. we initialize the model with
'Although BART and COMET do not support multiple events, and generate the same inferences for all events within a sentence, we add the <TGT> special token to indicate different events.
"We don't directly compare with an off-the-shelf Para- COMET model since it is limited to the fiction domain and uses an older L.M.

---

COMET-M-NLI. We follow the same approach as COMET-M-Overlag but devise different method to filter out event-specific inferences that are inconsistent with the context. We use a pre-trained NLI model based on RoBERTa (Liu et al., 2019), which was fine-tuned on Multi-NLI (Williams et al., 2018). Given the context sentence C, target event E expressed in a split sentence Sj, relation r, and 12, the inferences generated by COMET for the same relation in the simple sen- tence, we predict the entailment label between C and each inference E P. For most relations, we keep all inferences that are entailed by the context and discard other inferences. For the Hinderdby relation, we keep inferences that contradict the con- text, since we are interested in scenarios that make the context less plausible. Again, we fine-tune COMET these filtered inferences in the same way as COMET-M.
that, generated inference considered correct as long as overlaps with one of the reference in- ferences. We also report the average BERTScore (Zhang et al., 2020) which measures semantic simi- larity between the set of ground truth and generated inferences. As opposed to n-gram overlap based metrics, BERTScore doesn't penalize models for lexical variability.
We present the results of our automatic eval- uation Table 3. line with the findings pre- sented in COMET (Hwang et al., 2021), a notable disparity in performance is observed between the zero-shot BART and COMET models, emphasizing the value of incorporating explicit learning from commonsense knowledge graphs in addition to pre- training language.
The COMET-M-Split model demonstrates im- proved performance compared to the standard COMET model, owing to the fact that it is fine- tuned specifically to generate multi-event common- sense inferences, through the use of automatic la- and COMET-M-NT
COMET-M-Mimie Inspired by West et al. (2022a), we generated silver-standard inferences for all MEI training examples by prompting Chat- GPT 50 with instructions and exam- ples. Similarly to the other baselines, we continue training COMET on this data. The specific instruc- tions and format can be found in Appendix C.
bels. COMET-M-Ovenee further exhibit superior performance, as they incor- porate methods for filtering out contradictory and irrelevant inferences relation to the context. No- tably, COMET-M-NLI, which explicitly removes inferences that contradict the context, stands out as the best-performing automatic labeling method based on COMET. COMET-M-Mimic performs slightly better than COMET-M-NLI, reassessing previous findings larger model to generate training data for smaller models (West al., 2022a; Kim et al., 2023). Ulti- mately, COMET-M surpasses all variants, thanks to its training with Homman-written multi-event infer- ences. These results are demonstrated through high scores across all metrics, with substantial margin from the next best method (e.g. +10 in ROUGE).
5.2 Implementation Details
We train all methods (except the COMET and BART baselines) with the same hyperparameters as the original COMET, listed in Appendix D. During inference, we use beam search with a beam size of 5 and set the maximum decoding length to 50 tokens.
the effectiveness of using
6 Results
6.1 Automatic Evaluation
Following prior work, we measure the quality of the generated inferences from all models with re-
6.2 Human Evaluation
We assessed the quality of commonsense infer- ences generated by COMET, COMET-M-NLI (as described in Sec 5.1) and COMET-M (as deseribed in Sec 4) through manual evaluation. Table 4 presents the results, which were obtained by ran- domly sampling 100 events from the test set. We used the same MTurk qualifications as in Sec 3 and paid 10 cents per HIT.? Three workers were presented with a sentence, target event, and two inferences generated by each model for the rela-
spect to automatic metrics. We report both BLEU-2 (Papineni et al., 2002) and ROUGE-L (Lin, 2004) scores with respect to the reference inferences. We measure both BLEU and ROUGE as follows. For a given (context, target event, relation) triplet, we remove repetitions from both generated inferences and ground truth inferences. Each generated infer- ence is then scored against all the references. We take the maximum of score with any reference in- ference for this input. We then average this score across all generated inferences. The intuition is
"See Appendix A for the HIT template.

---

Model
Training Data
ROUGE-L
BLEU-2
BLEU4
BERTScore
BART
11.256
4.452
1.570
50
COMET
Gold standard (simple sentences)
15.855
8.391
3.798
59.2
COMET-M-Splee Split
Silver standard (multi-event)
18.071
10.074
4.681
60.5
COMET-M-OVern
Silver standard (multi-event)
18.438
10.781
4.916
60.5
COMET-M-NIL
Silver standard (multi-event)
21.205
12.614
5.627
61.0
COMET-M-Mimes
Silver standard (multi-event) event)
23.87
14,291
6.892
61.6
COMFT-M
Gold standard (multi-event)
33.560
25.077
12.412
64.9
Table 3: Test performance of off-the-shelf (Sec 5.1.1), silver-standard (Sec 5.1.2) and gold-standard baselines (Sec Multi-Event-Intee Inferecce (MEI), with respect to automatic metries (x100) computed on the generated inferences and compared to the reference inferences.
H
%M
%L
sometimes likely (M) with respect the given re-
Likelihood
44
30
26
lation. COMET, on the other hand, generated low likelihood for the target events 26% of the cases, which can be attributed to the fact that inferences of multiple events may be conflated. This is also confirmed by COMET's specificity scores being very low.
COMET
Specificity
26
44
30
Relevance
63
37
Likelihood
45
40
15
COMET-M-N M-NII
Specificity
31
53
16
Relevance
80
20
Likelihood Specificity
43
COMET-M HOMET-M-Minge M-Miic
COMET-M-NLI performs better than COMET on both likelihood and specificity, as is trained on inferences targeting specific predicates and is less prone to mixing events. Yet, the inferences are less specific compared to COMET-M, where 62% of the inferences were considered specific to the target event. COMET-M's specificity score leaves room for improvement, with significant portion of moderately specific inferences that pertain other events the context (27%).
Relevance
19
Likelihood
COMET-M
Specificity
Relevance
19
Table 4: Human evaluation results for the inferences generated by COMET, COMET-M-NIJI and COMET- M. H, M. and L denote high, moderate, and low.
tions in Table 1. Each inference was rated based on the following aspects:
COMET-M-Mines performs on par with the NLI variant on relevance, but fares much worse specificity, which means that it conflated in- ferences for different events. Further, that ChatGPT's inferences that were similar to the context, which is likely a result of its conservative "behaviour" and attempts to not make assumptions that go beyond the provided input.
Likelihood. How likely is the event in the given context? Always likely (High-H), sometimes likely (Moderate-M),, or never likely (Low-I.).
noticed
Specificity. How specific is the inference to the given target predicate? Highly specific (H), partial overlap with other events (M), or completely about another event (L).
Finally, all models scored highly in terms of relevance to the given context, with COMET-M achieving higher relevance of 81% % and COMET-M-NLI closely following with 80%.
How relevant the inference to the entire sentence? Relevant (H) or irrelevant (L).
Relevance.
The results show that COMET-M outperforms COMET, COMET-M-NLI and COMET-M-Minil in all three metrics. COMET-M-NLI and COMET- M-Mimic show better performance than COMET, but fall short in different aspects.
Analysis
7.1 Qualitative Analysis on Held-out Corpus
In order to test model's ability complex sen- tences unseen in the collected multi-event corpus, sampled 25 contexts from a held-out dataset, the DailyDialog (Li al., 2017). This is multi- turn dialogue dataset which contains conversations about every day life. We found 3 (out of 5) in-
In terms of likelihood, 69% COMET-M's in- ferences were always likely (H), and 24% were
We usked raters to mark inferences that simply repeat the target event verhatim as invalid and set low (I.) for all metrics for such inferences.

---

You see, the bungee participants take deep breath when they stand the starting spot and then, like diving.
their heads are over hools, and they jump off into the realm of the combination of heaven and carth
stand
take
take a brenth
take deep breath
they jump off of
they go heaven
COMET
they fall down
they got dizzy of
to jump off of
take hreath X
they get dizzyy
they down
they go to the starting spot of
take deep breath
they get ready for the race
to take a deep breath X
COMET-M-Split
they get the starting spol
bungee
they get the finish line
take deep breath. X
they get the
the bungee jump
the best the bungoe jump X
to be safe
the bungee jump X breathe deeply to have fun of to be happy x
COMET-M M. Overlap
have of be safio of to he happy
they to the starting spot of they stand the starting spot they are the starting spot of
the hungee participants to take a breath bungee purticipants take deep breath the bungee participant to take a breath breath
COMET-M-NII
they get bungee
bungee team
The bungee purticipunts scured heights X The bungoe participants feel dizzy x
bungee participunis are scured of heightt The bungee participants foel dizzy, of They have fear heights. of
COMET-M-Mimic
They have fear of heights.
The bungee participants feel exhilarated.
The participants are scared
The hungeo participants are ready to jump of They want jump off the bunger
The bungee participants are nervons of The bungee participants want relax They are nervous of
COMET-M
They wanted to jump into the air
They are at the starting spot /f They wanted to jump from height
They are jumping high /
They dizzyy
Table 5:
Inferences generated by different models for the two events stand and take in complex sentence taken from the held-out DailyDialog dataset for the relation xReason i.e the cause of the event (Li et al., 2017).
ferences to be highly specific to the target event in 90% of the contexts and true to the context in 100% of the contexts. In Table 5, we present the top inferences generated by each of the models for an example sentence from the held-out corpus. As expected, COMET tends to conflate the two events, stand and take, generating similar inferences for both. COMET-Split generates distinct inferences for the two events, but it is apparent from the infer- ence "They get ready for the race" that it doesn't 't consider the context.
(Sec 6.2).
Finally, COMET-M demonstrates both speci- ficity and relevance context by accurately in- ferring that the reason for the stand event that the participants are preparing to jump, while the reason for taking a deep breath is that the participants are feeling nervous.
7.2 Diversity of multi-event inferences
To further investigate whether inferences generated for different events are semantically distinct, we compare the inferences generated by COMET and COMET-M for different events within the same context.
Among the two baselines that filter out infer- ences, COMET-Overlap also conflates the infer- ences for the two events, likely because generic inferences such as "to be safe" and "to have fun" are applicable to many events. COMET-NLI per- forms better by selecting inferences that are con- sistent with the context, but it tends to repeat the context. This happens because the NLI-based filter- ing keeps inferences that repeat the context, since they entail it. COMET-M-Minin performs better in deriving accurate inferences for the take deep breath event, e.g. deducing that the participants are scared of heights. However, it generated similar inferences for the stand event, demonstrating a lack of specificity, in line with the human evaluation
We focused on the test set and measured the di- versity between the sets of inferences generated for each event within the same sentence. To quantify this, we use the ratio of distinct bigrams all the bigrams in the inferences of all events belonging to the same given context. We obtained a diver- sity score of 0.65 for COMET-M and 0.47 for the original COMET, confirming that COMET-M can better distinguish the multiple events and generate different outputs for them.
Figure 3 displays a t-SNE projection of the sen-

---

Causes
T-called
Story Understanding. Consider the example in Figure 4. The story is composed of two sentences with multiple events. COMET-M can be a valuable tool in unraveling the chain of events that form the story, allowing us to interpret the underlying mean- ing by reading between the lines. For instance, we can connect the two sentences by understanding COMET-dIO didn'
Tedla offiall moved la hoociquatorss from Falo Aos Callornia 30 Ausiin Toxas CEO
COMIT insuthed
Mask ennoureet company''s 2021 aroul shanerdoow mraeting
COMET-IN 14 called
Foe Sonthus
COMET dinn'n
Apet 2000, on Trels harninge cod, Muss inched out at Ca Comto government officile calling their dimporayy covlo rolated health orders "lanca an ouckirp laced 1art.
Figure COMET-M can be useful in deriving causes and effects to fill gaps hetween different lines in a story.
that covid-related health orders caused the com- pany to close down. Similarly, the effects of the moved event, can help in deducing that the com- pany is able focus more its business in the new location.
Figure 3: A t-SNE projection of the sentence em- beddings of 50 inferences generated by COMET and COMET-M for the Causes relation of the various target events the sentence "John insulted Mary, so she didn't reply when he called her".
8 Conclusions
We focused the task of generating common- sense inferences for multiple events in complex sentences. To address this task, we proposed a new model, COMET-M, trained COMET-M on human- written inferences that we collected for multiple events within complex sentences. Our experiments, through automatic and human evaluations, demon- strated the effectiveness of COMET-M in generat- ing event-specific and relevant inferences. In the future, we intend to investigate the effectiveness of incorporating COMET-M into discourse tasks such as dialogue, summarization, and story generation.
tence embeddings' of 50 inferences generated by COMET and COMET-M for the Causes relation of various target events in a sentence with three events: "John insulted Mary, so she didn't 't reply when he called her". As can be observed, the in- ferences from COMET-M form more distinct clus- ters, particularly for the events of call and insulted. Conversely, COMET generations for the different events overlap significantly.
7.3 Usecases for COMET-M
COMET-M can be used incorporate common- sense knowledge into any task that uses real world text, in particular discourse tasks such as event coreference, story understanding and dialogue. Ex- amples from an unseen dialogue task are explored in Sec 7.1 and the potential of multi-event infer- ences in solving event coreferences is shown in Ravi al. (2023b). Hence, we elaborate here on the usecase of story understanding.
Limitations
Automatic Evaluation Metrics. Lexical-overlap based automatic metrics were shown to have low correlation with human judgements on various NLG tasks (Novikova et al., 2017). This is espe- cially true for commonsense tasks, which are more open-ended in nature and where various answers may be acceptable. We collected 5-6 reference in- ferences per relation from human annotators, which cannot cover all plausible inferences for an event.
Event Mention Detection. Currently, ap- proach involves a two-step pipeline where we first identify target predicates, and then generate infer- ences for those predicates. We plan to simplify this in future, by developing a model that can detect tar- get events and generate inferences for these events.
wirraphrase-iinLL--GGG.. model from HuggingFace Sentence Transformers

---

Specificity. Human evaluation (Sec 6.2) revealed that COMET-M generated inferences that were not specific enough to the target event in 27% of cases. We leave it to future work to investigate whether training COMET-M on more data can reduce this.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877-1901.
Ethical Considerations
Data. The datasets used to gather the base events outlined in Sec 3 are publicly accessible. Some of these datasets include information from blogs and forums (e.g. Reddit) which may contain offensive, biased or hateful content. To minimize this, we asked our annotators to label any events they found to be offensive. We excluded such events. Still, we can't 't fully guarantee the absence of such content from our data.
Nathanael Chambers and Dan Jurafsky. 2008 Unsuper- vised learning of narrative event chains. In Proceed- ings of ACL-08: HLT, pages 789-797, Columbus, Ohio. Association for Computational Linguisties.
Hong Chen, Raphael Shu, Hiroya Takamura, and Hideki Nakayama. 2021. GraphPlan: Story generation by planning with event graph. In Proceedings of the 14th International Conference on Natural Language Generation, pages 377-386, Aberdeen, Scotland, UK. Association for Computational Linguisties.
Crowdsourcing. We collected 35k inferences and evaluated 1,200 inferences through crowd- soureing. The participants were compensated with an hourly wage of 16 USD, which is comparable to the minimum wages the US and Canada. We did not collect any personal information about the participants from MTurk.
Agata Cybulska and P. Vossen. 2014. Using a sledge- hammer to crack nut? lexical diversity and event coreference resolution. In LREC.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pages 1173-1178, Hong Kong. China, Association for Computational Linguistics.
Models. Our data sources, which include news events and opinions from blogs, may have inherent bias in the commonsense inferences (e.g, based on pronouns such as He/She or country names from news corpora). Our models that are trained on such inferences may produce offensive or biased content for certain events. Pre-trained language models used in our work which are trained on large corpora from the internet may introduce such bias into our models.
Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, and Yejin Choi. 2021a. Paragraph-level commonsense transformers with recurrent memory. In AAAI.
Saadia Gabriel, Antoine Bosselut, Jeff Da. Ari Holtz- man, Jan Buys, Kyle Lo, Asli Celikyilmaz. and Yejin Choi. 202 Discourse understanding and factual consistency in abstractive summarization. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 435-447, Online. As- sociation for Computational Linguistics.
A Aeknowledgements
This work was funded, in part, by the Vector Insti- tute for AI, Canada CIFAR AT Chairs program, an NSERC discovery grant, and a research gift from A12.
Kiril Gashteovski, Rainer Gemulla, and Luciano del Corro. 2017. MinlE: Minimizing facts in open in- formation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2630-264) Copenhagen, Denmark. Association for Computational Linguis- tics.
References
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021. Cider: Commonsense inference for dialogue expla- nation and reasoning. In SIGDIAL Conferences.

---

Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022. Cicero: A dataset for contextualized commonsense inference dialogues. ArXiv, abs/2203.13922.
Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11192
Bodhisattwa Prasad Majumder, Harsh Jhamtani, Tay- lor Berg-Kirkpatriek and Julian McAuley. 2020. Like hiking? you probably enjoy nature: Persona- grounded dialog with commonsense expansions. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9194 9206, Online. Association for Computa- tional Linguistics.
Bogdan Gliwa, wonn Mochol, Maciej Biesek, and Alck- sander Wawer 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Nasrin Mostafazadeh, Nathamael Chambers, Xiaodong Hc, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial- strength natural language processing in python.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras. Jeff Da. Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In AAAL
William R Kearns, Neha Kaura, Myra Divina. Cuong Vo, Dong Si, Teresa Ward, and Weichao Yuwen. 2020. A wizard-of-oz interface and persona-based method- ology for collecting health counseling dialog. In Extended Abstructs of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-9.
Nasrin Mostafazadeh, Aditya Kalyanpur. Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll 2020. GLUCOSE: Gener- alzi and COntextualized story explanations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4569-4586, Online, Association for Computa- tional Linguistics.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani. Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization.
Jekaterina Novikova, Ondřej Dušek, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2241-2252, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Seungone Kim, Sehrang Joo, Hyungjoo Chae, Chae- hyeong Kim, Seung won Hwang, and Jinyoung Yeo. 2022. Mind the gap! injecting commonsense knowl- edge for abstractive dialogue summarization. ArXiv, 00900000.0000..
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871 7871-7880, Online, Association for Computa- tional Linguistics.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mot- taghi, Ali Farhadi, and Yejin Choi. 2020. Visual- comet: Reasoning about the dynamic context of a still image. In ECCV.
Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: manually labelled multi-turn dialogue dataset. Proceedings of The 8th International Joint Conference on Natural Language Processing (IJCNLP 2017).
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Kyle A Pettijohn and Gabriel A Radvansky. 2016. Nar- rative event boundaries, reading times, and expecta- tion. Memory & cognition, 44(7):1064-107)

---

Karl Pichotta and Raymond Mooney, 2014. Statisti- cal script learning with multi argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 220-229, Gothenburg, Sweden. Asso- ciation for Computational L Linguistics.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu. Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue-based reading compre- hension. Transactions of the Association for Compu- tational Linguistics, 7:217-231.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization,, pages 39-63, Copenhagen, Denmark. Association for Com- putational Linguistics.
Daniele Pighin, Marco Cornolti, Enrique Alfonseca, and Katja Filippova. 2014. Modelling events through eemorrybasedd open-IE patterns for abstractive sum- marization. In Proceedings of the 52nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 892-901, Baltimore, Maryland. Association for Computational Linguis- tics.
Noah Weber, Rachel Rudinger, and Benjamin Van Durme, 2020. Causal inference of script knowl- edge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7583-7596, Online. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training. OpenAl blog-
Peter West, Chandra Bhagavatula, Jack Hessel. Jena Hwang, Liwei Jiang. Ronan Le Bras, Ximing Lu, Scan Welleck, and Yejin Choi. 2022a. Symbolic knowledge distillation: from general language mod- els commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-4622 Seat- tle. United States. Association for Computational Linguistics.
Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psy- chology of characters simple commonsense stories. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics VVolume 1: Long Pupers), pages 2289-2299, Melbourne, Aus- tralia. Association for Computational Linguistics.
Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023a. Vlc-bert: Visual question answering with contextualized common- sense knowledge. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion (WACV), pages 1155-1165.
Peter West, Chandrasekhar Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras. Ximing Lu, Sean Welleck, and Yejin Choi. 2022b. Symbolic knowledge distillation: from general language mod- els to commonsense models. In NAACL.
Sahithya Ravi, Chris Tanner, Raymond Ng. and Vered Shwartz. 2023b. What happens before and after: Multi-event commonsense in event coreference res- olution. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu-
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies. Volume 1 (Long Papers), pages 1112 1122. Association for Computational Linguistics.
tational Linguistics, pages 1708- 1724, Dubrovnik. Croatia. Association for Computational Linguistics.
Rachel Rudinger. Pushpendre Rastogi. Francis Ferraro, and Benjamin Van Durme. 2015. Script induction as language modeling. In Proceedings of the 2015 Con- ference Empirical Methods in Natural Language Processing, pages 168 1-81-1 Lisbon, Portugal, As- sociation for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Appendix
Sahand Sabour, Chujie Zheng, and Minlie Huang. 2021. Cem: Commonsense-awan empathetic response gen- eration. In AAAI Conference on Artificial Intelli-
A HIT Template
In this section, we provide the templates used for both data collection Figure 5 and human evaluation Figure 6 for the first three relations shown in Table
gence.
Maarten Sap, Ronan LeBras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if- then reasoning. In AAAI.
1.
B Input-Output COMET-M
In Table 6 and Table we provide examples from our datasct and inferences generated by COMET- M.
Roger C Schank and Robert P Abelson. 1975. Scripts, plans, and knowledge. In IJCAI, volume 75, pages 151-157.

---

Task: Pre-requisite, Before and After
Context
<contextt)
Pre-requisite
1. What are typically the prerequisites for the event $(vvent)?
(Avard words from event)
(wite an inference 3-15 words]
(witte another inference 3-15 words]
(witte another infermnce 3-15 worde- optiona]]
Before
typically happens immediately before the event #(avent)?
(Avold reusing words from the event)
Before
twitee Inferance wordk]
(witte another inference 3-15 words]
(witte another
(write another optaral)
After
3. What typically happens immediately after the event (event)?
(Avard reusing words from event)
After...
(wite an inference 3-15 wordk]
(witt another infermnce 3-15
optional)
(witte another inference optional)
Optional Feedback #1: This event predicate does not make sense in the given context.
Optional Feedback This context/event has hateful//flensive content.
Optional Feedback Something the unclear/You additional feedback:
We plan to post many rounds of these HITs in the near future.
Submit
Figure 5: Crowdsourcing template for obtaining before and after inferences.
C Prompts Template
different events. COMET-M exhibits improve- ment in diversity score as the inferences for a given event-relation focus on the target event alone. In future work, we aim to explore improving the diversity of generated inferences while remaining specific to the target event.
In Table 8 we show the prompt used for ChatGPT,
D Hyperparameters
The models were trained on a single GPU (NVIDIA GeForce GTX 1080 Ti). We show the hyperpa- rameters in Table 9.Diiverity of inferences We calculate the portion of unique 3-grams out of all 3-grams for the inferences generated per event- relation and report the results in Table 10. We note that silver-standard variants that mix the in- ferences of other events or deviate the provided context may have inflated diversity scores, as the variability among the inferences may due to

---

Reed a contevt inforences thout whar usutity tappene beftre and after trrget owene the proroou sites the event: inferense baaed lisall a heppan betors/atberraa prenequtats event speciiil and true to coment, you have
betore, p cose some time road the instrcct
Instructions
Tharks parteputin
be askood reed contect and asked infernnos about
ewent
each event, will provide Inferances:
Infrrmees sbout
inmmedienly this
this will assess these inforences and them based these aspeets (quesitins):
Newr likey the inference the premequ batte//btooriattte the exent (chooes one of: peter//mnttraaeeeeennn
inference tue for relevont to the contect contrsdcctigg incevvntt) (chosse are yeshelinaaid) infereece speciti targer ewest (choose onr yeo/partt tyrs in;
invali catagery only inforences
inferences just repeat event.
ecamptes below undimstand
internnoes.
Examples: Thr event in bold, (emaan//Coerran
Task: Rate event commonsense inferences
Context
(conteet)
prerequisites
Bolluate these inferences what NEEOS hoppen before, Le preregustt the eve (event)
Now Mikely inference prerequaate evont?
((rea))
shap/clia: sotatared Bicy forlds voltiont Des Alrsuoo enak)
this inference specthic to event #(even))?
[Le does with other events same connect))
to
Fartaly speditc
Unietted the eeet
Now likely inference prerequiate "waid//ppets ever)
((oo2)
swayychee conetersohbbll Sutectetprwe
this inference specific to the target event ((vvnn)) (1e does Unvioned evest
evonts same contect))
8s Paruls speats
Before
Pollaae ote these infevenons uint hoppees before the eveer ((vent))
Now Whely i this Inference before the event?
5(before3)
aiorpisiin: sarataendl ferlds vollsosr
Drad Atrxput caal)
given
this specific
event ((event))
[La it not mie with other eventa the sume comeet)) tec
tara'y quart:
sovee eveet
How likely inference before the event?
#(beforo2)
vasyocttr 13 =stetrracibbll Sutishettoeee "ie www.d//ppeess everr)
conte
Wechegating the
this interence specthic
#<event)?
dics nit rix with other ovents in the same Cortoot
Fartsl spedtt
Un-sated the eeet
Figure 6: Crowdsourcing template for human evaluation

---

Contest
Clusetsiteson.)
Lifectss (This Causes..)
Hindereliyy
Human Written
speay container
was overgrown.
The person
criming
The penson
tramming.
tinming I...] weed killer
The overgrown unkemgee
The more onderly
The were alreddy clinsedd
suked into the ground
The ugly
Penuu piersel with hush looks
The penon forgon hodge clipper
set
must
Thee person
roses
The penon
weedkilllr
gottes kusckte over, weed killer Persou set
roses.
The gound gets covered
They guid attutioa to they
souked into the ground near the roees.
The container was not closed.
The person can not get rid of weeds
Penson did not buy weed killer
Generatee
My Iruyer me you Neceepptial our They ware in long term relationship almony propusal and the division of
The person happy with ontcume
My lucyer nut war help me
They wanted divorce
The person is glad they accepted it
The alimony propoeal was rejected
property...)
They have a disagreestent their merriage
They say goodbye
The person does nox agres to the alimney
My lowyr [.], as well the csstady
The gerwon warns keep
The
The penan doss nor cam about
agrvement- Il keep the cot and you pet the The person wams lave a
The person is happy have cal
The othrr person does not want the dog
They hai two pels
The person takes cane the cael
The penam doss mi vomt the call
Table Human- (top) model-generated (bottom) examples from Multi-Evvent COMET for the xReason and Causes (what an event leads to) and HinderedBy relations. Some examples are slightly abbreviated for readability.
Context
(Refore thi.))
iAAeer (Aftr this..)
HesPienquistt (Nooded before)
Homan- Weillen
Tar the speay containee while was
He finds the molss
Ganlan looks nicze
The penan has toal.
trimming bush. [-.]] the weed killer
He fill his speay container.
Cleans up the bush
The westher is guod
ground
roses.
The person sharpess trinmer
Bush tranmed
The persoa has gacden
Tsst 1. and the container must have
The continer falls The person knucks it
I snstch the bottle form ground
They turs around nim
gotimn knocked over, and the weed killer
The mass will.
They se the spray containte down The container lox liquid
sudled into the ground the roscs. The peraon did not sce the contrincr
The person clams with
MedeatGeneratedd
My lawyer tells me you Ye accepted our They discuss the temns of the slimoay alimnny propussal and the divieaene of propertyl
The persou hupps with the agreement
They have a lawyer
The per talks the lawver
The person is glad they secppedd it
There is legel sgreement need married
The beuyer makes presebtttoon the couple
conclude the agremmnt
My lowyer I-J, as well the custody
They both meet
The person happy with the outcomt
The persoa has car The ofher person wants the dog
sgreensent & keep the and you the They diacus the castady agreementt
The person hupps have the eat takes
dog
sgreed divide propart..
Table 7: Human written (top) and model generated (bottom) examples from Multi-Event COMET for the isBefore. isAfter and Pasereqquee relations. Some examples are slightly abbreviated for readability.

---

System Message:
You are an expert annotator for commonsense reasoning tasks
User Mesage:
In this task, you will be asked to read a context and asked to write inferences specific to a target event enclased between <TGT> tag.
Generate 4 inferences for EACH of the following relations:
isBefore: What typically happens immediately before this event?
isAfter: What typically happens immediately after this event?
HasPrerequisite: What could be the prerequisite of this event?
Causes: What can cause this event to happen?
HinderedBy: What can hinder this event from happening?
Effects: What can be the effect of this event?
Instructions: Inferences are specific to the target event enclosed in <TGT> but need to stay true the context. Refer to the examples provided below. Do not mix inferences of different events the context. Provide diverse inferences think of many possibilities for given scenario. You need generate inferences for the next example Return the response
JSON format shown below.
1. The Bedford arrested last night charges of double homicide <TGT> killed <TGT> his mother and ex-girlfriend, according police and prosecutors
("isBefore": ["The decides murder them.", "He buys gun knife.", follows them their houses", keeps watching them"], "isAfter": ["He hides their bodies." "He regrets his decision.", "He cleans the blood from the scene". "HasPrerequssite" ["The man finds mother have weapon.", "He plans the murder". "He has heart to kill them. "J, "Causes": ["The man hates his mother
tries get of there"],
ex-girlfriend", "He needs to
his girlfriend." " "He is mentally il1", "He has bad memories with them", "He fought with them many times"),
"Effects": ["The family members of the deceased mourn them.". "The police search for him", "The man gets caught", , "It appears on the news"], "HinderedBy": ["The man could not find them", "He gets nervous". "He changes his mind", "They are out of town", "Someone intervenes"])
2. The New Bedford man <TGT> arrested <TGT> last night on charges of double homicide killed his mother and ex-girlfriend, according to police and prosecutors
["Two people are killed", "The police are convinced that he is the primary suspect", "He is on the CCTV footage", "The police find evidence"], "HinderedBy": ["The man flees from the country", "The police did not have an arrest warrant", "They did not find supporting evidence", "The man kills himself after"].
("Causes":
"Effects": ["The man goes to jail", "The family appreciates the police for their timely arrest.", ", "The media applaud the police", "He goes to prison"], "HasPrerequisite": ["The police need arrest warrant.", "He must not escape.", "The police chase after the suspect.", ", "The police receive a complaint"], "isAfter": ["The man is taken to the station", "The man calls his lawyer", "The police hold a press conference" "The police talk to prosecutors"],
"isBefore": ["The police chase after the suspect", "The police receive complaint", "The police find the bodies", "The police take out their handcuuffs"])
Table 8: Prompt used for ChatGPT.

---

Parameter
Value
Epochs
2
Batch Size
Learning Rate
le-5
Optimizer
AdamW
Table Hyperparameters used all supervised model
versions.
Model
Distinet-3-grams
COMET
62
COMET-M-NLI
63
COMET-M-Minie
64
COMET-M
63
Table 10: Distinct 3-grams for a given event and relation among the inferences generated by beam search


Reviews:
[{"review_id": "Zzo0Q8Svkh", "reviewer": "Reviewer_2nKM", "summary": "", "questions": "A. Have you experimented with not continuing training on COMET? \nB. Any results that trained with the larger model? Such as fine-tuning GPT3.5/4, or LLaMA?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback, and for identifying the potential contribution of the paper to enhance downstream tasks.\n\n**A. Have you experimented with not continuing training on COMET?**\n\nThanks for suggesting this interesting ablation. We retrained COMET-M by initializing with BART instead of COMET and obtained a ROUGE of 26.2.  While this baseline still improves upon COMET by ~10 ROUGE points, it lags significantly behind the COMET-M baseline initialized with COMET (ROUGE = 33.5). This is expected, since, COMET is trained on more than one million commonsense triplets, and is exposed to the task format as well as the event-relations. Hence, transfer learning from COMET instead of from BART is beneficial for the MEI task. We will include this baseline as an ablation in our paper. \n\n**B. Any results that trained with the larger model?**\nSince COMET is still widely used in commonsense reasoning research [1, 2], we aimed to improve upon it.  For assessing how to enhance COMET on Multi-Event-Inference, we compared against COMET trained on the responses generated from GPT 3.5 instead of directly fine-tuning GPT 3.5. Given the potential of larger LLMS such as GPT 3.5 (or Llama), we expect their fine-tuned versions to perform at least similarly, if not better, than COMET-based baselines. With the fine-tuning support released for GPT 3.5 recently, we will include a baseline fine-tuned on gpt3.5 (or llama2) in the final paper. We also note that the released MEI dataset can be used to train/evaluate any of the larger models as future work.\n\n**#1a. How diverse are the events in the dataset? (In line 195, you specify the top 50 topics; what are they? Why only cover 50?) How many types are there? How complex are the sentences in the dataset? Are there mainly only two events in the dataset?**\n\nThanks for highlighting the need to add additional details about MEI dataset.  To answer your question on event diversity, we measured the diversity of 1.5K  target predicates by counting the number of unique bigrams across all bigrams. We found the diversity score at 0.81, indicating substantial diversity. Regarding event distribution across sentences, out of 557 unique complex sentences, 62% have two events, 16% contain three events, 13% include four events, 6% feature five events, and 3% include only one event.  We will include this in the paper.   Regarding L195, the topics are pre-defined in the original datasets. For instance, in ECB+ dataset, the topics refer to different seminal events, while in the reddit dataset, they refer to different subreddits. We decided to pick events from the top 50 most frequent topics to represent each dataset, and make the annotation process more manageable and efficient.  \n\n**#1b. What are the failure cases in your COMET-M model based on your human evaluation's feedback?**\n\nHumans found 11% of the inferences from COMET-M to be non-specific to the target event. To demonstrate a failure, let us consider the same example: Stevie Wonder “announces” he'll be having kidney surgery during a London concert. Though the majority of the inferences are specific, a failure occurs when COMET-M predicts “Stevie Wonder needs to be very sick” as one of the prerequisites for both the ‘announces’ and ‘kidney surgery’ events. \n\n**#1c. How much different will your model react for the same target event but paired with different co-occurred events? For example, comparing \"Stevie Wonder announces he'll be having kidney surgery during London concert\" to \"Stevie Wonder announces he'll get married during London concert.\"**\n\nWe generated the inferences of \"Stevie Wonder announces he'll get married during London concert.\" to see how different they are from “Stevie Wonder announces he'll be having kidney surgery during London concert\" [Figure 2]. COMET-M predicts that, “Stevie Wonder prepares a speech” before the ‘announces’ event, and that, ‘Stevie Wonder gets lots of congratulations’ after.  For the ‘getting married’ event, COMET-M predicts that “Stevie Wonder buys a wedding ring” before, and that “He is very happy” after.  This indicates that the event interactions influence the predictions accurately.\n\n**#2. Dataset size**\n\nWith respect to dataset size, we believe that our dataset size of ~1.5K events and ~45K commonsense triplets is reasonable, especially considering that we are transfer learning from a strong commonsense model i.e., COMET which is trained on 1.1M triplets. We focus on six important event-based relations pertaining to causes, effects, hindrances, and prerequisites of events, and leave it to future work to investigate object-oriented relations.\n\n[1] Cui, Wanyun and Xingran Chen. “Free Lunch for Efficient Textual Commonsense Integration in Language Models.” Annual Meeting of the Association for Computational Linguistics (2023).\n\n[2] Zandie, Rohola et al. “COGEN: Abductive Commonsense Language Generation.” Annual Meeting of the Association for Computational Linguistics (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An exciting artifact that improves the previous model, COMET's drawback --- does not have detailed enough control on the event.\n2. I can foresee the potential contribution of the paper to enhance downstream tasks. As for real-case NLP, reasoning on complex sentences is a much more important issue we need to approach.", "reasons_to_reject": "1. The analysis of the dataset is weak. We would suggest at least providing these in the paper:\na. How diverse are the events in the dataset? (In line 195, you specify the top 50 topics; what are they? Why only cover 50?) How many types are there? How complex are the sentences in the dataset? Are there mainly only two events in the dataset? \nb. What are the failure cases in your COMET-M model based on your human evaluation's feedback?\nc. How much different will your model react for the same target event but paired with different co-occurred events? For example, comparing \"Stevie Wonder announces he'll be having kidney surgery during London concert\" to \"Stevie Wonder announces he'll get married during London concert.\" Understanding this can help the reader understands more about how the event's interaction influences the predictions.\n\n2. The dataset size is small (in terms of event counts), and the relation coverage is limited compared to COMET.", "questions_to_authors": "A. Have you experimented with not continuing training on COMET? \nB. Any results that trained with the larger model? Such as fine-tuning GPT3.5/4, or LLaMA?", "paper_topic_and_main_contributions": "The paper introduces a new human-annotated resource, the Multi-Event Inference dataset. This dataset mainly extends the setup of COMET with more specific event control on more complex sentences. Four domains are covered in the dataset, totaling around 1.5K events. Experiments mainly show the quality of the data is better than decompose complex sentences or using GPT3.5-turbo as data annotator.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 47, \"Stevie Wonder\"\n2. Line 206, missing space\n3. Line 946, Not sure what this sentence should be...."}, "extra_text": {}}, {"review_id": "CPutw4sJei", "reviewer": "Reviewer_VGTR", "summary": "", "questions": "How to extract the events in a complex sentence? Is the lexical information of events are taken into consideration and how? Do the relations of events affect the event-centric reasoning even the same context is given? Will COMET-M benefits from both silver standard and gold standard annotations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the importance of reasoning in multi-event complex sentences and for the helpful feedback .\n\n**How to extract the events in a complex sentence?** \n\n**Formal formulation and concrete example about what the events are.**\n\nThanks for pointing this out. In this paper, events refer to multiple target predicates in a sentence. It is the equivalent of verbs obtained by POS tags or semantic role labeling. In this sentence: “If you liked the music we were playing last night, you will absolutely love what we're playing tomorrow!”, the four events would be: liked, were playing, love, ‘re playing.\nThe target events/predicates are extracted by finding the verbs in the sentence. This can be achieved by semantic role labeling, information extraction or part-of-speech detection.  We use a simple rule-based matching algorithm based on parts-of-speech tags [1] to find specific patterns based on the parts of speech (e.g auxiliary verbs followed by verbs such as “is watching”, verbs followed by prepositions such as “bring about”). We will include the rules used to extract events in the paper. \n\n**Is the lexical information of events are taken into consideration and how?**\n\nWe explicitly instruct humans to consider the entire context of the complex sentences to write inferences. When annotators are asked to label event inferences, we believe that they consider the lexical information, because they interpret and understand the events based on the words and language used to describe them. \n\n**Given the same context, the relations of events will affect the inferences.** \n\n Could you please elaborate on this question?\n\n\n**Will COMET-M benefits from both silver standard and gold standard annotations?**\n\nThat is a great question. Our experiments on combining the NLI-based silver standard annotations with gold-standard resulted in worse performance [e.g ROUGE=32.65]. Likewise, combining ChatGPT inferences results in no performance gain w.r.t automatic metrics [e.g ROUGE = 33.53].  This is also reflected in the human evaluation, where the silver standard based models perform poorly, especially on specificity. We aim to explore how to filter the obtained silver standard annotations or perform more complex data distillation from larger models in future work.\n\n**A randomly initialized model trained on the proposed dataset MEI needs to be considered as a baseline to confirm the effectiveness of the annotations about multi-event inferences.**\n\n We trained a randomly initialized BART model on the proposed dataset MEI and obtained a ROUGE score of 26.3 points.  This model performs better than COMET on the task of MEI, but worse than the model initialized with COMET and further trained on MEI. Given that COMET is trained on millions of commonsense triplets, this result is expected, and this was our motivation to perform transfer learning from COMET instead of BART. \n\n **Figure 4 is difficult to read, is there any statistics of the repetition of words or n-grams counts to show the diversity of multi-event inferences.**\n\nWe will include the missing references and a better version of Figure 4 in the paper. Following your suggestion, we quantitatively measured the diversity of multi-event inferences in the test set, using the ratio of (distinct bigrams/all possible bigrams) of inferences across events belonging to the same given context.  We obtained a diversity score of 0.65 for COMET-M and 0.47 for the original COMET. This indicates that there is higher diversity in the inferences across events of the same context, for COMET-M compared to COMET. This is in line with what we try to demonstrate in Figure 4. We will include this in the paper.\n\n**The details about the dataset are missing**\n\nThanks for pointing out that we need to include more details and statistics about the MEI dataset. The target predicates have a diversity score of 0.81, obtained by counting the number of unique bigrams across all bigrams. While a majority i.e 62% of our sentences have two events, MEI includes 16% complex sentences with three events, 13% sentences with four events, 6% sentences with five events and 3% with a single event.  We sample our dataset from different domains to ensure topic diversity.  Given the extra page for the camera-ready version, we will include the statistics as well as an analysis of the topics covered in the paper.\n\n[1] https://spacy.io/api/match", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe reasoning of multi-event complex sentences is an important problem in language understanding and generation.\n2.\tThis paper creates a new dataset with human-written inferences corresponding to multiple events in long sentences collected from news, dialogue, narratives, and blogs.\n3.\tThe proposed model COMET-M show the effectiveness of event-centric annotations on commonsense reasoning of complex sentences.", "reasons_to_reject": "1.\tDescription of the motivation is not very clear. It would be better if the authors give a formal formulation and concrete example about what the events are. It is not very clear about the impacts of lexical relation between events and the contextual information inner sentences on the annotations of inferences.\n2.\tThe details about the dataset are missing to measure the coverage, distinction, balance, and key statistical characteristics of it.\n3.\tThe baseline models need to be elaborated more. Besides the off-the-shell models, BART and COMET, all other supervised models are initialized the with COMET and then trained on multi-event inferences. It seems like finetuning COMET to adapt to the downstream tasks. It might make the comparison between COMET and COMET-M unfair because the data distributions of them are different. A randomly initialized model trained on the proposed dataset MEI needs to be considered as a baseline to confirm the effectiveness of the annotations about multi-event inferences.", "questions_to_authors": "How to extract the events in a complex sentence? Is the lexical information of events are taken into consideration and how? Do the relations of events affect the event-centric reasoning even the same context is given? Will COMET-M benefits from both silver standard and gold standard annotations?", "paper_topic_and_main_contributions": "This paper proposes COMET-M to generate commonsense inferences for multiple target events within a complex sentence. To overcome the limitation of lacking data about multi-event sentence annotations, the authors curate a Multi-Event Inference (MEI) dataset of 35K human-written inferences. They train COMET-M on MEI and create baselines using automatically labeled examples by prompting Chat- 388 GPT (gpt-3.5-turbo) with instructions and examples. The experiments, through automatic and human evaluations, demonstrated the effectiveness of COMET-M in generating event-specific and relevant inferences.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1.\tMuhao Chen, Hongming Zhang, Haoyu Wang, and Dan Roth. 2020. “what are you trying to do?” semantic typing of event processes. In Proceedings of the 24th Conference on Computational Natural Language Learning (CoNLL 2020). Association for Computational Linguistics.\n2.\tLi Zhang, Qing Lyu, and Chris Callison-Burch. 2020c. Reasoning about goals, steps, and temporal ordering with WikiHow. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4630–4639, Online. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "In section 4, the format of input and output needs to be clarified.\nFigure 4 is difficult to read, is there any statistics of the repetition of words or n-grams counts to show the diversity of multi-event inferences."}}, {"review_id": "kkPupOo7Ss", "reviewer": "Reviewer_QbAU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful feedback and questions.\n\n**Why fine-tuning BART on the dataset performs better than ChatGPT-prompted answers, especially given that ChatGPT has been shown to be proficient in commonsense reasoning.I would like to raise my scores if my consideration could be solved.**\n\nThat is a good question. Regarding ChatGPT’s performance on reasoning tasks, it is important to note that we did not finetune ChatGPT, but used the responses from a 6-shot ChatGPT [shown 6 in-context examples]  to train COMET.  This means that we are not evaluating ChatGPT's ability to perform MEI, but we are the ability of a smaller model [here BART] that uses ChatGPT as the annotator instead of a human. The reason why this automatically labeled baseline fails compared to human annotations is two-fold. First, though ChatGPT has been shown to be proficient in commonsense reasoning, it lacks training on the task structure and specific relations of MEI/COMET.  A supervised version of ChatGPT is expected to perform at least similarly, or even better, than the BART-based versions. Second, simply imitating a large model’s responses (ChatGPT) may result in small models (BART) mimicking the style of the larger model, but not its reasoning abilities [1]. So it may be a failure of BART to effectively learn from ChatGPT responses.\n\n**Why COMET fails on MEI**\n\nWith respect to COMET’s poorer performance on MEI- COMET is primarily trained on single-event sentences and mixes up the inferences for different events when presented with a multi-event sentence as shown in Figure 1.  We further show the intricacy of multi-event sentences in our results in Table 2 and analysis in Table 4. We show that COMET does not perform well on the MEI task, even when we decompose the sentence and feed it to COMET (COMET-M-Split and COMET-M-NLI). We believe that this empirically shows why COMET is not designed to work with multi-event sentences. \n\n[1] Gudibande, Arnav et al. “The False Promise of Imitating Proprietary LLMs.” ArXiv abs/2305.15717 (2023): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors have demonstrated the effectiveness of their dataset by evaluating several baseline models and showing that their proposed models outperform the baselines. Overall, the paper is well-written and presents a clear and comprehensive description of the dataset.", "reasons_to_reject": "While the paper presents a valuable contribution by curating an annotated multi-event sentence dataset, I would like to see more experimental analysis of the dataset. Specifically, it would be interesting to see how human annotation considers the relationship between multi-events, as this would provide insights into the complexity of the dataset and demonstrates why COMET fails. Additionally, I would like to see a more detailed analysis of why fine-tuning BART on the dataset performs better than ChatGPT-prompted answers, especially given that ChatGPT has been shown to be proficient in commonsense reasoning. I would like to raise my scores if my consideration could be solved.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main contribution of this paper is the authors have curated a Multi-Event Inference (MEI) dataset of 35K human-written inferences. The MEI dataset is designed to overcome the challenges of complex reasoning in multi-event sentences", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new dataset for multi-event inference on complex long sentences. This is an addition to the line of common-sense inference resources such as COMET and ATOMIC. The key contributions include the resource itself, a demonstration of the utility of the resource on the proposed task.

The main problem of wanting to address reasoning over complex sentences and the gap with respect to an existing resource like COMET is well motivated. The solution methodology of obtaining a manually curated resource is reasonable. The resulting resource is likely valuable. The paper is well written overall with the caveat of the lack of some details and motivations for the design choices, as identified by the reviewers. The originality of the work lies in stating and tackling a problem that hasn't been addressed yet. The methodology is sound overall and the resulting resource is of high quality. The scope and scale of the resource however are not as big as that of prior resources on similar problems. This is hard task to create resources for and the authors make a reasonable empirical case for the utility of the resource.
Decision: Accept-Findings