ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems
Year: 2023

Abstract:
Commonsense reasoning is omnipresent in human communications and thus is an important feature for open-domain dialogue systems. However, evaluating commonsense in dialogue systems is still an open challenge. We take the first step by focusing on event commonsense that considers events and their relations, and is crucial in both dialogues and general commonsense reasoning. We propose ACCENT, an event commonsense evaluation metric empowered by commonsense knowledge bases (CSKBs). ACCENT first extracts eventrelation tuples from a dialogue, and then evaluates the response by scoring the tuples in terms of their compatibility with the CSKB. To evaluate ACCENT, we construct the first public event commonsense evaluation dataset for open-domain dialogues. Our experiments show that ACCENT is an efficient metric for event commonsense evaluation, which achieves higher correlations with human judgments than existing baselines.

heading: 1 Introduction
text: Open-domain dialogue systems aim to have natural and engaging conversations with users (Chen et al., 2017). The abundance of dialogue corpus (Dziri et al., 2018) and the development of neural models (Radford et al., 2019; Lewis et al., 2020) enable open-domain dialogue systems to generate grammatically correct and meaningful responses (Zhang et al., 2020d; Bao et al., 2021; Ghazarian et al., 2021). Despite the success, systems still struggle to consistently produce commonsensecompliant responses as humans do. As shown in Figure 1 Example A, the generated response is not compliant with commonsense since “need
∗ Equal contribution † The work was done while the author was conducting a summer internship at UCLA. ‡ The collaboration started when the author was a graduate student at USC.
an oxygen mask” is not a reasonable prerequisite for “like to paint”. Commonsense issues for dialogue systems can also be manifested when we consider the dialogue history. For instance, in Figure 1 Example B, the system’s response “That is interesting!” after the user talks about their car accident violates commonly accepted social norms (Frischmann, 2021).
In this work, we study automatic dialogue commonsense evaluation by focusing on event commonsense (Sap et al., 2020; Rashkin et al., 2018), which concerns commonsense knowledge about events and their relations. Our focus on event commonsense is motivated by the following three observations: First, advanced open-domain dialogue systems have been pre-trained on large corpus and thus suffer less from factoid commonsense issues (Petroni et al., 2019). Second, events and their relations are key components of commonsense reasoning (McCarthy and Hayes, 1981), and our
4398
study shows overall commonsense and event commonsense are highly correlated (see §4). Third, event commonsense aligns well with the interactive nature of open-domain dialogue systems (Huang et al., 2020) to complete certain social goals.
To automatically evaluate event commonsense in open-domain dialogues, we propose ACCENT, a reference-free AutomatiC Event Commonsene EvaluatioN meTric which leverages commonsense knowledge bases (CSKBs) and measures the quality of generated responses without having groundtruth reference responses. For example, comparing the examples in Figure 1 against the CSKB easily reveals commonsense errors in the responses because when “PersonX likes to paint”, what he/she needs may be “to get a paint brush” instead of “to get an oxygen mask”, and when “PersonX loses arms from a car accident”, the other person is expected to feel “sad”.
While these judgments are intuitive to human, two challenges exist in automating the evaluation process. First, there is a considerable gap between free-form conversational data and the compact commonsense knowledge in the CSKB. Second, locating relevant knowledge in the CSKB is non-trivial.
ACCENT addresses these challenges through a pipeline method that uses an intermediate symbolic representation for commonsense reasoning. ACCENT first extracts event-relation tuples from the target response and its preceding dialogue history via a prompt-based generative model trained in a low-resource setting. Those extracted tuples bridge the gap between the free-form dialogue and the compact form of CSKB. Then, a compatibility score is computed to decide how well each extracted tuple aligns with the CSKB.
To train and evaluate ACCENT, we construct the first publicly available event commonsense evaluation dataset for open-domain dialogues (see §4). Besides collecting human commonsense judgments, we request annotators to manually extract event-relation tuples for further analysis.
Our main contributions are three-fold:
• We propose ACCENT, an event commonsense evaluation metric for open-domain dialogue systems. To the best of our knowledge, this is the first work that systematically studies event commonsense in dialogue systems.
• We construct the first publicly available event commonsense evaluation dataset for open-
domain dialogues.1
• Extensive experiments show that ACCENT achieves better correlation with human judgments for dialogue commonsense evaluation than several well-designed baselines, and enables easier interpretability of results.

heading: 2 Background: Event Commonsense
text: Endowing machines with human-like commonsense reasoning capabilities has been an ultimate goal of artificial intelligence research for decades (McCarthy and Hayes, 1981; LeCun, 2022). While many early works focused on factoid commonsense or the knowledge about concepts (Lenat, 1995; Liu and Singh, 2004), event commonsense emerges as an important aspect for machine commonsense measurement (Chen et al., 2021). Compared with concepts or entities, events are more informative, involving actions, participants, time etc. Besides, event commonsense also requires understanding various relations between events (Kuipers, 1984; Rashkin et al., 2018) which would facilitate complex reasoning, especially in interactive scenarios such as dialogues.
Among the current commonsense resources (related works in Appendix A), ATOMIC2020 (Hwang et al., 2021) is a comprehensive CSKB including physical-entity, event-centered, and socialinteraction knowledge. Its event-centered and social-interaction components take up 84.4% tuples of the entire knowledge base, providing knowledge regarding how events/human actions are associated with other events/actions. For example, given the event “X runs out of stream”, according to ATOMIC2020, this event may happen after “X exercises in the gym”, and the person X is likely to “feel tired”.

heading: 3 Method
text: We present ACCENT, as a framework for event commonsense evaluation. Figure 2 gives an overview of ACCENT with two major components.

heading: 3.1 Symbolic Intermediate Representation
text: ACCENT uses event-relation tuples as the symbolic intermediate representation. Each tuple contains a head event and a tail event which are connected through an event relation. We formally define events and relations below.
1We release ACCENT and our collected datasets at https://github.com/PlusLabNLP/ACCENT.
Event Following Pustejovsky et al. (2003), we define events as short phrases with a trigger word and its arguments (e.g., I like to paint). To better align with ATOMIC2020, we normalize the event by replacing tokens referring to people with Person variable (e.g., PersonX likes to paint). Relation We select R̃ ={xIntent, xWant, oWant, xReact, oReact, xNeed, xAttr, xEffect,
oEffect, HinderedBy, IsAfter, HasSubEvent}2 from ATOMIC2020 relations. These relations cover human behaviors, i.e., motivation, want, reaction, need, description, towards events (Sap et al., 2019b), the cause-effect and constraint in force dynamic (Talmy, 1988), the temporal information, as well as the parent-child relation in event hierarchy. Examples for each relation are in Appendix C.

heading: 3.2 Event-Relation Extraction
text: The input of the event commonsense evaluation task is a list of utterances {u0, u1, · · · , un−1} representing the dialogue history and the target response un. ACCENT first converts the free-form text into event-relation tuples. To retain the information in un, ACCENT extracts tuples whose head and tail events are both from the target response (denoted as “Single”). Besides, to capture event commonsense issues conditioned on the dialogue history (e.g., Figure 1 Example B), ACCENT also extracts tuples whose two events come from un and un−1 respectively (denoted as “Pair”).
As illustrated in Figure 3, the event-relation extractor in ACCENT is a T5 modelM (Raffel et al., 2020) guided to generate the head and tail events via designed prompts for each relation. ACCENT concatenates the prompt for r ∈ R̃ and the dialogue as the input and fine-tunesM in a low resource setting. When the relation r exists in the input utterances, the fine-tunedM is expected to generate the head and tail events following a particular format, i.e., “event1: {head}; event2: {tail}”, so that the tuple can be parsed from the decoded sequence (from Block A to Block B in Figure 3).
2“x” and “o” pertain to PersonX and other person(s).
Otherwise, the fine-tuned M is expected to output “None”. For each r ∈ R̃, the designed prompt explains the semantic meaning of r and triggers the model to generate the head and tail events (the prompts are included in Appendix C). At the inference time, we queryM with prompts for each r and parse the generated outputs to get h and t to construct tuples.

heading: 3.3 Compatibility Test
text: After extracting event-relation tuples, ACCENT checks whether these tuples are sensible through a compatibility test. Denoting the CSKB as C, the compatibility test aims to learn a scoring function f based on C, where f((h, r, t)|C) represents the compatibility of the target tuple (h, r, t) with the CSKB C. we propose to score (h, r, t) by querying a dynamic version of C with h and r. Figure 4 gives an example of the whole process.
Specifically, ACCENT uses COMET (Bosselut et al., 2019) as the dynamic CSKB. COMET adapts the pre-trained language model by fine-tuning it on C through a conditional generation task where “{head} {relation} [GEN]” is the source and a tail event is the target. To score (h, r, t), we query the model by requiring it to generate tgen given “{h} {r} [GEN]”. The beam search method is applied for decoding, so we obtain a set of generated tail events, {tigen}ki=1, where k is the beam size.
The compatibility score for (h, r, t) is then computed by checking the similarity between t and the most similar tgen among {tigen}ki=1:
f((h, r, t)|C)= max 1≤i≤k cos(embed(t), embed(tigen)) (1)
Here, embed(·) is parameterized by a SentenceBERT model (Reimers and Gurevych, 2019).
After getting the compatibility scores for each extracted tuple, we average them to get the final score for the target response (see Figure 2).

heading: 4 Data Collection
text: We construct the first event commonsense evaluation dataset for open-domain dialogues through crowdsourcing on Amazon Mechanical Turk (MTurk). In this section, we describe the collection procedure and the details of the dataset.

heading: 4.1 Dialogue Data Preparation
text: We select dialogue histories from DailyDialog (Li et al., 2017), PersonaChat (Zhang et al., 2018), and TopicalChat (Gopalakrishnan et al., 2019) humanhuman dialogues. The dialogue history is limited to at most 4 consecutive utterances. Since human utterances barely contradict event commonsense, to better evaluate machine generated dialogues, we collect responses using advanced dialogue systems, DialoGPT (Zhang et al., 2020d), PLATO-2 (Bao et al., 2021), DiSCoL (Ghazarian et al., 2021).
To ensure most samples contain events and are meaningful for event commonsense evaluation, we filter samples using the following criteria: (1) the response contains at least 5 words; (2) the response contains at least 1 non-interrogative sentence3; (3) the response is more than a courtesy (e.g., “It’s been nice chatting with you.”)4. After filtering, we randomly select 300 samples and split them into 200 for training and 100 for testing. We name this dataset DECO (Dialogue Event Commonsense Dataset).

heading: 4.2 Tuple Extraction
text: To train the event-relation extractor of ACCENT, we collect human extracted event-relation tuples from DECO training set. Annotators are shown with the target response, the dialogue history, a specific relation, and are requested to compose eventrelation tuples. They are allowed to tick “I cannot find any tuple” if no tuple can be found. We also request them to select whether the tuple belongs to “Single” or “Pair” (defined in §3.2) for each tuple they extract. Figure 8 in Appendix D shows our data collection panel. We launched HITs5 for relations in R̃ repeatedly until we obtained at least 20 tuples for each relation. In order to ensure the test set is comprehensive, we particularly request annotators to compose tuples for all 12 relations in R̃ (100 samples × 12 relations in total).
3We check this by finding sentences that are not ended with a question mark (“?”).
4These responses are manually filtered out. 5HIT is an assignment unit on Amazon MTurk.
A separate validation round was conducted to check whether each extracted tuple satisfies (1) the head and tail are events, (2) the head and tail come from un or un−1, (3) the relation between the head and tail can be inferred from the dialogue. A tuple is deemed valid if the majority of 3 annotators vote “yes”. After removing invalid tuples (the dialogue numbers remain unchanged), we collected 307 tuples for training and 467 tuples from the DECO test set. Figure 5 shows the relation distribution in the densely annotated test set. More details about DECO statistics are included in Appendix D.

heading: 4.3 Commonsense Scoring
text: We instruct annotators to score target responses in terms of event commonsense by focusing on the events and their relations (the guideline is shown in Figure 7). Each response was annotated by 3 individual annotators with a scale of 1 to 5. Following Mehri and Eskenazi (2020), we measure the inter annotator agreement (IAA) by correlating each annotation with the mean of the other annotations for the same sample, and the Spearman correlation is 0.578 showing an acceptable agreement.6 The final event commonsense score assigned to each sample is the average of 3 individual ratings.
We also requested the annotators to judge the overall commonsense of a dialogue response before introducing event commonsense to annotators. Among the 900 annotation pairs we collected, the Spearman correlation between the two scores reaches 0.862, which indicates that event commonsense is a key component in overall commonsense reasoning.

heading: 4.4 Additional Human-Machine Dialogues
text: We further explore the generalization ability of ACCENT on responses with human-machine dialogue histories. We select 100 samples from ConTurE (Ghazarian et al., 2022a), a turn-level evaluation dataset, to annotate event commonsense scores. We denote this dataset as ConTurE Subset. Its statistics are also included in Appendix D.
60.40-0.69 implies strong relationship.

heading: 5 Experiments

heading: 5.1 Setups
text: We compare ACCENT with baseline methods for event commonsense evaluation and also examine its two components separately. Therefore, our experiments include three setups for the evaluation: Setup 1 (Metrics Performance) Our main goal is to evaluate the commonsense metric, and we achieve this by computing the correlation between automatic scores and human judgments. ACCENT and baseline metrics are tested on DECO test set and ConTurE Subset. Setup 2 (Event-Relation Extraction) We evaluate the performance of the event-relation extraction component of ACCENT by comparing the automatically extracted tuples with human extracted tuples on DECO test set. We view checking whether a tuple with relation r is extracted from the utterances un and un−1 as a binary classification problem and compute the F1 score. We also measure how “close” the automatically extracted head and tail events are to human extraction results. We convert the tuple into a sentence by concatenating the head and tail, and then compute BLEU-2 (Papineni et al., 2002) and BERTScore (Zhang et al., 2020c). Setup 3 (Compatibility Test): The compatibility test component of ACCENT can be viewed as a tuple scoring task. We compare our proposed approach with other tuple scoring methods on a large-scale benchmark (Fang et al., 2021a) which contains event-relation tuples with 0 (compatible to a given CSKB) or 1 (not compatible to the CSKB) scores. Since the training relations in this benchmark differ from relations supported by the off-theshelf COMET, we train our own COMET on its training set (see Appendix E.2 for more details) to make our compatibility test component applicable to this test set. This benchmark dataset covers all 12 relations in R̃ as well as 6 more relations.

heading: 5.2 Baselines
text: We compare ACCENT with 5 baseline metrics: (1, 2) FED-understandable/appropriate (Mehri and Eskenazi, 2020) are two off-the-shelf baselines. “Understandable” and “Semantically Appropriate” are closer to commonsense compared to the rest of the criteria in FED. (3) Cross-encoder is a widely used model for sentence-pair regression tasks. We use BART (Lewis et al., 2020) as the backbone. (4) Cross-encoder (COMET) is a variant of (3) with COMET trained on ATOMIC2020 as the back-
bone. (5) MLP regressor (Zhou et al., 2021) is trained with neural features from DialoGPT and symbolic features from ConceptNet (details in §7). The cross-encoders and the MLP regressor require event commonsense scores to train the model in an end-to-end manner. We use the annotated scores in DECO training set to train them, and split 20% data for validation to conduct hyperparameter search.
For Setup 2, we consider the following baseline approaches: (1) ASER Extractor (Zhang et al., 2020b) first extracts events through patterns from dependency parsing and then uses a neural classifier to predict the relation. (2) CSKB Search (Zhou et al., 2021) searches the one-hop neighbors in ATOMIC2020 through keyword matching.
For Setup 3, we consider 4 tuple scoring baselines. These baselines convert a tuple to an embedding and train a binary classifier to give score: (1) BERT feeds h, r, t to BERT and concatenates their [CLS] embeddings to get the tuple embedding. (2)BERTSAGE (Fang et al., 2021b) further concatenates the average embedding of the neighbors of h and t in an event knowledge graph. (3) KGBERT (Yao et al., 2019) inputs “[CLS], h, [SEP], r, [SEP], t” to get the tuple embedding. (4) KGBERTSAGE (Fang et al., 2021a) further concatenates the average embedding of neighboring nodes. We use RoBERTaLARGE (Liu et al., 2020) as the backbone which has roughly the same parameter budget with COMET to have a fair comparison.
The details of the baseline implementations are in Appendix E.1.

heading: 5.3 ACCENT Implementation
text: The proposed ACCENT framework is implemented using the Transformers library (Wolf et al., 2020). For event-relation extraction, we fine-tune T5base7 for 50 epochs with the batch size of 4 and the learning rate of 5e-5. The training data comes from the human extracted tuples from DECO training set. We additionally select 5 negative samples (dialogues that do not have a certain relation) per relation from the training set and set their target output as “None” to guide the model to handle cases which do not contain a certain relation. During inference, if no tuple is extracted after considering all relations, we assign a score of 0.5 to the sample. For compatibility test, we use the off-the-shelf COMET model trained on
7huggingface.co/t5-base
ATOMIC2020 (Hwang et al., 2021) 8. When querying COMET through generation, we use beam search with a beam size of 10 to get commonly sensible tail events. The embed(·) in Equation (1) is parameterized by paraphrase-MiniLM-L6-v2 provided in the Sentence-Transformers library9.

heading: 6 Results and Analysis

heading: 6.1 Metrics Performance
text: Table 1 shows the correlations between automatic scores and human annotations. ACCENT uniformly outperforms the baselines on both two test sets. Specifically, off-the-shelf metrics (“FED-appropriate”, “FED-understandable”) perform poorly. For “Cross-encoder (COMET)”, its results show that implicitly using the CSKB through transfer learning cannot yield satisfactory performance. Besides, cross-encoders fail to generalize well to ConTurE Subset whose dialogue histories are from human-machine dialogues. For “MLP Regressor”, although it tries to utilize the CSKB explicitly, it is not as effective as ACCENT.
Some examples from the DECO test set and their event commonsense scores given by ACCENT are shown in Table 2. These scores are close to human judgements and enjoy great interpretability owning to the extracted event-relation tuples.
Apart from the sample-level correlation, we further examine whether ACCENT can reflect model performance in terms of event commonsense. Figure 6 shows the rankings of three dialogue systems used in DECO construction given by human and ACCENT. Human and ACCENT rank the three systems exactly the same and the two sets of averaged scores highly correlates with each other.
8github.com/allenai/comet-atomic-2020 9huggingface.co/sentence-transformers/
paraphrase-MiniLM-L6-v2

heading: 6.2 Tuple Extraction Performance
text: Table 3 shows the results of Setup 2 where we evaluate the event-relation extraction performance on DECO test set. Our proposed method achieves much higher BLEU and BERTScore than two baselines, indicating that the composed events in tuples have reasonable quality. However, joint eventrelation extraction remains challenging because it combines the event extraction and relation identification. Although our proposed method has higher score than ASER Extractor by F1, it still has plenty of room for improvement. As for CSKB Search, it usually returns a lot of tuples, thus resulting in high recall and very poor precision. Also, searching CSKB is not applicable in our framework because this method can only return sensible tuples.

heading: 6.3 Compatibility Test Performance
text: Table 4 depicts the test results on the benchmark dataset. Our method outperforms all baselines, and it does not require negative samples for training. The major difference between our method and those tuple scoring baselines is that we use the tuples in the existing CSKB to train a dynamic CSKB, i.e., COMET, instead of a discriminative model. We assume our strong results may be due to the generalization ability of the dynamic CSKB.

heading: 6.4 Ablation Studies
text: We conduct ablation studies to explore (1) whether the proposed event-relation extraction method can
lead to better final metric performance; (2) given the automatically extracted tuples, whether the proposed compatibility test method can lead to higher correlation with human judgment.
To answer (1), we compare different methods to get the event-relation tuples (Part I in Table 5). Among the event-relation extraction baselines, we only consider ASER Extractor because CSKB search is not applicable in our framework as discussed in §6.2. Note that the event-relation extractor in ACCENT considers tuples in both “Single” and “Pair” settings to cover two potential types of errors (see §3.2). To verify this, we compare the variations of our proposed method where we only use tuples marked as “Single” or “Pair” for model training. Also, the human extracted tuples in DECO test set are used to provide an upper bound.
To answer (2), we fix the event-relation extraction part and change the compatibility test part (Part II in Table 5). We consider BERT and KG-BERT trained on the CSKB compatibility benchmark because they do not need event graph information and can be seamlessly applied to our compatibility test. Also, while we query COMET through tail generation, another intuitive design is using the model loss with “{h} {r} [GEN]” as the source and t as the target to give scores. We map the loss to (0, 1) through an exponential function, and name this alternative as “COMET (neural)” for it skips the symbolic decoding of tgen.
Table 5 demonstrates that the whole ACCENT gives the best result. Considering the variations of our design, “w/o Pair” gives much lower results, indicating that limiting the symbolic intermediate representation to only the information contained in the target response is not enough. This observation is in accord with our finding that some event commonsense errors occur when we take the dialogue history into account.
Another empirical discovery is that although “COMET (neural)” is a direct way of using the dynamic CSKB, its performance is poorer than what we propose in ACCENT. We assume that comparing t and tgen in a symbolic fashion can yield more comparable scores among tuples with different relations (details in Appendix F).
In our implementation of ACCENT, the comparison of t and tgen is done by calculating the cosine similarity between their Sentence-BERT embeddings. We further experiment with other sentence embedding methods based on contrastive learning. Specifically, we consider DiffCSE (Chuang et al., 2022), ESimCSE (Wu et al., 2022) which are two unsupervised contrastive learning frameworks for learning sentence embeddings. We also consider Sup-SimCSE (Gao et al., 2021) which leverages annotated natural language inference datasets by using “entailment” pairs as positives and “contradiction” pairs as hard negatives in the contrastive learning objective. As shown in Table 6, ACCENT can benefit from the improvement of the sentence embedding method, i.e., using Sup-SimCSE (Gao et al., 2021). We support both Sentence-BERT and Sup-SimCSE in our released ACCENT codebase.

heading: 6.5 Error Analysis
text: Since ACCENT is a pipeline framework, there is likely error propagation. In section 6.4, we rule out the errors introduced by the event-relation extraction component by using human-extracted gold tuples. Results show that ACCENT with gold tuples (see “Gold Tuples” in Table 5) gives higher correlation with human judgment than “ACCENT (whole)” which uses the model-extracted tuples, indicating that ACCENT can benefit from high quality symbolic intermediate representation. We further include a qualitative analysis of the automatically extracted tuples in Appendix G, and believe improving the joint event-relation extraction is a worthwhile direction for future work.

heading: 7 Related Work
text: Automatic Open-Domain Dialogue Evaluation The evaluation of open-domain dialogue systems has long been a challenge due to the system’s open-
10https://huggingface.co/voidism/ diffcse-roberta-base-sts
11https://huggingface.co/ffgcc/ esimcse-roberta-base
12https://huggingface.co/princeton-nlp/ sup-simcse-roberta-base
ended goal (Huang et al., 2020), and simply scoring the overall quality is far from enough (Finch and Choi, 2020). Thus, researchers have decomposed the evaluation of open-domain dialogues into multiple facets and developed corresponding automatic evaluation metrics (Pang et al., 2020; Mehri and Eskenazi, 2020). While aspects like context coherence (Tao et al., 2018; Ghazarian et al., 2022b), diversity (Hashimoto et al., 2019), engagement (Ghazarian et al., 2020), have been systematically studied in the literature, the aspect of commonsense has long been neglected.
The closest related work is Zhou et al. (2021) which is mainly about commonsense-focused dialogues collection but also proposes an automatic metric for commonsense evaluation by training an MLP regressor on both symbolic and neural features. The symbolic features include the numbers of one-hop and two-hop triplets in ConceptNet (Speer et al., 2017) that can be found between the target response and its dialogue history. Although this metric utilizes the CSKB explicitly, it is limited to the direct search with surface form and only considers the number of triplets, and the CSKB used in the work is more about concepts but not event commonsense.
Joint Event-Relation Extraction While event extraction (Ahn, 2006) and relation identification (Do et al., 2011) are well-studied, how to jointly acquire them remains a challenge. We argue that joint event-relation extraction is an important problem because in practical use cases, the input is usually free-form text without pre-extracted events. Zhang et al. (2020b) is a pioneer work trying to jointly extract event and relation through a pipeline to automatically construct large knowledge graphs. Researchers in this work resort to rule-based methods for event extraction and train a classifier to predict the relation between a pair of events.
CSKB Compatibility CSKB population enlarges CSKB automatically by adding new links or nodes which are compatible with the commonsense knowledge to the existing CSKB. In Fang et al. (2021a,b), researchers try to add events from a large event knowledge graph to a CSKB. Compatibility test component of ACCENT is relevant to CSKB population task and it is defined in a more general setting where the head and tail of the tuple can be arbitrary events.

heading: 8 Conclusion
text: We present ACCENT, an automatic evaluation metric for event commonsense evaluation of opendomain dialogue systems. We show that by using event-relation tuples as the symbolic intermediate representations, ACCENT can effectively utilize the CSKB and achieve a decent correlation with human judgments for dialogue commonsense evaluation.

heading: 9 Limitations
text: In this work, we conduct research on event commonsense of open-domain dialogue systems for the first time. While achieving higher correlations with human judgments than existing baselines, ACCENT has some limitations:
First, the ACCENT framework is based on a fixed set of event relations and the commonsense knowledge in ATOMIC2020 which may fail to cover some potential event commonsense aspects. We believe augmenting the current framework with more commonsense resources is a worthwhile direction for the further improvement of ACCENT.
Second, the event-relation extractor in ACCENT framework is a T5 model fine-tuned in a low resource setting. Although the current model can yield fairly strong performance, it is an important research direction to improve the joint event-relation extraction component because the extracted tuples serve as the symbolic representation for commonsense reasoning in ACCENT framework. Since human extracted tuples are very costly to collect, we hope to explore whether we can improve this component through high-quality synthetic data construction or transfer learning in the future.

heading: 10 Acknowledgments
text: We thank the PlusLab members and the anonymous reviewers for their constructive feedback. This work is supported in part by the DARPA Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4032, and a Meta Sponsored research award.

heading: A Commonsense Knowledge Bases
text: To endow machines with commonsense reasoning abilities, a growing number of CSKBs are developed through human annotation and information extraction. From earlier CSKBs, ConceptNet (Liu and Singh, 2004; Speer et al., 2017) focuses more on taxonomic (e.g., “IsA”) and lexical (e.g., “Synonym”, “RelatedTo”) knowledge; TransOMCS (Zhang et al., 2020a) automates the knowledge base construction by leveraging the same limited set of relations defined in ConceptNet.
Recent CSKBs give more focus on event commonsense knowledge. In this work, we select ATOMIC2020 (Hwang et al., 2021) as the knowledge source of ACCENT framework because it is a comprehensive CSKB with rich knowledge regarding how events and human actions are associated with each other. For comparison, ATOMIC (Sap et al., 2019a), as the pioneer of ATOMIC2020, consists of only nine relations and therefore poses limitations. Another recent event-centric CSKB is GLUCOSE (Mostafazadeh et al., 2020), which however focuses on a specific part of event commonsense (mostly on causal inference) and is less comprehensive and suitable for our work.

heading: B Pseudo Code of ACCENT
text: In §3, we introduce the symbolic intermediate representation and the two components in ACCENT. Algorithm 1 displays the skeleton of the whole framework.
Line 3-9 in Algorithm 1 show the joint eventrelation extraction in ACCENT. We query the eventrelation extraction modelM with prompts for each relation. The head and tail events can be parsed from the generated result if it is not “None” and follows the pre-defined format, i.e., “event1: {head}; event2: {tail}”. Line 16-21 in Algorithm 1 show the compatibility test in ACCENT. Each tuple is given a score based on the maximum cosine similarity between its tail and the commonsense tails obtained from the dynamic CSKB C. After calculating scores for each extracted tuple, we average them to get the event commonsense score for the target utterance (Line 24 in Algorithm 1).

heading: C Event Relations
text: As introduced in §3.1, ACCENT selects relations from ATOMIC2020 which are related to event commonsense. These event relations can help cover
Algorithm 1: ACCENT framework. Input :Dialogue history h, target
utterance u, prompt dict P , extractorM, dynamic CSKB C, sentence embedder E
Output :Event commonsense score s 1 tuples← [ ] 2 // Event-relation extraction. 3 foreach (rel, prompt) in P do 4 raw_output← generate(M, h, u, p) 5 if check_format(raw_output) then 6 (head, tail)← parse(raw_output) 7 tuples.append((head, rel, tail)) 8 end 9 end
10 // Compatability test. 11 if is_empty(tuples) then 12 return 0.5 13 else 14 tuple_scores← [ ] 15 foreach (head, rel, tail) in tuples do 16 score← 0 17 cs_tails← query(C, head, rel) 18 foreach cs_tail in cs_tails do 19 x← cos(E(tail), E(cs_tail)) 20 score← max(score, x) 21 end 22 tuple_scores.append(score) 23 end 24 return average(tuple_scores) 25 end
different types of insensibility for event commonsense evaluation of open-domain dialogues. Table 7 shows examples in DECO where the system response violates event commonsense in terms of different types of event relations.
Note that although “Cause” and “xReason” in ATOMIC2020 are also related to event commonsense, we exclude them from the selected subset R̃. This is because the cause-effect relation can be covered by “xEffect”/“oEffect” and tuples with “Cause” and “xReason” relations take up less than 0.1% percent of ATOMIC2020. Moreover, we exclude “IsBefore” because a tuple with “IsBefore” relation can be equivalently converted to a tuple with “IsAfter” relation by switching the head and tail. As shown in Table 8, for each relation in R̃, a prompt is manually designed to explain its semantic meaning. These designed prompts give more hints to the pretrained model and allow a single model to extract tuples for different relations.

heading: D Additional Details of Data Collection
text: D.1 Quality Control
To ensure the annotators have a good understanding of event and event commonsense, we restrict the annotators from English-speaking countries, and those who have finished at least 5,000 HITs with an acceptance rate > 97%. The compensation rate for annotators is calculated using a per hour wage of $16. 13
For commonsense scoring (see §4.3), we requested 3 annotators to score each sample, and we instructed them to specifically consider events and their relations in the dialogue to give the event commonsense score. Figure 7 shows the annotation guideline we used for event commonsense scoring. We also set a sample for attention check in each HIT. HITs that failed the check were reassigned to other annotators.
For tuple extraction (see §4.2), we conducted a training round before the large scale annotation and 8 annotators proceeded to the final round. Each HIT in this task was assigned to 2 individual annotators. The template used to collect event-relation tuples is shown in Figure 8. When validating the extracted tuples, 3 annotators judged each tuple, and we achieved Fleiss’ Kappa (Fleiss, 1971) κ = 0.491 (moderate aggreement). Tuples marked
13We pay $1 per HIT for the scoring task and $3 per HIT for the tuple extraction task. An additional bonus is sent to annotators who successfully pass the training round.
as invalid by the majority vote are not included in the final dataset.
D.2 Dataset Statistics Table 9 gives the statistics of DECO and ConTurE Subset. Although machine generated responses in DECO are given by advanced open-domain dialogue systems, some event commonsense errors still exist. For ConTurE Subset, we use it to test the generalization ability of different metrics. Table 10 gives the numbers of human extracted eventrelation tuples. Note that the test set of DECO is thoroughly annotated (we consider every relation on each sample) to provide a test set for the joint event-relation extraction task. All the data we collected are in English.
D.3 License The development of DECO and ConTurE Subset is based on the dialogues coming from DailyDialog, PersonaChat, TopicalChat, and ConTurE. PersonaChat14, TopicalChat15 and ConTurE16 are licensed. We ensure that we did not violate any license conditions when developing our datasets.

heading: E Additional Details of Experiment
text: This section describes more details about baseline implementation, applying ACCENT to CSKB benchmark, and computational resources. The implementation details of the proposed ACCENT framework are discussed in §5.3
E.1 Baseline Implementation We compare ACCENT with 5 baseline metrics on event commonsense evaluation. All the metrics are tested on DECO test set and ConTurE Subset. For metrics which require training, the training set of DECO is used, and we split 20% data for validation. The implementation details are as follows:
• FED-understandable/appropriate: We use their released model17.
• Cross-encoder: We use the cross-encoder with a regression head implemented in the Sentence-Tranformers library (Reimers and
14https://github.com/facebookresearch/ ParlAI/blob/main/LICENSE
15https://github.com/alexa/ Topical-Chat/blob/master/DATALICENSE
16https://github.com/alexa/conture/ blob/main/DATALICENSE
17https://github.com/Shikib/fed
Gurevych, 2019). We use BART as the backbone model to align with the COMET model used in our method. For hyperparameter search, we fine-tune the cross-encoder for 10 epochs with the batch size of 8 and the learning rate from {1e-5, 3e-5, 5e-5}.
• Cross-encoder (COMET): We use the off-the-shelf COMET model trained on ATOMIC2020
18 as the backbone model. Other implementation details are the same with Cross-encoder.
• MLP regressor: Since the code of Zhou et al. (2021) is not publicly available, we produce the results using our own implementation based on scikit-learn19. Our implementation is available in our released codebase.
For event-relation extraction baselines, the im18github.com/allenai/comet-atomic-2020 19https://scikit-learn.org/stable/ modules/generated/sklearn.neural_network. MLPRegressor.html
plementation details are as follows:
• ASER Extractor: We use their provided code20 to extract events. The neural classifier for relation prediction is trained on the human annotated tuples in DECO training set.
• CSKB Search: We search tuples related to the target response and its previous response in ATOMIC2020 following the CSKB search pipeline described in Zhou et al. (2021). A potential concept set is built from the utterances by identifying nouns, verbs, and adjectives that are not stopwords through part-of-speech (POS) tagging and lemmatizing them. We return tuples whose head and tail both contain words in the concept set as the search result.
For CSKB population baselines, we use the implementation in Fang et al. (2021a)21. For the backbone model, we use RoBERTaLARGE which has roughly the same parameter budget with COMET. We train all models for 1 epoch with the batch size of 64. The learning rate is searched from {1e-7, 1e-5, 1e-3} on the validation set.
E.2 Applying ACCENT to CSKB Benchmark
In §5 Setup 3, we apply the compatibility test approach in ACCENT to a CSKB benchmark. Such an application is seamless because the compatibility test also assigns a score to each tuple, and tuples which receive higher compatibility scores are naturally more suitable to populate the CSKB. We train the COMET model on the positive samples in the training set of the benchmark dataset for 1 epoch with the batch size of 64. The learning rate is searched from {1e-7, 1e-5, 1e-3} on the validation set. Note that our approach does not require any negative sample in the training stage. It also does not need the event graph information provided in the benchmark dataset, but results in Table 4 shows
20https://github.com/HKUST-KnowComp/ ASER
21https://github.com/HKUST-KnowComp/ CSKB-Population
that our method outperforms baselines which require manually created negative data and take in graph information.
E.3 Computational Resources
We run BERTSAGE and KG-BERTSAGE for the CSKB benchmark experiments on a single Nvidia V100 GPU with 32 GB memory where these models require large memory consumption in the run time. The rest of experiments is done on a single Nvidia A10 GPU with 24 GB memory.
Note that although we develop the ACCENT framework based on large language models, the only part which requires training is the T5 model (with 223M parameters) for event-relation extraction. As discussed in §3.2, the model is fine-tuned in a low resource setting and the training takes less than 0.5 GPU hour.

heading: F More Discussion of the Compatibility Test Approach in ACCENT
text: ACCENT checks whether an event-relation tuple (h, r, t) is compatible with the commonsense knowledge by comparing the similarity between t and commonsense tails generated by the Dynamic CSKB (COMET). Ablation results in Table 5 show that the compatibility test approach in ACCENT yields better performance than the “COMET (neural)” alternative which also uses the COMET model. To exclude the potential noise introduced by the automatically extracted tuples, we further compare these two methods using humanextracted tuples on DECO test set. Results in Table 11 demonstrate that the conclusion still holds under this experimental setting. Table 12 gives
two samples with a breakdown of tuple results. Compared with the compatibility scores given by ACCENT approach, the scores given by “COMET (neural)” are less comparable among tuples with different relations, thus making this method unsuitable for ACCENT framework.

heading: G Error Analysis
text: We conduct a qualitative analysis of the eventrelation component in ACCENT. Table 13 shows some examples of the extracted tuples. While most of the head and tail events are nicely composed and capture the major information in the given text, they are not perfect. Multiple participants involved in the dialogue further increase the difficulty of the task. We note that the model sometimes confuses the multiple participants in the dialogue and makes mistakes when using “PersonX” or “PersonY”. For example, in the third sample of Table 13, the model confuses different participants since the subjects of “raise money” and “go tomorrow for treatment” should be different. Such confusion will lead to the wrong tuples which cannot truly reflect the meaning of the dialogue. Also, identifying relation from the given dialogue is challenging. Although we include negative samples (dialogues that do not have a certain relation) when fine-tuning T5, errors still exist (e.g., the tuple with “xAttr” relation in the third sample of Table 13).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Sections 6.2, 6.4 and Appendix H include the error analysis of the ACCENT framework. A separate section of “Limitations” is also included in Appendix A.
7 A2. Did you discuss any potential risks of your work? We do not include potential risks of our work, since we believe that none of the components in our model and the dataset by itself can produce offensive results. The responses generated and augmented to DECO dataset are coming from previously proposed state-of-the-art models which are trained on datasets without profanity and inappropriate utterances. Other than that, our work is pertinent to evaluation and has less feasibility of potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction summarize the paper’s main claims. Three experimental setups and results show the ACCENT ’s superiority versus baselines.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In sections 5.1 and 6.3, we cite the CSKB population benchmark. Section 4.1 has the citations for each of the dialogue models used for the DECO response generation.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Some dialogue sources we used come with licence. We discuss them in Appendix D.3.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 5.1 and 6.3 use existing CSKB population benchmark that are compatible with the conditions that data was granted to be used. Our collected data is anonymized and section 4 describes the data use cases and statements.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 4 is mainly about data collection and creation, it discusses the data included in the collection process which are human judgments and it doesn’t reveal the identity of users participated in the data collection process. The inputs for evaluation are coming from existing human-written conversational data and responses generated by dialogue models do not include inappropriate content.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix D includes the language of collected dataset alongside the information about the data and its details.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 and Section D.2 in the Appendix are about the size of datasets.
C 7 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E.3 reports the computational resources.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes we discussed them in Appendix E.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 5 and 6 include the descriptive statistics about your results. Table 1 also demonstrates the results for ACCENT that all are significant (p < 0.05).
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 4 and Appendix D contain all the details about human annotations and conducted experiments.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? The crowdsourcing platform is discussed in Section 4, and we discuss how we pay the annotators in Appendix D.1.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Yes, it was indicated in the AMT HIT pages.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. We did not include ethics review board for out data collection process.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix D.1 reports the basic demographic and geographic characteristics of the annotator population.

heading: Future Work
text: Since human extracted tuples are very costly to collect, we hope to explore whether we can improve this component through high-quality synthetic data construction or transfer learning in the future