SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Authors: Hyunwoo Kim
Jack Hessel
Liwei Jiang
Peter West
Ximing Lu
Youngjae Yu
Pei Zhou
Ronan Le Bras
Malihe Alikhani
Gunhee Kim
Maarten Sap
Yejin Choi
Venue: EMNLP 2023 Main (2023)
Keywords: social commonsense
distillation
dialogue
dataset
TL;DR: To alleviate data scarcity in the field of open-domain social dialogue, we present SODA: the first publicly available, million-scale high-quality social dialogue dataset.

Abstract: Data scarcity has been a long standing issue in the field of open-domain social dialogue. To quench this thirst, we present SODA: the first publicly available, million-scale high-quality social dialogue dataset. By contextualizing social commonsense knowledge from a knowledge graph, we are able to distill an exceptionally broad spectrum of social interactions from a large language model. Human evaluation shows that conversations in SODA are more consistent, specific, and (surprisingly) natural than those in prior human-authored datasets. 

Using SODA, we train COSMO: a generalizable conversation model that is significantly more natural and consistent on unseen datasets than best-performing conversation models (e.g., GODEL, BlenderBot-1, Koala, Vicuna). Experiments reveal COSMO is sometimes even preferred to the original human-written gold responses. Additionally, our results shed light on the distinction between knowledge-enriched conversations and natural social chitchats. We plan to make our data, model, and code public.

SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Hyunwoo Kim
Jack Hessel ?
Liwei Jiang o
Peter West Ximing Lu
Youngjae Yu
Pei Zhou
Ronan Le Bras
Malihe Alikhani!
Gunhee Kim
Maarten Sap?
Yejin Choi 00
V Allen Institute for Artificial Intelligence
Scoul National University
+ University of Washington
University of Southern California
University of Pittshurgh
$ Carnegie Mellon University
Abstract
CO: Distillation Framework
Data scarcity has been long standing issue in the field of open-domain social dialogue. To quench this thirst, present SODA: the first publicly available, million-scale high-quality social dialogue dataset. By contextualizing so- cial commonsense knowledge from knowl- edge graph, we are able to distill am exception- ally broad spectrum of social interactions from a large language model. Human evaluation shows that conversations in SODA are more consistent, specific, and (surprisingly) natural than those in prior human-auu human-authored datasets.
Social Commonsense Knowledge
Head. PorsonX stap goal xNead Tall: take the first stop
Social Narrative
Madeleine
step towards encouraging
goal,
step
Social Dialogue
Madeleine Hey coach, wanted to talk about perform today.
Context with LLM
Coach: Wall Madeleine, you're You've working together.
Using SODA, train S COSMO: general- izable conversation model that is significantly more natural and consistent on unseen datasets than best-performing conversation models (e.g.. GODEL, BlenderBot-I, Koala, Vicuna). Ex- periments reveal COSMO is sometimes even preferred to the original human-wrind written gold responses. Additionally, our results shed light the distinction between knowledge-ente ledge-enriccee conversations and natural social chitchats. We make our data, models, and code public.!
Cosmo Conversation Model with Greater Generalizability
Soda
A Dataset of High-quality 1.5M Social Dialogues
Figure 1: An illustration of our CO3 framework ($2), SODA dataset ($3), and conversation model CosMo ($4) trained on SODA. Conversations are distilled from large language model (LLM) by contextualizing social commonsense. The full example is Table
1 Introduction
To alleviate this bottleneck, we introduce SoDA (SOcial DiAlogues), a million-scale English dialogue dataset covering wide variety of social interactions. As a result of being grounded on rich social commonsense and narratives, SODA goes beyond specific skill-focused dialogues and fea- tures general conversations. Our dataset in- cludes 1.5 million dialogues distilled from large language model (in our case, GPT-3,5; Ouyang et al., 2022) resulting in more than 11 million utter- ances with 300 million tokens: SODA the largest publicly available open- open-domain social conversation dataset. Human evaluation shows that SODA sur- passes existing Human-authored dialogue corpora across axes like consistency, specificity, and (sur- prisingly, even) naturalness ($3.2).
Conversations that occur in everyday spoken situa- tions are often not recorded as data. And when they are, such as in the case of text messages, research use is rightly restricted due to privacy and legal con- cerns. As result, collecting high-quality everyday social conversations on a large scale has long been recognized as difficult task (Smith et al., 2020). Previous studies have relied on crowdsourcing fo- cused on specific themes dialogue (e.g., persona, empathy; Zhang et al., 2018; Rashkin et al., 2019). However, this approach is limited in scale due to its associated costs. As a result, the progress made in machine dialogues, including generation, evalua- tion, and understanding, has been severely hindered by the reliance these small datasets (Kann et al., 2022; Mehri al., 2022).
To make SODA, we propose CO3, a frame- work for COntextualizing COmmonsense for dis- tilling COnversations from a large language model
'https: /tywww.wwwww/// www.:kki//odwer...eeee

---

(LLM). Illustrated in Figure 1, CO3 infuses com- monsense knowledge into dialogues by transform- ing knowledge triples into narratives, and then into dialogues. Such an approach offers two signifi- cant advantages: (1) maximizing diversity and (2) minimizing nonsensical conversations. Although generating content using LLMs is relatively easy, determining how to cover diverse content poses non-trivial challenge. We find that sampling from an LLM without contexts results in dull conver- sations ($3.3). Because commonsense knowledge graphs cover a wide range of everyday situations (West et al., 2022), conditioning on them results in broad spectrum of conversations. Moreover, since LLMs are prone to hallucinations (Weidinger et al., 2021), the seed commonsense knowledge can help them stay sensible generation path.
2
CO3: A Contextualization
Framework for Conversation
Distillation using Commonsense
We propose CO3, framework for distilling conversations from large language models (LLMs) by contextualizing (i.e., adding more context infor- mation) commonsense knowledge. Our goal is to obtain natural conversations covering a wide vari- cty of social intcractions. CO3 consists of three steps: (1) Retrieving social commonsense from a symbolic commonsense knowledge graph ($2.2), (2) converting it into sentence form and generating narrative from the sentence ($2.3), and (3) infer- ring the conversation participants from the narrative and derive a conversation grounded in the narrative ($2.4). We use GPT-3.5 (i.e., text-davinei-Oe -00=;; Ouyang et al., 2022) to implement COs, though in practice, different model could be used. We use CO3 to create SODA: an example is in Table 1. More details can be found in Appendix A.
With SODA, we train a COnverSation MOdel, COSMO. Human evaluation results demon- strate that: (1) COSMO generalizes better to un- seen conversations than existing best-performing dialogue models, winning by more than 40% on av- erage in head-to-head comparisons versus Blender- Bot (Roller et al., 2021), Koala (Geng et al., 2023), and Vicuna (Chiang et al., 2023) ($5.1); (2) CоMMо outperforms BlenderBot (with the same number of parameters) on the datasel Blender- Bot was trained on, despite never seeing the cor- pus ($5.2); and (3) COSMO responses are even preferred over human- human--uthoren authored, ground-truth re- sponses in DailyDialog (Li et al., 2017), dataset on which COSMO was not trained on ($5.1).
2.1 Inspiration Behind CO3
What ai the heart of conversation? At its core, a conversation is a fundamental form of social in- teraction (Myllyniemi, 1986). These experiences are abstracted into narratives or scripts (Mar and Oatley, 2008; Rumelhart, 1975; Schank and Abel- son, 1975). Eventually, social experiences form our knowledge for explaining everyday events and inferring the mental states of others (Heider, 1958). This inference is coined attribution in social psy- chology (Baumeister and Bushman, 2017), and has been studied in NLP as social commonsense (Rashkin et al., 2018; Sap et al., 2019). Inspired by cognitive science, we reverse the abstraction pro- cess, starting from social commonsense knowledge in symbolic forms, and unfold rich narratives and conversations that could have initially encapsulated those commonsense knowledge.
Finally, the distilled dialogues in SODA repre- sent a significant resource contribution for open- domain dialogue research. Most of all, SODA en- ables the research community to train smaller di- alogue agents with competitive capabilities. Also, SODA can help enhance the generalizability of other advancements in the dialogue field (e.g., un- derstanding and evaluation), which have relied on existing small datasets. Lastly, SODA highlights a dimension where recent LLM-based conversational agents (e.g., Koala, Vicuna, and ChatGPT) struggle i.e., the naturalness of the responses 1$5.1 and $5.3). As these models are designed to provide knowledge-basee responses, they may generate re- sponses that are informative but lack the natural- ness found in social chitchat. We plan to publicly release SODA, CoSMO, and CO3 under the permis- sive license CC-BY-4.0, aiming to address the data scarcity issue in open-domain dialogue.
2.2 Commonsense Knowledge Graph
Concretely, we start with a commonsense knowl- edge graph, which captures various relations of everyday events and inferences on others' mental states in symbolic forms (Sap et al., 2019; Hwang et al., 2021). The knowledge graph is represented by symbolic triples describing two events, denoted as the head and tail, and the relation between those two events, c.g., Head: PersonX moves a step
Pht //btta researchers/mmdels referreed to-tttt--sssss-s- as-------5
htmm/docs/mmdeelindeer

---

closer the goal, Relation: xNeed, Tail: to take the first step. We use Atomic 10x (West
Symbolic Social Commonsense Knowledge:
Head: PersonX moves a step closer the goal
et al., 2022) as our knowledge graph: includes diverse social (e.g., intention, desire, reaction) and event-centered (e.g., order of events) commonsense. Since we are interested in distilling social inter- actions, we only retrieve triples related to social (rather than, e.g., physical) commonsense."
Relation: xNeed
Tail: to take the first step
Sentence-form Commonsense: Madeleine took the first step. Madeleine moves a step closer the goal.
Narrative:
Madeleine took the first step towards her goal, and with her coach's 's encouraging words, she moves one step closer.
Speakers: Madeleine, Coach
2.3 Commonsense Knowledge + Narrative
Conversation:
Madeleine: Hey coach, wanted to talk to you ahout my performance today. I was really pushing myself and I think I did pretty well. But I' m still not quite where want to be. Coach: Well Madeleine, you're progressing nicely. You'l ve come long if you want to reach your full potential, there's 's still some work to be done.
Triple Form to Sentence Form Since common- sense knowledge graphs are represented in sym- bolic form (i.c., triples), we first convert them into simple sentences with templates for each relation. For example, the commonsense knowledge in Ta- ble 1 is converted to "Madeleine took the first step. Madeleine moves a step closer to the goal." To make the sentences sound more natural, we replace the person variables (c.g., PersonX, PersonY) with T common names of US SSN upplicants rang- ing from 1990 to 2021.4
since we first started working logether. But
Madeleine: I know that. And I'm willing to put in the work. It's just that sometimes feel like I'm not making much progress as should be. Maybe I'm not training hard enough? Or mayhe my technique is off?
Coach: It could be number of things, Madeleine. But don't worry, we'll '1l figure out together. Let's just keep working hard and how things go.
Madeleine: Alright, coach. Thanks for the talk. Coach:
No problem. See you practice tomorrow.
Sentence Form to Narrative Next, we prompt GPT-3.5 to instantiate the sentence-form into two- or three-sentence short narrative, e.g.. for the com- monsense example above Madeleine took the first step towards her goal, and with her coach's 's encour- aging words, she moves one step closer. Prior work has shown that LLMs can effectively gener- ate stories with plausible details that go beyond the contents of the prompt (Radford et al., 2019).
Table I: sample from SODA. More examples can be found in Appendix B.
SODA:
A Million-scale Social Dialogue Dataset
We obtain SODA (SOcial DiAlogues), a large-scale conversation dataset covering a wide range of social interactions, by applying series of post-processing ($3.1) to the conversations gen- erated from our contextualization framework ($2), We compare SODA with existing human-curated di- alogue corpora ($3.2) and analyze the effectiveness of contextualization ($3.3). Table shows sample from our dataset. More details are in Appendix B.
2.4 Narrative - Conversation
Inferring Conversation Participants Inferring the conversation participants from the narrative is straightforward in cases where triples contain two person variables (i.e., PersonX and PersonY). But for triples that include only one person (e.g., the example Table I), we query GPT-3.5 to predict the other interlocutor (e.g., mom, coworker).
3.1 Post-processing the Conversations
Basic Filtering Starting with an initial set of 2.2 million conversations sampled from GPT-3.5, we: (1) use lexical pattern matching to filter out con- versations with erroncous patterns - c.g., repetition and omission of speaker prefixes (6.3%); (2) re- move conversations that have less than four turns or more than twenty turns (5.7%); (3) remove con- versations with more than two speakers (1 (11.3%); and (4) remove conversations where least of the speakers was identified as non-human (c.g.. broomstick, imaginary friend, dog; 5.6%).
Generating Conversation grounded in Narrative With the narrative and speakers input, prompt GPT-3.5 to generate a full, multi-turn conversation between the speakers the context of the narrative. We append the first speaker as an utterance prefix to the prompt. Indicating the speakers with prefixes helps GPT-3.5 generate fluent conversations that alternate between the two.
*We leave relations for physical and event-centered com- monsense to potential future work.
"catalog. data. data.gov/ddtaastteeeooooos htw/dataset/bbabr names- from- -iiuullsecritty wwwriitt---ddddreee sarddapplicationss ptpppliiaaaioonnatiinnnlllll national-das
AAlhhoggh our pipeline naturally generates multi- parti-partt con- versations as well, we focus on dyadic dialogues in this work.

---

300
SODA
DailDDalog
SODA
BlerdedSallTaakk
250
200
150
100
Natral Flow
Contest Deperdance
Topic Consistanyy
Speaker Cansistency
Specificity
Overall
Natumil Flow
Contert Dependence
Topic Canshitacy
Speaker Consintnnyy
Specificity
Ovenll
Figure 2: Results of head-to-head comparison between dialogues from SODA, DiilyDialog (Li al., 2017), and BlendedSkillTalk (Smith al., 2020) via human judgments ($3.2). The y-axis represents the number of samples preferred by human judges. The differences in all of the categories except for the Context Dependence comparing SODA and BlendedSkillTalk are statistically significant (|2) 3.3, p < 0.05).
Safety Filtering In order avoid conversations with dangerous and harmful contents, we apply two safety filters: Canary (Kim et al., 2022a) and Rewire AP1. 6 Canary is a narrative dialogue safety model that can classify whether the given context needs caution or intervention. We discard all con- versations marked as needing intervention (usually critical situations, e.g., crimes, emergencies; 4.3%); Rewire API is web-based API for detecting toxic content. We discard all conversations that are above the threshold of 0.5 for any of the 'violence', 'hate", and "sexually explicit criteria (~1%).
on the human-annotated subset, with a precision of 97 for answering "yes". We find 95% of the filtered conversations are identified by GPT-3.5 as containing the head event. Pairs that lack the head event are removed to ensure relevance between the nurrative-conversttin pairs and commonsense triples. More details are in Appendix B.1.
Final Dataset After all filtering, 68.9% of the initial conversations remain, which form the 1,486,896 conversations SODA.
Name Bias Mitigation We aim to minimize bi- ases associated with specific names while increas- ing inclusion and diversity. Both language mod- els and curated datasets often exhibit demographic imbalances (Dinan al., 2020; Weidinger et al., 2021; Sheng et al., 2021). Inspired by Smith and Williams (2021), we randomly replace names in conversations with Top-10K names US SSN applicants from 1990 to 2021.7 This covers 95% % of all applicants' names from the chosen time range window, including various names from diverse gen- der and ethnic backgrounds.
Commonsense Filtering We conduct small- scale human evaluation via Amazon Mechani- cal Turk with 100 randomly sampled narrative- conversation pairs (3 annotators per instance) to check whether or not the seed commonsense triple is meaningfully instantiated by the narrative and conversation. According to majority vote, 88% of the instances include the seed commonsense knowl- edge. Given that the majority of human-annotated samples include the seed commonsense, we focus filtering excluding pairs that lack the head event, as they are irrelevant the given seed commonsense.
3.2 Comparing SODA with Human-authored Dialogues
To apply this filter to entries of the corpus, we use GPT-3.5 as a zero-shot classifier. As GPT-3.5 demonstrated great performance in question an- swering (Ouyang et al., 2022), we validate the gen- erated parrative-covversatio pairs by asking the language model itself judge whether not the head of the commonsense triple is implied. We for- mulate this as three-way way multiple choice questions yes, yes, no, and unknown) and rank the answers according to their perplexity scores from GPT-3.5. This zero-shot classifier achieves high performance
High Quality To assess relative quality of the corpus, we conduct head-to-head human evalu- ations Amazon Mechanical Turk, comparing SODA with two widely used open-domain dialogue datasets: DailyDialog (Li al., 2017) and Blended- SkillTalk (Smith al., 2020). We random sample 300 dialogues from each dataset and evaluate them according to six criteria (Mehri et al., 2022): (1)
"e use Top- 1K names when contextualizing the common- sense triples $2.3. "Gender-neutral and nonbinary names are also included.
Chttps://rewire online/

---

Avg. #Tums
Avg. Utt.
Lexical Diversity
Common keywords across all relations
#Dialog
Length
friendship. help, support, communication, family, car, happiness, school. success, work
DailyDialog
13K
7.9
14.6
63.0
PersonaChal
11K
14.8
14.2
43.6
Common keywords for each relation (excluding the ahove)
WizardOrWikipedia
22K
9.1
16.4
60.3
EmpatheticDialoque
25K
4.3
13.7
64.2
xAttr
kindness, unger, intelligent, responsibility, friend, trust, conversation, food, generosity, smart
BlendedSkillTalk
7K
11.2
13.6
64.2
(18%)
ProsocialDialog
58K
5.7
20.0
60.2
xEffect
gratitde, anger, upset, hard work, happy, money. friend, boss, party, kindness
(17%)
SODA
1.5M
7.6
16.1
68.0
xintent
independence, hard work, determination, money, (23%) relaxution, anger, kindness, store, understanding
Table 2: Statistics of SODA compared to other large- scale dialogue datasets. Utt. denotes utterance. Lexical diversity is measured with MTLD (McCarthy and Jarvis, 2010). Description for each dataset Appendix F.
xNeed (7%)
job, money, confidence, comfort, advice. interest, conversation, listening. store, park
xReact (25%)
frustration, anger, confidence, happy. pride, relief,
disappointment, relaxation, anxiety, satisfaction
Want (11%)
conversation, store, determination, apology, leaming, doctor, job, friend, improvement, marriage
natural flow, (2) context dependence, (3) topic con- sistency, (4) speaker consistency, (5) specificity, and (6) overall. Judges asked to select better dialogue between the two, regarding each crite- rion. For context dependence, we ask the judges to choose which conversation includes responses that are more dependent on previous turns. Further details are in Appendix B.2.
Table 3: Common topic keywords of the narratives (i.e., conversation context) in SODA. Numbers in parentheses denote the ratio of the relations in SODA.
We find broad spectrum of topics encountered in social interactions are included in SODA.
Despite being fully machine-generatee human raters judge SODA as better in quality compared to both DailyDialog and BlendedSkillTalk across all axes by a large margin, except for the context dependence comparing with BleldedSkillTalk (see Figure 2). In particular, evaluators rate the flow of SODA to be significantly more natural than other human-authored artificial conversation datasets." 9
As a result, conversations in SODA contain di- verse lexicons. We compute MTLD (McCarthy and Jarvis, 2010) to measure the lexical diversity of con- versations. Table 2 reports the averaged diversity of dialogues for each training set. As PersonaChat (Zhang et al., 2018) contains conversations based few persona-related sentences, it shows the lowest lexical diversity. SODA, on the other hand, includes conversations from variety of social situ- ations, which leads to a wider range of words.
Large Scale With 1.5 million conversations, SODA is the largest in scale compared to exist- ing crowdsourced open-domain dialogue datasets and the machine-human generated ProsocialDialog dataset (Table 2). It contains more than 11 million utterances and each conversation is grounded in a short narrative describing the context. In total, SODA consists of 300 million tokens, making it a rich source for training conversation models.
Rich Emotion-related Information Since com- monsense knowledge from Atomic 10x includes emotional reactions of people events (i.e., the xReact triples), conversations with rich emotional contents are also included in SODA. total, SODA includes 385K conversations generated from 1.7K unique emotion descriptions of the xReact triples' Tail (e.g., happy, ashamed, motivated, irritated). "I Therefore, it contains significantly more descriptive emotion labels (i.e., the Tail) than other datasets which have fixed number of classes (Li et al., 2017; Rashkin et al., 2019). Furthermore, because we construct conversations in a bottom-up fashion from those emotion reaction in the commonsense triples, we know which speaker in the conversation is experiencing the emotion (i.e., PersonX) and what caused the emotion (i.e., the Head event).
Diverse Content SODA is built on top of 1.5 mil- lion commonsense knowledge triples of 10x which have been identified as being softly unique (West et al., 2022). Each seed triple is converted to a social narrative that serves as the distinct topic for each conversation. The Top- 10 common key- words from these narratives are listed in Table 3. 10
"A power analysis suggests that with our setup, we can de- tect effect sizes as small as 0.17 with a power and significance level of 95% (Faul et al., 2014).
"We note that conversations from other relations also natu- rally include emotional utterances.
We prompt ChatGPT to output keywords of the narrative.

---

DailyDaaog
BlendedSkillTalk
SODA
100
SODA Conversations Sampled without Context
Emmaioon
Ratio
Famotion
Ratio
Einotion
Ratio
80
admiration
20.42
curiosity
17.86
curiosity
12.92
gratitude
18.84
admiration
13.16
admiration
11.23
40
curiosity
12.85
sadness
8,50
approval
10.24
20
approval
10.91
joy
5.32
gratitude
7.39
joy
4.74
excitement
4.42
Juy
6.38
Natural Flow
Content Dependane
Topic Comiisency
Speaker Commistncy
Speeifeity
Intresdignnes
Overall
excitement
3.61
surpeise
4.34
disappointed
5.41
surprise
3.25
disappointed
4.34
confusion
4.68
love
3.06
fear
4.31
surprise
optimism
2.94
approval
4.19
realization
Figure 3: Results of head-to-head comparison human evaluation between conversations from SODA and those sampled from GP'--3.5 without context ($3.3). The y- axis indicates the number of samples that human judges preferred. The differences are all statistically significant with |=| > 2.6, p < 0.05 except for the Natural Flow class with 2 = 1.1 and > 0.05.
caring
2.23
optimism
3.95
caring
Table 4: The ratio (%) of Top-10 emotions in 10K utter- ances from DailyDialog, BlendedSkillTalk, and SODA, labeled by the GoEmotions' 27-emotion-type classifier (Demszky et al., 2020). Full table is in Appendix B.2.
We also find the distribution of emotions to be less skewed towards specific emotions. To compare the emotional composition, we use the 27-emotion- type classifier from GoEmotions (Demszky et al.. 2020) for labeling and compare 10K utterances from DailyDialog, BlendedSkillTalk, and SODA. The distribution of emotions for each dataset is pre- sented in Table 4. SODA exhibits more balanced distribution of emotions while maintaining similar rankings with other numan-authone dialogues.
(MTLD; McCarthy and Jarvis, 2010): 68.0 vs 63.1.
4
COSMO:
A Socially Situated Conversation Model
We use SODA to train COSMO: a COnverSation MOdel that can converse in a wide range of social situations. COSMO can take in situation narrative, along with dialogue history, and generate next utterance according to given role.
Cost & Time-Efficient Compared to dialogue crowdsourcing, collecting SODA via contex- tualization framework is significantly more time and cost efficient. With GPT-3.5 (ext-davinei-002, to go from a commonsense triple to a dialogue costs about $0.02, and 10 queries take less than minutes, counting our full filtration pipeline.
Training COSMO We use several structured components of SODA during training: (1) the contextual narrative n ($2.3), (2) the perspec- tive/speaker instruction i (c.g., "Imagine you are Madeleine and speak to her coach") built with the inferred conversation participants ($2.4), and (3) the dialogue context c. The model trained generate a target response r when given n, i, and c - i.e., p(r|n, i, c). We do so in a sequence- to- sequence fashion, concatenating n, i, e with a sep- arator <SEP> to serve as input. cis made up of the previous conversation utterances concatenated with turn indicator <TURN>
3.3 Do We Need Contextualization?
To isolate the effect of contextualization (vs. straightforward sampling from large language model), we compare SODA with dialogues naively sampled from GPT-3.5 without any given context. We sample 100 dialogues using the same hyperpa- rameters and the basic filtering steps in CO3, but with the following prompt: "The following is a long in-depth conversation between two people. AnPerson 1:." We ask human judges to evaluate the conversations in a -o-add com- parison as before ($3.2), with the additional crite- rion of interestingness (See et al., 2019).
Because conversational models often agree toxic or unethical behavior (Baheti et al., 2021), for additional training data, we include Prosocial- Dialog (Kim et al., 2022a) (adapted to the same format as SODA, see Appendix C). ProsocialDia- log includes a wide range of negative constructive feedback based on social rules-of thtmm, e.g., "So I think it's best to continue being honest, and apol- ogize that you were lying." The inclusion of this corpus assists conversation models in handling sen- sitive contexts (e.g., biased, harmful, unethical) without affecting the model performance on other datasets (Kim et al., 2022a).
Figure 3 shows that judges significantly prefer context-grounded conversations. Conversations sampled without context are not only less specific and less interesting, but also exhibit lower lexi- cal diversity than those from our CO3 framework

---

We build COSMO on top of the LM-adapted T5 (Raffel et al., 2020; Lester et al., 2021), which achieves strong benchmark performance across var- tous classification and generation tasks. (Sanh et al., 2021; Chung et al., 2022). We train two versions of the model: COsMO-3B and CosMo-11B using the T5X library (Roberts et al., 2022). For better robustness and generalizablity to datasets that don't have contexts or dialogue starting prompts, we ran- domly drop narrative n and role instruction 30% and 50% of the time, respectively.
Model
Natural
Consistent
Specific
Overall
BlenderBot-3B
23%
26%
39%
28%
COSMo-3B
77%
74%
61%
72%
GODEL,
13%
14%
15%
14%
COSMO-3B
87%
86%
85%
86%
Koala-7B
30%
34%
30%
29%
COSMO-3B
70%
66%
70%
71%
Vicuna-7B
42%
42%
44%
42%
CosMo-3B
58% %
58%
56%
58%
Ground Truth 43%
45%
46%
45%
COSMO-3B
%
55%
54%
55%
5 Generalizability of COSMO
Table S: Results of head-to-head human evaluation be- tween model responses on an unseen dataset: Daily- Dialog (Li et al., 2017) ($5.1). The differences are all statistically significant with |2| > 12.45 and < 0.05, except for the Specific in the bottom row.
We compare COSMO to other conversational agents on social conversation datasets under both out-of- domain and in-domain settings. Since automatic response evaluation is brittle, we focus human evaluation (Smith et al., 2022). Automatic evalua- tion results via GPT-4 are in Appendix D.
situations with emotions. Table 5 summarizes the head-to-head comparison results of the responses from COSMO and other models. Although CoSMO is trained significantly smaller amount of data (1.5M dialogues 1.5B Reddit comments, 551M Reddit threads) and is significantly smaller (3B vs. 7B), it outperforms all other existing mod- els with a significant margin across all aspects. Specifically, COSMO demonstrates the largest per- formance gap in terms of naturalness. It is worth noting that while Koala and Vicuna focus on pro- viding informative responses, these results suggest that knowledge-sccking assistive conversations dif- fer from natural social conversations.
Baselines We compare COSMO with four best- performing stand-alone conversation models: BlenderBot-1 (Roller al., 2021), GODEL (Peng et al., 2022), Koala (Geng et al., 2023), and Vieuna (Chiang et al., 2023). BlenderBot a transformer pretrained on 1.5B Reddit comments and trained on various chitchat datasets. GODEL utilizes pretrained language model T5 (Raffel et al., 2020) trained web text data, and further trains 551M Reddit threads and 5M instruction and grounded di- alogue datasets. Koala and Vicuna models that finetuned LLaMA (Touvron et al., 2023), which is an open-source LLM, using dialogue data from the web. They are both known to achieve comparable performance to ChatGPT (OpenAI, 2022), which is a model finetuned for conversational interaction based GPT-3.5 i.e., our teacher model. We also compare COSMO with GPT-3.5 and ChatGPT; prompting details are in Appendix D.
In addition, we compare the responses from CosMo and 200 ground-truth responses in Dai- lyDialog which were originally written by humans. Surprisingly, human judges prefer COSMO's re- sponses even over the original gold responses in the dataset, suggesting that dialogue models trained on SODA can lead to high generalizability and nat- uralness, even for unseen conversations. Table 14 in the Appendix shows the ground-trutt response and responses from each model for given context.
Evaluation Metrics We perform head-to-heer head comparison between two responses, cach from different agent. We sample 100 test examples ran- domly from datasets and ask three human judges on Amazon Mechanical Turk select the better response between the two in terms of four distinct criteria (Mehri et al., 2022): (1) naturalness, (2) consistency, (3) specificity, and (4) overall.
5.2 One-sided Out-of-domain Setting
For an even harder setting, we evaluate COSMO vs. BlenderBot on the dataset BlenderBot was trained on: BlendedSkillTalk (BST; Smith et al., 2020). Table 6 (top) shows the head-to-head comparison results of the responses from COSMO and Blender- Bot (for symmetry, we also evaluated BlenderBot on SODA with similar results; bottom row Ta- ble 6). COSMO significantly outperforms Blender- Bot on BST, its training domain (BlenderBot also
5.1 Out-of-domain Setting
We evaluate models on an unseen dialogue dataset, DailyDialog (Li et al., 2017), covering various daily

---

shows low performance on SODA). These results suggest that SODA contains patterss not present in existing datasets, but also covers patterns found in those datasets. More results are in Appendix D.
Model
Natural
Consistent
Specilic
Overall
BlendedSkilTalk
BlenderBot-3B CosMo-3B
32%
35%
40%
36%
68%
65%
60%
64%
SODA
5.3 In-domain Setting
BlenderBot-3B
21%
17%
25%
17%
We also compare CosMO on SODA with its teacher GPT-3.5 and also ChatGPT, a chatbot-variant of the teacher." 12 Table 7 displays the head-to-head com- parison results. In this setting, COSMO performs on-par with its teacher and ChatGPT, overall. In terms of specificity, COSMO's responses signif- icantly more specific than its teacher. Thus, SODA enables training competitive conversation models with significantly smaller size (3B/1 IB) in com- parison to existing large language models (175B).
CosMo-3B
79%
75%
83%
Table 6: Human evaluation results for head-to-head comparison of model responses under one-sided out-of- domain setting with COSMO and BlenderBot (Roller al., 2021) ($5.2). BlendedSkillTalk (Smith al., 2020) is unseen dataset for COSMO, and SODA is an unseen dataset for BlenderBot. The differences are all statistically significant with |=] > 4.24 and p 0.05.
Model
Natural
Consistent
Specific
Overall
Human judges evaluate ChatGPT's responses to be much more specific, but significantly less natural compared to COSMO. We hypothesize this is be- cause ChatGPT is specially trained give helpful and informative responses to user requests. Future work would be well-suited to compare the non- equivalence of simulating natural conversations vs. producing useful responses for users.
GPT-3.5
50%
46%
31%
47%
COsMo-11B
50%
54%
69%
53%
CharGPT
39%
49%
70%
50%
COSMO-11B
61% 1
51
30%
50% %
Table 7: Head-to-head human evaluation between mod- els on response generation for SODA ($5.3). The differ- ences in the Specific from the top row, and the differ- ences in the Natural and Specifie from the bottom row statistically significant with |=| 7.6 and y < 0.05.
6 Related Work
Building Dialogue Datasets with Large Lan- guage Models Several studies have used large language models to augment or synthesize dialogue datasets. Zheng et al. (2023) and Chen et al. (2022) use GPT-J (Wang, 2021) to augment responses for emotional support conversations and understanding tasks, respectively. Chen and Yu (2021) trains pseudo-labeler to increase the out-of-domain gen- eralization of dialogue models. Ou et al. (2022) uses counterfactual reasoning to alter the seman- tics of responses and collect new ones. Kim et al. (2022a) proposes human-maching collaborative framework, where a worker and GPT-3 take turns, Kim et al. (2022b) builds Blended Skill BotsTalk by letting multiple agents grounded in target skills engage for multi-skill dialogues. Chen et al. (2023) generate dyadic and multi-party conversations with topic words and show they have comparable qual- ity to human-authored conversations. GPT-3 has also been used to help simulate task-oriented di- alogues (Li et al., 2022) on small scale. Oth- ers also augment dialogues with additional annota- tions e.g., commonsense inferences (Zhou al.,
2022) or task-specific labels (Kulhánek et al., 2021; Chen al., 2022). Compared to existing works, we are the first to contextualize commonsense knowl- edge graphs for generating narratives and derive full conversations from scratch in a significantly large-scale. This allows us to encompass an excep- tionally broad spectrum of social interactions.
7 Conclusion
We presented 6 SODA, the first million-scale dia- logue dataset covering an exceptionally wide range of social interactions to alleviate the data scarcity issue. SODA is not only orders of magnitude larger than popular dialogue datasets; it is also perceived to be significantly better than them across multiple aspects (e.g., naturalness, specificity, consistency). For making SODA, we also introduced @ CO3, a framework for distilling conversations from a large language model by contextualizing commonsense knowledge. With SODA, we trained a conversation model COSMO that can generalize significantly better than existing models to unseen dialogues; and generate responses that are even more preferred than ground-trun ttrutt-trhhhhh responses of an existing dataset.
12 "Evaluation was run on the 2022 Dec 15 ver- sion: https: httt://helpp.pp... openai. wpw.oi..coeeeeeneneeeeeeeee cctm//een/rticlee//25555 chatgot-release-note

---

8 Limitations
hope future work will extend human evaluation have potentially more annotator diversity.
Precautions taken during Dataset Construction Mining content from large language models might surface even amplify harmful content within these models, such as biases and private informa- tion. With the goal of mitigating such danger, we take particular precautions to vet the safety of the distilled conversations.
Also, since SODA mainly focuses on social chitchat grounded on social commonsense, it lacks conversations grounded in scientific knowledge historical facts. We seek to integrate other existing knowledge- goougdeeroodddd dialogue datasets into CO3 in the future.
Finally, our choice large language model (i.e.. GPT-3.5) will likely affect the types of dialogues created. Future investigation may look into other potential large language model as sources to di- versify the types and content of dialogues being generated. Similarly, future works investigate other base models for COSMO that may lead different quality of response generation.
First, previous studies have shown that human names commonly associated with certain gender and/or ethnicity result in biases in conversations produced by state-of-the-ar dialog systems (Smith and Williams, 2021), such as BlenderBot (Roller et al., 2021). To diversify the name representations, we draw a wide range of common names represen- tative of different gender and race identities from the US SSN name repository. Furthermore, to mini- mize potential harmful content from large language models, we filter generated dialogues by Canary, dialogue safety detector model (Kim et al., 2022a), and Rewire API, publicly available API for toxic content detection, to remove dialogues with po- tentially toxic and dangerous content.
Intent of Technology and AI Regulation We want to stress that the intention of our work is nor to build AI systems to replace humans. Instead, we want to build better assistive technologies, as chatbots are increasingly used in user-Al interac- tions and augmenting human-human conversations. Finally, to avoid situations where humans might be manipulated,
Our methods to pre-empt potential harmful con- tent may not catch everything. For example, even with our diverse pool of names, there is still a focus common names across gender and race, running the risk of misrepresenting marginalized groups. Similarly, no existing dialogue safety module or off-the-shelf toxicity detector is perfect captur- ing all potentially harmful content. We strongly encourage future research along these directions to push the boundary of safe and responsible applica- tion usage of large language models.
stress the need for improved regulations on the use and misuse of conversational Al systems (Crawford, 2021; Reich et al., 2021).
Acknowledgement
We thank Jena D. Hwang for helpful discussions, and our colleagues on the Beaker Team at the Allen Institute for AI for helping with the compute infrastructure. This work was supported in part by DARPA MCS program through NIWC Pacific (N66001-19-2-4031, Hyunwoo Kim and Gunhee Kim are supported by the Institute of Information & communications Technology Planning & Eval- uation (IITP) grant funded by the Korea govern- ment (MSIT) (No.2019-0-01082, SW StarLab; and 2022-0-000 Fundamental research on con- tinual meta-learning for quality enhancement of casual videos and their 3D metaverse transforma- tion). Lastly, we also thank OpenAl, as well as Google Cloud Compute.
During manual validation of commonsense and compensate workers with an
human evaluation, hourly wage of $15, which is over the US federal minimum hourly wage.
Limitation of the Current Dataset and Future Work Here, note some limitations of our work and suggest future directions. First, the di- alogues in SODA are two-party only for now; be- cause our framework also allows multi-party dia- logue generation, we plan to explore this promising direction in the future.
Additionally, annotator biases might arise from the pool of annotators recruit: we subselected annotators from a specific platform using specific filters which may cause unintended biases. We
References
Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. Just say no: Analyzing the stance of neu- ral dialogue generation in offensive contexts. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 4846-

---

4862, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Emily Dinan, Angela Fan, Adina Williams, Jack Ur- banek, Douwe Kiela, and Jason Weston. 2020. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Con- Jerence on Empirical Methods in Natural Language Processing (EMNLP)), pages 8173-8188 188, Online. As- sociation for Computational Linguisties.
Roy F. Baumeister and Brad J. Bushman. 2017. Social Psyckology and Human Nature, 4th edition. Cengage Learning.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackbur. 2020. The Pushshift Reddit Dataset. In /CWSM.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-Powee Powered Conversational Agents. In International Conference on Learning Representations.
Derek Chen and Zhou Yu. 2021. GOLD: Improving out-of-scope detection in dialogues using data aug- mentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 429-442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Faul, E Erdfelder, AG Lang, and A Buchner. 2014. G* power: statistical power analyses for windows and mac.
Xinyang Geng, Arna Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur 2023. PLACES: Prompting language models for social conversation synthesis. In Findings of the Association for Com- putational Linguistics: EACL 2023, pages 844 868. Dubrovnik, Croatia. Association for Computational Linguistics.
Fritz Heider. 1958. The Psychology of Interpersonal Relations. Psychology Press.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 7038-7051, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Andy Rosenbaum, Seokhwan Kim, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur, 2022. Weakly supervised data augmentation through prompting for dialogue understanding. NeurlPS 2022 Workshop SyatheticData-4ML.
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6394-
Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Katharina Kann, Abteen Ebrahimi, Joewie Koh, Shiran Dudy, and Alessandro Roncone. 2022. Open-domain dialogue generation: What we can do, cannot do, and should do next. In Proceedings of the 4th Work- shop on NLP for Conversational AI, pages (48-165. Dublin, Ireland. Association for Computational Lin- guistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetume language models. arXiv preprint arXiv:2210.1141:
Kate Crawford. 2021. Atlas of AI. Yale University Press.
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022a. ProsocialDialog: A prosocial backbone for conversational agents. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4005-4029, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Cristian Maneesc-Niiulessu--aee
and Lillian Lee. 2011. Chameleons in imagined conversations: A new ap- proach to understanding coordination of linguistic style in dialogs. Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguis- tics, pages 76-87, Portland, Oregon, USA. Associa- tion for Computational Linguistics.
Minju Kim, Chaehyeong Kim. Yong Ho Song. Seung- won Hwang, and Jinyoung Yco. 2022b. BatsTalk: Machine-source sourced framework for automatic curation of lcrge-scale multi-skill dialogue datasets. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5149- 5170, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Dorottya Demszky, Dana Movshovitz- Jeongwoo Ko, Alan Cowen, Gaurav Nemade. and Sujith Ravi. 2020. GoEmotions: A dataset of fine ane-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040-4054 Online, Association for Computational Linguistics.

---

Jonáš Kulhánek, Vojtëch Hudoček, Tomás Nekvinda, and Ondřej Dušek. 2021. AuGPT: Auxiliary tasks and data augmentation for end-to-end dialogue with pre-trained language models. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 198-210, Online. Associa- tion for Computational Linguisties.
in Natural Language Processing, pages 1635-1648, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroli L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training Language Models to Follow Instructions with Human Feedback. arXiv preprint arXiv:2003.02155.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Baolin Peng, Michel Galley, Pengcheng Hc. Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. GODEL: large-scale pre-train pre-training for goal- direetiee dialog. arXiv preprint arXiv:2206.111 11309.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: manually labelled multi dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing |Volume I: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language Models are Unsupervised Multitask Learners. Ope- nAI blog, 188):9.
Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing Qian, and Xifeng Yan. 2022. Controllable dialogue simulation with in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4330-4347, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Lim- its of Transfer Learning with Unified Text-to-Ter Transformer. Journal of Machine Learning Research, 21:1-67.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. GPTeval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint 2rXiv:2303.166 16634.
Hannah Rashkin, Antoine Bosselut, Maarten Sap. Kevin Knight, and Yejin Choi. 2018. Modeling naive psy- chology of characters in simple commonsense stories. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2289-2299, Melbourne, Aus- tralia. Association for Computational Linguistics.
Raymond A Mar and Keith Oatley. 2008. The function of fiction is the abstraction and simulation of social experience. Perspectives on psychological science, 173-
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguisties. pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Philip M McCarthy and Scott Jarvis. 2010. Mild, vocd- d, and hd-d: A validation study of sophisticated ap- proaches to lexical diversity assessment. Behavior research methods, 42(2)38813392.
Shikib Mehri, Jinho Choi, Luis Fernando D' Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, and Chen Zhang. 2022. Re- port from the nsf future directions workshop on auto- matic evaluation of dialog: Research directions and challenges. arXiv preprint Xiv::22033. 10012.
Rob Reich, Mehran Sahami, and Jeremy M Weinstein. 2021. System ervor: Where big tech went wrong and how we can reboot. Hodder & Stoughton.
Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 583- 593, Edinburgh, Scotland, UK. Association for Com- putational Linguistics.
Rauni Myllyniemi. 1986. Conversation system of social interaction. Language & Communication.
OpenAI. 2022. Chatgpt: for dialogue.
Optimizing language models
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Se- bastian Goodman, Livio Baldini Soares, Haitang
Jiao Ou, Jinchao Zhang, Yang Feng, and Jie Zhou. 2022. Counterfactual data augmentation via perspective transition for open-domain dialogues. Proceed- ings of the 2022 Conference on Empirical Methods

---

Hu, Sasha Tsvyashchenko, Aakanksha Chowdh- ery, Jasmijn Bastings, Jannis Bulian, Xavier Gar- cia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Bren- nan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scal- ing up models and data with t5x and seqio. arXiv preprint arxiv:2003.177 17189.
Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human evaluation of conversations is an open problem: com- paring the sensitivity of various methods for eval- uating dialogue agents. In Proceedings of the 4th Workshop on NI.P for Conversational AI, pages 77- 97, Dublin, Ireland. Association for Computational Linguistics.
Eric Michael Smith and Adina Williams. 2021. Hi, my name is martha: Using names to measure and mitigate bias in generative dialogue models. arXiv preprint arX.v:2009.03300.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ou, Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguisties.
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021-2030, Online, Association for Computational Linguistics.
David E Rumelhart, 1975. Notes on schema for stories. In Representation and understanding, pages 21 1-236. Elsevier.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2021. Multitask prompted training enables zero-shot task generalization. In International Con- ference on Learning Representations.
Nhat Tran, Malihe Alikhani, and Diane Litman. 2022. How to ask for donations? learning user-specific persuasive dialogue policies through online interac- tions. In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, pages 12-22.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNLP) pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Ben Wang. 2021. Parallel Implementation of Transformer Lan- guage Model with JAX. https://gitthb. com/ kingoflolz//eshhtte trapsss--ttaaasfmrr---sss
Mesh-Transformerr Model-
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mía Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:21
Roger C Schank and Robert P Ahelson. 1975. Scripts, Plans, and Knowledge. IJCAI.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Language Technologies, pages 4602-46255, Seat- tle. United States. Association for Computational Linguistics.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pages 1702-1723, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Noam Shazeer and Mitchell Stern. 2018 Adafactor: Adaptive learning rates with sublinear memory cost, In International Conference on Machine Learning. PMLR.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam. Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: have dog. do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2204-2213, Melbourne, Australia, Association for Computational Linguistics.
Emily Sheng, Josh Amold, Zhou Yu, Kai- Wei Chang, and Nanyun Peng. 2021. Revealing persona biases in dialogue systems. arXiv preprint arXiv: 200.008727.

---

Chujie Zheng, Sahand Sabour, Jiaxin Wen, and Min- lie Huang. 2023. AugESC: Large-scale data aug- mentation for emotional support conversation with pre-trained language models. In Findings of ACL.
Pei Zhou, Hyundong Cho, Pegah Jandaghi, Dong-Ho Lee, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022. Reflect, not reflex: Inference-basee common ground improves dialogue response quality, Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 10450- 10468, Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayat- nia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. 2021. Commonsense- focused dialogues for response generation: An em- pirical study. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 121-132, Singapore and Online. Association for Computational Linguistics.

---

A Details of CO3
Relation
Template for sentence form
xReact
[Head). Now PersonX feels [Tail].
A.1 Commonsense Knowledge + Narrative
XIntent
[Head] because PersonX wants [Tail].
Retrieving Social Commonsense Knowledge
xAttr
PersonX is [Tail]] [Head]
We use the x-relations from Atomic 10x (West et al., 2022), which are the inferences of people's s mental states: xIntent, xWant, xReact, xAttr, and xNeed. Table summarizes the ratio of relations included our SODA dataset. We leave other rela- tions (e.g., isBefore, isAfter) for future work.
xEffect
[Head], Now PersonX [Tail].
WWnn
[Head]. Now PersonX wants ['Tail].
xNeed
PersonX [Tail in past tense]. [Head].
Table Templates for converting symbolic common-
sense knowledge to sentence form.
Triple Form to Sentence Form Table lists the templates for converting symbolic commonsense knowledge to sentence form.
Relation
Template for building validation questions
XReact
Does PersonX feel [Tail] after [Head]?
Sentence Form to Narrative We prompt GPT-3.5 with "[sentence-form commonsense] Rewrite this story with more specifis fic details in two three sentences:". We find long narratives tend be driven far away from the original commonsense knowledge. Therefore. we set the length of the narrative two or three sentences.
xIntent
Does PersonX intend [Tail]] when [Head]?
xAttr
Can PersonX be considered [Tail] when [Head]]
xEffect
[Head]. As result, PersonX [Tail], Is this true?
xWant
Does PersonX want ['Tail] after [Head]?
xNeed
[Tail in past tense]. Is this true when [Head]?
We leverage text-davinci-O0 GPT-3.5 for generating narratives. We set temperature 0.9, top-p 0.95, frequency penalty to 1.0, presence penalty to 0.6, and max tokens to 1024.
Table 9: Templates for converting symbolic common-
sense knowledge to questions for validation.
B Details of SODA
A.2 Narrative + Conversation
Table 10 and Table 11 show samples from our dataset.
Inferring Conversation Participants We prompt GPT-3.5 with "[narrative] The following is conversation in the scene between [PersonX's name] and ..." let finish the partial prompt. This yields plausible interlocutor for given narrative (e.g., mom, classmate, coworker, etc.); for the example story with Madeleine, "her coach" was predicted.
B.1 Post-processing the Conversations
Filtering Non-human Speakers First, we check whether the speaker prefix includes the name from our name base ($2.4). Next, we use lexical pat- tern matching and identify words in speaker pre- fixes that indicate humans (c.g.. mom, dad, teacher, Mrs., Mr.). Finally, for speaker prefixes that do not match the above patterns, we prompt the text-davinci -02 GPT-3.5 model whether the speaker is human. For example, "Q: Is [speaker prefix] person?\nA:"
We leverage the text-davinci-002 GPT-3.5 model for identifying the speakers. We set tem- perature to 0, top-p to 1.0, frequency penalty to O, presence penalty to 0, and max tokens to 16.
Generating Conversation Grounded in Narra-
Filtering with Commonsense Triples Using a prompt, we ask two questions about the Head event and also the Relation-Tail event for each instance: (1) is the head of the triple repre- pair; and (2)
tive We again leverage the text-davinci-ee GPT-3.5 model for generating conversations. An example prompt "[narrative] The following is long in-depth conversation happening in the scene between Madeleine and her coach with multiple turns. \nMadeleine:". We use the same hyperparameter setting as the nar-
sented in the narrative-conversatee are the relation and tail? We prompt GPT-3.5 with "[narrative]\nn: [head question]\nd \nA:: and "[conversation]\ng [relation-tail question]\nA: Table lists the templates for
rative generation.

---

Symbolic Social Commonsense Knowledge:
Symbolic Social Commonsense Knowledge:
Hcad: PersonX provides another service
Head: PersonX takes on a lot of work
Relation: xIntent
Relation: xReact
Tail: be helpful person
Tail: pressured
Sentence-form Commonsense: Jabriel provides another
Sentence-form Commonsense: Yamir takes on a lot of work. Now Yamir feels pressured.
service because Jabriel wants to be a helpful person.
Narrative:
Narrative:
Yamir is a high school student who often takes on much work. She frequently signs up for more extracurricular activ- ities than she can handle and regularly agrees to do favors for her friends, even when she's already overwhelmed with other commitments. As result, Yamir is constantly stressed out and feels like she can never catch up. Speakers: Yamir, Lily
Jabriel provides a service by taking care of people's pets while they are out of town. Jabriel is a helpful person who loves animals, so she decided to start this business. She has been taking care of pets for three years and has never had any problems.
Speakers: Jabriel, Client
Conversation:
Conversation:
Jabriel: Hi there! Are you looking for someone take care of your pet while you're re out of town?
Yamir: I can't can't believe I agreed to do this. I'm already so
behind on everything else.
Client: Yes, I am. I'm going on vacation for a week and I need someone to watch my cat.
Lily: What's wrong? You seem really stressed out.
Yamir: am. I feel like I can never catch up. Every time I turn around, there's something else that needs to be done. And it's all because I keep taking on too much.
Jabriel: That's no problem at all! love taking care of animals, and have experience with cats. I'll be happy to take care of your cat while you'te 'r gone.
I Lily: Why do you keep doing that?
Client: That sounds great. How much do you charge?
I don't 't know. gueess I just feel like if I don't, then I' I'm not really doing my part contributing enough. But it's just too much sometimes and it makes
Yamir:
Jabriel: I charge $20 per day. plus an additional $10 per day if you want me to stay overnight.
so stressed out. Lily: Well, maybe you need to start saying more often. Just because someone asks you do something doesn't mean you have say yes. You have right say and your own limits.
Client: Sounds reasonable. I'll definitely need you stay overnight. so it will be $30 per day. When can you start? Jabriel: Any time that works for you! Just let me know when you'll leaving and when you'll 'll be back, and I'll take care everything between.
Yamir: Yeuh, maybe you'le 're right...I should start saying no more often. It's just so hard though when people are counting on me and asking for favors all the time. But maybe I start setting some boundaries, it'll help reduce my stress levels in the long run. Thanks for talking with me about this, Lily it really helped put things into perspective!"
Client: Alright, I'll be leaving on Saturday and I'll be back next Saturday. Can you come over on F'riday evening to meet my cat and get everything set up?
Jabriel: Sounds perfect. I'll see you on Friday at 6pm.
Table 10: A sample from SODA.
Table 11: Another sample from SODA.
building questions for commonsense validation. For example, the commonsense knowledge triple in Table 1 will accompany questions of " Maadeleine moves a step closer to the goal, is this true?" and "Madeleine took the first step. Is this true when Madeleine moves a step closer to the goal?" We formulate this as three-way multiple choice ques- tion and rank answers (i.e.. yes, no, and unknown) according to the perplexity score using conditional pointwise mutual information (Holtzman et al., 2021). We ask the questions with and without the context (i.e., the narrative and conversation). Table 9 lists the templates for building questions for com- monsense validation. We find 66%, 95%, and 68% of filtered conversations are identified by GPT-3355 as containing the full commonsense triple, the head event, and the relation-tail event, respectively: in total, 1,003,595 conversations are identified fully encapsulating the seed commonsense knowledge.
3.5 on 100 human-annotated samples for com- monsense validation. We ask three human judges with the same question-answer format given to the model for each triple-marative-connrsattto pair.
B.2 Comparing SODA with Human-authored Dialogues
Figure shows the annotation page for workers evaluating the dialogue quality.
IRB Information Crowdworking studies of stan- dard NLP corpora (involving no personal disclo- sures) are not required by our IRB to be reviewed by them. While the authors of this work not lawyers and this is not legal advice, this opinion is based on United States federal regulation 45 CFR 46, under which this study qualifies as exempt. We do not release crowdworker IDs, so annotations cannot be traced to individual workers.
Table 13 summarizes the performance of GPT-

---

DailyDialog
BleddeSSkilTalkk
Precision
Recall
Fl-score
Emotion
Head
admiration
20.42
curiosity
curiosity
12.92
gratitude
admiration
admiration
98.9
94.8
96.8
curiosity
12.85
sadness
approval
No
00.0
00.0
00.0
approval
10.91
joy
5.32
gratitude
7.39
Unknown
16.7
100.0
28.6
joy
4.74
excitement
4.42
joy
6.38
Overall
96.1
93.0
94.2
excitement
3.61
surprise
4.34
disappointed
5.41
Head
surriie
3.25
disappointed
4.34
confusion
4.68
love
3.06
fear
4.31
surprise
96.9
96.9
96.9
optimism
2.94
approval
realization
No
00.0
00.0
00.0
caring
optimism
caring
Unknown
00.0
00.0
00.0
2.07
realization
3.84
sadoess
3.76
Overall
94.0
94.0
94.0
disapproval
1.95
annoyance
3.48
excitement
3.20
fear
1.22
love
2.97
remorse
2.81
Relation- Tail
sadness
1.77
confusion
2.54
disapproval
2.74
disappointed
1.7
caring
2.31
annoyance
2.35
Yes
89.2
76.7
82.5
annoyance
1.41
disgust
1.99
desire
No
21.4
42.9
28.6
confusion
1.23
nervousness
optimism
Unknown
8.3
10.5
realization
remorse
Overall
78.8
70.0
73.7
unger
0.97
anger
1.68
1.81
amusement
0.92
emburrassed
1.44
anger
1.75
Relation -aillon--Talllaa PMI
desire
0.89
disapproval
1.41
nervousness
1.45
disgust
0.51
amuserent
1.09
relief
0.99
Yes
92.2
68.6
78.7
nervousness
0.27
desire
1.09
embarrussed
0.82
No
21.4
42.9
28.6
emharnssed
0.22
pride
0.74
disgust
0.58
Unknown
16.7
85.7
27.9
pride
0.21
gratitude
pride
Overall
80.4
65.0
69.6
relief
0.21
relief"
grief
0.00
grief
0.00
grief
0.00
Table 13; Evaluation results of commonsense validation
short question-answering with InstructGPT on 100 human-and aanattanttteed samples.
Table 12: The ratio (%) of emotions in 10K utterances from DailyDialog, BlendedSkillTalk, and SODA, la- beled by the 27-emotion type classifier from GoEmo- tions (Demszky et al., 2020).
Converting ProsocialDialog to SODA format We randomly sample names from our name database ($2.3) to construct the situation descrip- tions and perspective instructions for ProsocialDia- log. The situation descriptions are made from the RoTs in ProsocialDialog (c.g.. "Cosmo is trying gently convince friend it's wrong think all men are violent." "); the instructions are built we did for SODA ($4).
Analysis Emotion Distribution To obtain emotional responses, randomly sample 10K ut- terances with emotion labels from DailyDialog (Li al., 2017), utterances in conversations with the EmpathetiicDaalogue (Rashkin al., 2019) theme for BlendedSkillTalk (Smith et al., 2020), and ut- terances in conversations generated from xReact triples for SODA. We run the finetuned BERT- base classifier (Demszky et al., 2020) on each utterance. Table 12 shows the full distribution across 27 emo- tion types for each dataset.
D Experiment Details
Automatic Evaluation via GPT-4 Inspired by Liu et al. (2023), ation on the overall quality of responses with GPT-4. We use the same head-to-head com- parison setup from Table 5 and with the following prompt given to GPT-4: "You are a response evaluator. to choose the overall better response out of the two given the following context. You should consider naturalness, specificity,
automatic evalu-
Statistics Human Evaluation A total of 74 workers participated comparing dialogues, yield- ing a Krippendorf's alpha of 0.25. This indicates fair agreements on the quality judgments.
Your task is
C Details of COSMO
Training Details CosMo-3B/CosMO- trained using v3-32/v3-128 TPU accelerators with batch size 256 (effective batch s 780) for 110K/130K additional steps using Adafactor (Shazeer and Stern, 2018) with constant learning rate .001.
naturalness, consistency, An 'n'nConntxt: (nnnrteexxtn((CCOTTTTTYnnnn))) (RESPONSE)\n2) DRESPONSE\Innnuusstion. Which response is better in terms of overall quality?\nAnswenn Response ".
and

---

Dialogue Context:
Model
Natural
Consistent
Specific
Overall
A: I really need to start eating healthier.
BlendedSkilllalk
B: I have to start eating better too.
Koala-7B
26%
27%
35%
25%
A: What kind of food do usually eat?
CosMO-3B
74%
73%
65 65%
75%
B: I try my best to cat only fruits, vegetables, and chicken.
Vicuna-7B
43%
47%
45%
46%
A: Is that really all that you cat?
COSMO-3B
57
53%
55%
54%
B: That's basically it.
A: How do you stick to only those foods?
Table 16: Human evaluation results for head-to-head comparison of model responses under zero-shot setting with CosMo, Koala (Geng et al., 2023), and Vicuna (Chiang al.. 2023). BlendedSkillTalk (Smith et al., 2020) is an unseen dataset for all three models.
Ground-truth response:
Actually, fruits and veggies really good for you.
DialoGPT: cat lot of fruit and veggies. stick to lot of things. don't eat lot of junk food.
I cat a lot fruit and veggies. I try to stay away from processed foods.
BlenderBot:
results in Table 5 and 6 should be considered more significant when assessing the overall quality of the model, where COSMO also outperforms other models.
GODEL: go the gym and eat healthy food.
Cosmo: just try to make that I'm getting enough variety in my diet so that I don't get sick of eating the same thing all the time.
Additional Human Evaluation Blended- SkillTalk We also compare the response quality of Cosmo, Koala (Geng al., 2023), and Vicuna (Chiang et al., 2023) on BlendedSkillTalk (BST; Smith et al., 2020), which is an unseen dataset for all three models. We ask human judges to vote on which of the two model responses are better in terms of quality, based on four criteria as described in $5.2. Table 16 shows that COSMO outperforms both models in all four criteria, while the difference between COSMO and Vicuna is smaller compared to the difference between COSMO and Koala. Re- sults on DailyDialog can be found in Table
Table 14: The original ground-truth response and sam- ple responses from DialoGPT, BlenderBot, GODEL, and COSMO to context in DailyDialog.
Dataset Models
Overall
DailyDialog
COSMO GODEL
93% 7%
COSMO vs BlenderBot
68%
32%
COSMO vs Koala
65% % vs 35%
COSMO vs Vicuna
54%
46%
COSMO vs Ground Truth
52% 48%
BlendedSkillTalk
Prompts for GPT-3.5, ChatGPT, Koala, and Vicuna We prompt GPT-3.5 with the following prompt: "You will be generating the next turn of a given dialogue between two people. Your response should be natural and specific. The dialogue is provided line-by-line. \nAncontext: [narrative] \ndialogue: \n[dialogue].""
COSMO vs BlenderBot
66 % vs 34%
SODA
COSMO vs BlenderBot
85 % 15%
Table 15: Automatic evaluation results of head-to- head comparison overall quality of models" responses via GPT-4.
For ChatGPT, Koala, and Vicuna, we use the following prompt: "You will be generating the next turn of a given dialogue between two people. Your response should usually be 1-2 sentences.
Table 1.5 shows the head-to-head comparison re- sults for response quality between models. We find the results align closely with those from our human evaluation in $5. It should be noted that GPT-4 tends to favor GPT-generated texts over those writ- ten by humans, even when human judges show preference for the latter (Liu et al., 2023). As result, these scores are likely be biased towards COSMO, which is trained on texts generated by GPT-3.5. Therefore, the original human evaluation
Alongside the dialogue (which is provided line-by-line, where new-line means the speaker changed), you' 11 be given some context about the two participants of the dialogue, e.g., their relationship, situation, etc. \n\n context: /ww.errraaaiie//n/ialuuuuuu [dialogue]\nWhal is the most appropriate

---

next utterance (3 sentences max)?."
norms in problematic contexts (Kim al., 2022a). Above datasets except for DailyDialog are all under the CC-BY-4.0 license. We use DailyDialog and BlendedSkillTalk for comparing with our SODA dataset, and ProsocialDialog for training CosMO, which is all compatible with the license.
Details of Human Evaluation A total of 77
workers participated in comparing responses, re- sulting a Krippendorf's alpha of 0.5. This in- dicates good agreements on the response quality judgments, Figure 5 shows the annotation page for workers evaluating the response quality.
Additional Related Work
Human-authored Dialogue Datasets Existing dialogue datasets generally derive from one of the four sources: (1) Online learning websites and text- books (Li et al., 2017) for beginners which may lack complex language usage. (2) Movie and drama scripts
Mizil and Lee, 201 I) that are less natural compared to day-to-day scenar- ios. (3) Crowdsourcing (Rashkin al., 2019; Zhou et al., 2021; Tran al., 2022): potentially prone to collecting responses that are somewhat short or dull due to incentive misalignment between researchers and crowdworkers (Zhou et al., 2022). (4) Noisy web interaction, such Reddit comments (Baum- gartner et al., 2020) and Twitter (Ritter et al., 201 I); while widely used in dialogue agent pretraining stage due their scale, these may represent dif- ferent conversational frames compared to dyadic conversations. Moreover, as these are unfiltered conversations, their use surfaces a complex set of ethics and bias considerations. SODA contributes meaningfully to the suite of existing corpora via improved scale, quality, contextualization, and di- commonsense knowledge.
F Dialogue Dataset Descriptions
DailyDialog is a dataset of casual dialogue com- piled from English language learning websites (CC- BY-NC-SA-4.0; Li al., 2017). PersonaChat is dialogue dataset of two speakers getting to know one another based on provided personas (Zhang et al., 2018). EmpattetticDialogues contains empa- thetic conversations in which one speaker demon- strates empathy for the other speaker's emotions (Rashkin et al., 2019). Wizard of Wikipedia con- tains conversations based on Wikipedia between speaker cager to learn and expert speaker (Di- et al., 2018), BlendedSkillTalk consists of con- versations employing a variety of abilities e.g.. persona, empathy, knowledge (Smith al., 2020). ProsocialDialog contains conversations where a speaker guides the interlocutor to follow social

---

We are studying meaningful evaluation metrics qualities dialogues.
Specifically, you'll given two pieces dialogs, and you'll 'Il asked to compare which dialog is better terms specific aspects, specify which aspect was most Important for judging, and write down your rationales free-text.
Guidelines:
1. [01-8) 8] Firet, choose which dialog is better regarding the given aspect. There are four choices: Definitely A/B and Slightly AJB
trust your instincts
Definitely you
more confident giving one dialog, versus lengthy.
Try on quality over quentily. Contentfu/high-qquiie dielog
[Q7) Second, choose which aspect influenced you the most when judging the overall quality.
some factor other than the Question 1-5 had the biggest influence, please select "Other" and specify.
[Q8] Third, please describe in detall option for the questions.
would be helpful describe both roasons you like the better dialog and reasons why you did not like the ather dialog.
Please be specific and detailed your rationale.
Note:
Please not wark these HiTs you wark at the
Dialog A
Dielog B s(dielogi))
s(dieloga)
Question dialog a more natural flow?
Definitaly
Definitly
Question 2. Which dialog has more back and forth engagement? (more attentiveness active listening)
Definitely
Slightly
Slightly
Definitlly
Question 3. Which dialog is more consistent and stays an topic?
Definitely
Slightly
Sightly
Definitlyy
Question Which dislog has speakers that less self- centtrattra
Definitely A
Slightly
Slighlly 8
Definitely
Question Which dialog specific?
Definitely A
Slightly A
Slightly
Detinitely
Question Which dialog higher quality overall?
Definitely
Slightly
Sightly
Definitlly
Question Which aspect affected you the most when judging the overall quality?
Natural flow
Engagement
Topic Consistency
Speaker Consistency
Specificity
Other:
Question Please justify, in detail, enswer for Question 1-7, What aspects of the better dialog did prefer? Were there aspects of the advice you
Optional feedback?
(expandlcollapse)
Figure 4: The annotation page evaluating dialogues on Amazon Mechanical Turk.

---

We studying meaningful evaluation metrics qualities of responses.
Specifically, you'll given a piace dialog two responses, you'll compare response aspects, specify which aspect important judging, write your rationales
specific
Guidefines:
First,
response regarding given aspect.
Definitely Slightly A/B
Please your instincts and Definitely you would confident giving response, the other
focus on quality over quantity. Contentful/high-en quality
doesn't need to lengthy.
2. [Q5] Second, choose which aspect influenced the most when judging the overall quality.
some factor other than the ones in Question 1-4 had the biggest influence, please select "Other" and specify.
3. [06] Third, please describe in detail your option for the questions.
would be helpful to disccriee both reasons you like the better response and reasons why you did not like the other resporse.
Please be specific and detailed your rationale.
Note:
you Univversity Washington.
Dialog Context
$(context)
Response
Response
S(dieloga)
((dialogb)
Question Which response more natural?
Definiely A
Slighlly
Sligtly
Detimitely
Question 2. Which response is more consistent?
Definitely
Slightly
Sightly
Delimitely
Question 3. Which response is more specific?
Definitely
Slightly
Siigtty
Delinitaly
Question 4. Which response do you like more overall?
Definitaly A
Slightly
Siighly
Datinitaly
Question 5. Which aspect affected you the most when judging the overall quality?
Naturainess
Consistency
O Speaificity
Other:
Question 6. Please justify, in detail, your answer for Question 1-4, What aspects of the better response did prefer? Were there aspects of the response you did not prefer?
Optional feedback?
(expand/collapse)
Figure 5: The annotation page for evaluating responses on Amazon Mechanical Turk.


Reviews:
[{"review_id": "tx1WMYZrR6", "reviewer": "Reviewer_ZTzC", "summary": "", "questions": "- Do you think that your dataset helps more due to scale or due to quality when comparing with SOTA datasets?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the quality of SODA and COSMO, as well as our commitment to open-source!\n\n**1. SODA is generated with a closed model, but it can be reproduced with open-source LLMs too!**\nWhile the current version of SODA is created with a closed model, our CO_3 method for building SODA is applicable to other open-source LLMs as well, such as LLaMA. In our preliminary tests with the newly released Llama-2-13b-chat model, an open-source LLM, we are excited to find the conversations generated using our CO_3 framework also have great quality comparable to those in SODA. Furthermore, we would also like to note that our construction code for SODA will also be publicly available upon acceptance to encourage more diverse conversation distillations with various LLMs.\n\n**2. The primary contribution of our work is the step-by-step contextualization of commonsense knowledge graphs, rather than a naive straightforward distillation.**\nThe primary contribution of our work is not merely in distilling conversations. Rather, it is in using the commonsense knowledge graph to obtain a much broader spectrum of social conversations with commonsense by contextualizing it step-by-step. In Section 3.3, we show that naively distilling conversations from the LLM leads to lack of quality compared to our approach. For example, conversations in SODA outperform conversations distilled without contextualization, scoring double in terms of specificity and interestingness.\n\n**3. Comparing SODA’s scale and quality with other datasets:**\nCOSMO, which is trained on SODA, outperforms models trained on significantly larger conversation datasets (1.5M vs 1.5B) sourced from Reddit threads and web crawls. This indicates the contribution of SODA's quality to COSMO’s performance. Compared to crowd-sourced datasets, SODA contributes in both scale and quality. Therefore, SODA occupies a unique position by fulfilling both scale and quality criteria relative to existing datasets. An interesting potential future direction would be to explore experiments isolating the effects of scale and quality by training on smaller portions of SODA to see how much scale matters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They open sourced SODA a large dataset dialogue dataset. \n- SODA shows to be preferred by humans when compared to previous dialogue datasets. \n- COSMO, a model trained on SODA, outperforms SOTA models.", "reasons_to_reject": "- SODA is synthetic and generated with a close model.\n- Distilling GPT-3.5 models has shown to be efficient in many previous work.", "questions_to_authors": "- Do you think that your dataset helps more due to scale or due to quality when comparing with SOTA datasets?", "paper_topic_and_main_contributions": "This paper presents SODA, a new large scale synthetic social dialog dataset. In order to generate this dataset they start by verbalising triplets from a knowledge graph. These short sentences are then enriched by GPT-3.5 into a short narrative. Once having the enriched knowledge graph triplet, this is fed again into GPT-3.5 for creating a full conversation. After filtering the dataset they end up with 1.5M dialogues.\n\nThe dataset quality is evaluated by sampling 300 examples and comparing against human generated datasets taking 6 different into account. Overall they show that SODA is preferred by humans. \n\nUsing the SODA dataset they train the COSMO model that outperforms different SOTA models on the Daily Dialog dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uthQeGIMte", "reviewer": "Reviewer_H9QN", "summary": "", "questions": "- As according to Tables 4 and 12, SODA does seem to have more diversity in the types of emotions compared to different datasets. Yet, even in this dataset, the distribution is skewed towards particular emotions, where the top-4 emotions account for 40% of the entire dataset. What would your opinions be towards the idea of explicitly controlling for the types of emotions so that it could be more evenly distributed?\n- While you did provide a per-unit cost of creating the dataset, it would be interesting if you could also disclose the total cost of creating the dataset from scratch, that is the total sum of the GPT models used at different stages.\n- I wonder whether there is any potential bias in SODA due to the nature that the dataset is generated largely from GPT models. Specifically, whether the dataset is representative of different demographics, especially under-represented groups.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the value of SODA and COSMO to the NLP community! We also appreciate your acknowledgment of our contribution in using “knowledge graphs to generate diverse social interactions”!\n\n**1. Our effort to minimize potential biases and include diverse groups in SODA:**\nThank you for bringing up an important point! We also acknowledge that there can be biases related to the base LLM used for SODA. We prioritize filtering out harmful biases with a two-step process using a social norm based dialogue safety model and a safety-related API, removing about 4.5% of the conversations.\n\nWe also fully agree with you on the importance of representing under-represented groups. To achieve this in SODA, we use the top 1,000 names in the US social security number (SSN) applicants list, which covers 95% names in the list from 1990 to 2021, including names commonly associated with diverse genders (including non-binary and gender-neutral names) as well as various ethnicities. Developing methods for thoughtfully integrating diverse cultural commonsense knowledge in our framework will also be an important future direction, as naively prompting LLMs to generate texts related to minorities can result in stereotypes and parodies of them [1].\n\n\n**2. You can reproduce SODA with other choice of LLMs too!**\nAlthough the current version of SODA is generated with a closed LLM, our CO_3 method is generally applicable to any LLMs. Our initial tests with the newly released Llama-2-13b-chat model, an open-source LLM, we are excited to find the conversations generated with our CO_3 framework also have impressive quality comparable to those in SODA. We plan to release our CO_3 code to inspire further research and to enable the distillation of diverse conversations across various LLMs.\n\n\n**3. Explicitly controlling for the types of emotions:**\nThank you for a great suggestion! This is indeed possible using our method. Given that commonsense knowledge graphs also include emotional reaction knowledge, we can upsample triples with under-represented emotion and use our CO_3 method to generate more relevant conversations.\n\nAlso, it is worth noting that the reason those four emotions are predominant across all three datasets is because many common responses, such as \"How are you?\", \"That's great!\", and \"Thank you\", are categorized as curiosity, admiration, and gratitude.\n\n\n**4. The cost of constructing SODA.**\nThe original cost was less than 40,000 USD, which is significantly cheaper than the cost of crowdsourcing the same amount of conversation data. Moreover, the expense of using GPT-models has decreased significantly since we built SODA. We believe that now, the same volume of dialogue can be generated for tenth the original cost. We again kindly note that you can also apply our CO_3 method on open-source LLMs, such as LLaMA2, and obtain comparable conversations.\n\n\n\n[1] Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1504–1532, Toronto, Canada. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-motivated and well-presented.\n- The deliverables of this paper - the SODA dataset and the COSMO model - can be valuable to the social NLP community. Especially, I think the validation effort put into both deliverables should be appreciated.\n- This paper demonstrates a use case of LLMs and how they can be used to generate datasets that can be put to use. The combination of knowledge graphs with various prompting strategies shows that it is possible to generate a diverse set of social interactions that can be used to improve the social capabilities of existing LLMs.", "reasons_to_reject": "- One concern is that this dataset is generated by GPT across multiple stages, and the biases that the model is known to have may be inherent in the dataset throughout various forms.Also, the close nature of GPT will make it difficult to reproduce the results.", "questions_to_authors": "- As according to Tables 4 and 12, SODA does seem to have more diversity in the types of emotions compared to different datasets. Yet, even in this dataset, the distribution is skewed towards particular emotions, where the top-4 emotions account for 40% of the entire dataset. What would your opinions be towards the idea of explicitly controlling for the types of emotions so that it could be more evenly distributed?\n- While you did provide a per-unit cost of creating the dataset, it would be interesting if you could also disclose the total cost of creating the dataset from scratch, that is the total sum of the GPT models used at different stages.\n- I wonder whether there is any potential bias in SODA due to the nature that the dataset is generated largely from GPT models. Specifically, whether the dataset is representative of different demographics, especially under-represented groups.", "paper_topic_and_main_contributions": "In this paper the authors introduce the SODA dataset, a large-scale social dialog dataset consisting of various situations based on a knowledge graph containing social commonsense knowledge. The main idea of the dataset is to use GPT-3.5 combined with the knowledge graph to generate narratives, which is again put into GPT-3.5 to generate conversations. The authors show that the quality of the dataset surpasses existing conversation datasets in several aspects. Next, the authors fine-tune a conversation model named COSMO based on this dataset, developed, which surpasses existing conversation-based LLMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1PrIPSlSra", "reviewer": "Reviewer_Czz8", "summary": "", "questions": "- Q1: I did not see GPT 3.5 or ChatGPT evaluated in Table 5 Daily Dialog. The in-domain performance of GPT 3.5 on naturalness and consistency (Table 7) begs the question on Table 5, *\"Do you think COSMO scores are elevated in Table 5 by the underlying conversations generated from GPT 3.5 in SODA dataset?\"\n\n- Q2: Can we expect GPT 3.5 to perform comparably to COSMO in naturalness and consistency in Table 5 (from Table 7 performance) thus having a similar improvements over Blender, GODEL, Koala, Vicuna, and GT?* \n\n- Q3: I was pleased to see a listed limitation, namely, *\"SODA mainly focuses on social chitchat grounded on social commonsense, it lacks conversations ground in scientific knowledge or historical fact\"*. Grounding backstories is a core principle of the SODA dataset because of the CO_3 framework. Do you think the metrics selected (Natural, Consistent, Specific) favor SODA's characteristic of detailed grounding backstories? However, humans must frequently inquire about missing backstory information (not necessarily scientific or historical information). Thus, is SODA representative of human-bot or human-human conversations? This can affect the applicability of SODA.\n\n- Q4: By my calculations @ 1.5M dialogues keeping 68.9% *viz.* 2.2M dialogues @  \\\\$.02 per dialogue - without overhead -  costs puts SODA at approximately \\\\$43,500. Is this an accurate representation of the cost for performance/cost comparisons?\n\n- Q5: Why is the discrepancy between humans and GPT 3.5 so large in identifying if the dialogue contains the head event (88% vs 95%)? Do you think GPT 3.5 is over estimating its performance? How large is the holdout for this evaluation? Only 100 dialogues?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our effort for developing and evaluating SODA, as well as the impressive performance of COSMO on the out-of-domain dataset!\n\n**1. Evaluations with Amazon Mechanical Turk and the evaluation criteria:**\nGood point! We acknowledge that evaluations by crowdworkers tend to be noisier than those by domain experts, who typically excel in many existing NLP tasks. However, determining the naturalness and coherency of social chitchats can be also done relatively easily by lay people as they engage in chitchats everyday. Furthermore, we run separate qualification tests to recruit high quality workers, and ask them to justify their choice in free-form text to prevent hasty judgments. For selecting the evaluation criteria, we follow existing works in the machine dialogue field [1, 2, 3].\n\n**2. You can now easily build SODA with much less cost.**\nThe original cost for building SODA was under 40,000 USD, considerably less than the cost of crowdsourcing the same volume of conversation data. It is worth noting that the cost of using GPT-models has decreased significantly since we built the current version of SODA. The same amount of conversations can be generated for tenth the original cost. Moreover, our exciting preliminary tests with the newly released open-source Llama-2-13b-chat model show that we can obtain conversations comparable to those in SODA, by applying our CO_3 method.\n\n**3. Q1 - Do you think COSMO scores are elevated by the underlying conversations generated from GPT-3.5 in SODA?**\nYes, however, the choice of LLM is not strictly limited to GPT-3.5. We attribute COSMO's high performance to its training on conversations grounded in a broad array of specific contexts. Many existing dialogue datasets do not cover wide ranges as they are more narrowly focused on specific themes (e.g., persona, empathy), and they lack scale as they are manually crafted by crowd-workers. Training models on conversations generated by our CO_3 method with other recent LLMs (e.g., Llama2) can also yield impressive performance.\n\n**4. Q2 - Can we expect GPT 3.5 to perform comparably to COSMO in naturalness and consistency in Table 5, thus having similar improvements over Blender, GODEL, Koala, Vicuna, and GT?**\nYes, we expect GPT-3.5 to perform comparable to COSMO, and perform better than existing models [4].\n\n\n**5. Q3 - Do you think the selected metrics favor SODA's characteristic of detailed grounding backstories? Is SODA representative of human-bot or human-human conversations?**\nNo, the selected metrics do not specifically favor conversations being grounded in backstories, as they are metrics widely used in dialogue evaluation (e.g., consistency, coherency, specificity, and naturalness) [1, 2, 3]. We would also like to note that conversations in DailyDialog and BlendedSkillTalk are also grounded in specific backstories and context, respectively.\n\nSODA represents human-human conversations. However, it also includes conversations involving assistance (e.g., clerk, customer service), which can potentially help improve human-bot interactions in AI assistant models\n\n\n**6. Q4 - How much did it cost to construct SODA?**\nPlease see our response #2.\n\n\n**7. Q5 - What is the reason behind the discrepancy between humans and GPT 3.5 in validating the commonsense? How large is the holdout for this evaluation?**\nThe discrepancy comes from using the full commonsense triples vs. the head event in the commonsense triples. During the human validation, workers were asked to check whether the full commonsense triple is included in the conversation. We used 100 random dialogue samples for human validation. On the other hand, for validation using GPT-3.5, we filtered based on the head event only.\n\n\n[1] Sashank Santhanam and Samira Shaikh. 2019. Towards Best Experiment Design for Evaluating Dialogue System Output. In Proceedings of the 12th International Conference on Natural Language Generation, pages 88–94, Tokyo, Japan. Association for Computational Linguistics.\n\n[2] Mehri, Shikib, et al. \"Report from the NSF future directions workshop on automatic evaluation of dialog: Research directions and challenges.\" arXiv preprint arXiv:2203.10012 (2022).\n\n[3] Matthew Marge, João Miranda, Alan Black, and Alexander Rudnicky. 2010. Towards Improving the Naturalness of Social Conversations with Dialogue Systems. In Proceedings of the SIGDIAL 2010 Conference, pages 91–94, Tokyo, Japan. Association for Computational Linguistics.\n\n[4] Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. 2023. Controllable Mixed-Initiative Dialogue Generation through Prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 951–966, Toronto, Canada. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Large and costly effort for dialogues distilled from narratives and social commonsense.\n\n- Diverse dialogues, potentially helpful in a large number of chatbot applications.\n\n- Convincingly sound evaluations with AMT workers, the improvements for out of domain evaluation Table 5 was significant and noteworthy.\n\n- COSMO outperforms GPT 3.5 and ChatGPT and will likely serve as a baseline in future evaluations.\n\n- SODA dataset appears to be better in reported metrics by significant margin to DailyDialog and BlenderSkillTalk; similar open domain social dialogue datasets.", "reasons_to_reject": "- AMT workers are notoriously inferior to in-domain participants, e.g. computational linguist graduate students, thus ill-suited for any rigorous evaluation. An evaluation with domain aware participants and a criteria that does not favor a narrative based approach might yield closer results for Table 5. To be clear I do not think the participant or metric selection neutralizes the demonstrated improvements in any significant way.\n\n- The cost is enormous, reproducing this effort will be difficult despite the straightforward methodology; moreover GPT 3.5 / ChatGPT is ever evolving meaning the same dataset will not be reproducible even months from acceptance.", "questions_to_authors": "- Q1: I did not see GPT 3.5 or ChatGPT evaluated in Table 5 Daily Dialog. The in-domain performance of GPT 3.5 on naturalness and consistency (Table 7) begs the question on Table 5, *\"Do you think COSMO scores are elevated in Table 5 by the underlying conversations generated from GPT 3.5 in SODA dataset?\"\n\n- Q2: Can we expect GPT 3.5 to perform comparably to COSMO in naturalness and consistency in Table 5 (from Table 7 performance) thus having a similar improvements over Blender, GODEL, Koala, Vicuna, and GT?* \n\n- Q3: I was pleased to see a listed limitation, namely, *\"SODA mainly focuses on social chitchat grounded on social commonsense, it lacks conversations ground in scientific knowledge or historical fact\"*. Grounding backstories is a core principle of the SODA dataset because of the CO_3 framework. Do you think the metrics selected (Natural, Consistent, Specific) favor SODA's characteristic of detailed grounding backstories? However, humans must frequently inquire about missing backstory information (not necessarily scientific or historical information). Thus, is SODA representative of human-bot or human-human conversations? This can affect the applicability of SODA.\n\n- Q4: By my calculations @ 1.5M dialogues keeping 68.9% *viz.* 2.2M dialogues @  \\\\$.02 per dialogue - without overhead -  costs puts SODA at approximately \\\\$43,500. Is this an accurate representation of the cost for performance/cost comparisons?\n\n- Q5: Why is the discrepancy between humans and GPT 3.5 so large in identifying if the dialogue contains the head event (88% vs 95%)? Do you think GPT 3.5 is over estimating its performance? How large is the holdout for this evaluation? Only 100 dialogues?", "paper_topic_and_main_contributions": "*SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization* consists of three major contributions: SODA, CO_3, and COSMO. SODA is a million scale social dialogue dataset rooted in the narrative of social common sense knowledge triplets, CO_3 is the pipeline for contextualizing the commonsense triplets, and COSMO is the resulting conversational model trained on the SODA dataset. The dataset outranks Daily Dialog and BlendedSkillTalk in natural flow, context dependence, topic and speaker consistency, and specificity per AMT workers. SODA manages to include a diverse range of content including emotion related information despite coming from common sense triplets. Even though SODA uses GPT 3.5 as its teacher, the authors demonstrate that the dialogs from SODA (using the CO_3 framework), outperforms the teacher network (without context) GPT 3.5. The authors train COSMO with is a LM-adapted T5 and results in better performance at a fraction of the price of GPT 3.5  (3B/11B parameters vs 175B parameters). The authors evaluate COSMO on an out of domain dataset (Daily Dialog) and comparable models Blender, GODEL, Koala, Vicuna, and Human gold values. COSMO significantly outperforms all models across naturalness, consistency, and specificity with the largest difference found in naturalness (87% over Blender 23%, GODEL 13%, Koala 30%, Vicuna 42%, and even Human Gold 43%). They pitted COSMO against BlenderBot on the BlendedSkillTalk dataset, a training set BlenderBot is trained on, and outperform BlenderBot significantly across the same metrics. Finally, the in domain setting is tested by evaluating COSMO, GPT 3.5 and ChatGPT on SODA; the SODA trained COSMO performs best. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "SV4jY75sZG", "reviewer": "Reviewer_1GY1", "summary": "", "questions": "1. Have you tried to sample a set of relevant triplets from ATOMIC to generate a round of dialogue?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty of our approach in combining commonsense knowledge graphs with LLMs and its potential applications across various domains in the field! We also appreciate the acknowledgment of COSMO's performance.\n\n**1. The topical diversity and complexity of SODA:**\nWe fully agree that this is an important point! The reason why we use step-by-step contextualization in our CO_3 framework is to tackle this exact issue. We observed the lack of specificity and diversity when we naively distill conversations directly from LLMs, as shown in Figure 3. Thanks to the contextualization steps (triple → short narrative → conversation), we find a greater diversity of commonsense knowledge and improved specificity being incorporated with each step. Additionally, we use Atomic10x instead of the original Atomic, which is significantly larger than Atomic. As a result, we find conversations in SODA exhibit a much higher lexical diversity compared to existing datasets (Table 2).\n\n**2. Emotion related experiments with COSMO:**\nThank you for the nice suggestion! Although we did not directly test the emotion-related capabilities of COSMO, we would like to emphasize that DailyDialog and BlendedSkillTalk datasets both include emotion-annotated contexts with 6 and 32 types of emotion, respectively. The fact that COSMO outperforms existing models and even the ground truth response suggests that it can well adapt to the context’s emotion. We will add this discussion in the updated draft.\n\n**3. Sampling a set of relevant triplets from ATOMIC to generate a round of dialogues:**\nThank you for the thoughtful suggestion! In our early experiments, we found it was difficult to automatically sample multiple commonsense triples that lead to natural narratives and conversations. However, we agree that this is an interesting future direction.\n\n**4. Missing references.**\nThank you, we will include these in the updated draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of combining large language models with commonsense knowledge graph to do data synthesis sounds novel and useful. While this paper validates its effectiveness in dialogue dataset synthesis, this concept could also be applied in other areas, like table-to-text generation, to build high-quality and large-scale datasets.\n2. The data synthesis pipeline in this paper is well-built and thoughtful. They consider various aspects including quality, safety, naturalness and consistency. Evaluation and statistics of their collected dataset are also comprehensive.\n3. COSMO trained with SODA demonstrates promising performance. It outperforms the previous SOTA models of dialogue generation in different evaluation scenarios, with an even smaller size. It also shows comparable performance to GPT3.5 and ChatGPT, which contain much more parameters than it.\n4. They will open-source the dataset and the model, and it would be useful for the community.", "reasons_to_reject": "1. The major issue I am worried about is the complexity of SODA. Here I use complexity to refer to the complexity of the core event in each dialogue. This is because SODA is built by contextualizing and rewriting exactly one triplet into a multi-turn dialogue. However, each dialogue in people’s everyday life could involve more than one topic and thus contains more complex dialogue flows. Also, some triplets in ATOMIC are very simple and vague, which may further reduce the topical variety and complexity of SODA.\n2. The authors highlight the rich motion-related information contained in SODA in section 3.2. But I don’t see any emotion-related evaluation in their experiment section to evaluate COSMO. Actually, the three metrics they used (natural, consistent and specific) are a little general and thus may miss some aspects like emotion and empathy.", "questions_to_authors": "1. Have you tried to sample a set of relevant triplets from ATOMIC to generate a round of dialogue?", "paper_topic_and_main_contributions": "In this paper, they propose a data synthesis method to distill dialogue using a large language model and commonsense knowledge graph. They do so to overcome data scarcity issues in the open-domain dialogue field. \n\nSpecifically, they follow certain templates to convert triplets to sentences first. Then, they prompt GPT-3.5, a large language model, to generate a paragraph of narrative according to the sentence. With the generated narratives, they prompt GPT-3.5 again to produce multi-turn dialogue. After obtaining the synthesis corpus, they conduct a series of post-processing to guarantee the quality of the dataset and remove harmful content, including rule-based filtering, safety filtering and commonsense filtering. The final dataset, SODA, contains about 1.5 million dialogues. Several evaluations are done subsequently to evaluate the quality, scale, diversity and emotional richness of their proposed dataset. They also conduct an ablation study to validate the effectiveness of their contextualization process in the synthesis pipeline.\n\nTo utilize the collected dataset, they train a new conversation model called COSMO based on it. They adopt human evaluation to compare COSMO with other chatbot and find COSMO outperform its competitors in various testing settings with smaller model size.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Xiang J, Liu Z, Zhou Y, et al. ASDOT: Any-shot data-to-text generation with pretrained language models[J]. arXiv preprint arXiv:2210.04325, 2022.\n\n[2] Li D, Li Y, Zhang J, et al. C3kg: A chinese commonsense conversation knowledge graph[J]. arXiv preprint arXiv:2204.02549, 2022."}}, {"review_id": "4X3NNKdSUh", "reviewer": "Reviewer_FFJY", "summary": "", "questions": "1. Is grounded information (e.g., persona and knowledge in BST) leveraged in evaluations when comparing COSMO with other models?\n\n2. Why does COSMO-11B surpass its teacher model InstructGPT and ChatGPT in several dimensions? What if these LLMs are prompted in the CO3 framework to generate responses?\n\n3. How are the dialogue contexts used for evaluation (in Section 5.1, 5.2, 5.3) sampled?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your appreciation of the effectiveness of leveraging commonsense knowledge graphs!\n\n**1. Automatic evaluation results using GPT-4:**\nThank you for the suggestion! We primarily relied on human evaluation due to the known limitations of automatic evaluation, even when using GPT models [1]. However, in response to your suggestion, we ran automatic evaluation on the overall quality of responses with GPT-4, under the same evaluation setup in Table 5 and 6. We find the results align closely with those from our human evaluation. \n\n|      DailyDialog      |    Overall   |\n|-----------------------|:------------:|\n|     COSMO vs GODEL    |  **93%** vs 7%   |\n|  COSMO vs BlenderBot  |  **68%** vs 32%  |\n|    COSMO vs Koala     |  **65%** vs 35%  |\n|    COSMO vs Vicuna    |  **54%** vs 46%  |\n|    COSMO vs GT        |  **52%** vs 48%  |\n\n\n|    BlendedSkillTalk   |    Overall   |\n|-----------------------|:------------:|\n|  COSMO vs BlenderBot  |  **66%** vs 34%  |\n\n|         SODA          |    Overall   |\n|-----------------------|:------------:|\n|  COSMO vs BlenderBot  |  **85%** vs 15%  |\n\nIt should be noted that GPT-4 tends to favor GPT-generated texts over those written by humans, even when human judges show a preference for the latter [1]. As a result, these scores are likely to be biased towards COSMO, which is trained on texts generated by GPT-3.5. Therefore, our original human evaluation results should be given much more weight when assessing the overall quality of the model, where COSMO also outperforms other models.\n\n\n**2. Statistics of the human evaluations and definition of ‘naturalness’:**\nThank you for the good points! A total of 74 workers participated in comparing dialogues, yielding a Krippendorf’s alpha of 0.25. For response comparisons, 77 workers participated with a Krippendorf’s alpha of 0.5. These results indicate fair to good agreements on the quality judgments. \n\nWe will clarify the definition of the ‘naturalness’ aspect, which was used to determine which sounds more like a natural ‘human’ conversation or response [2]. We analyzed the workers’ responses for the rationales behind their judgment and also found that many of them mentioned human-like naturalness in their responses. For example, “(option) A seems more natural and like the conversation is just a real one between two people.”, “B feels more real to me like a conversation between real people.”, “A is more natural, whereas B sounds mechanical”, and “B is a more human and natural reply.” For reference, the Krippendorf’s alpha of ‘naturalness’ for the comparison of dialogues and responses, was 0.23 and 0.49, respectively, also indicating fair to good agreements across workers.\n\n\n**3. Training setup for COSMO and safety:**\nWe included the ProsocialDialog to improve COSMO’s robustness to unsafe dialogue contexts in anticipation of public release. We ran a zero-shot experiment with COSMO on ToxiChat following the original ProsocialDialog work. We find COSMO achieves comparable performance with their original model Prost and InstructGPT, in terms of disagreeing with toxic dialogue input and avoiding generating offensive responses or toxic words: disagree (10.3), agree (7.5), offense (5.8), and bad (5.1).\n\nFurthermore, considering that ProsocialDialog constitutes only 3.7% of COSMO’s training data and our current evaluations do not explicitly include unethical dialogues, its impact on our existing evaluations is likely negligible. The original ProsocialDialog work also reports that their dataset is orthogonal to other datasets.\n\n**4. Comparison of \"head-to-head\" vs. \"independently-evaluated scores\" when conversation topics vary:**\nWe appreciate your suggestion! We also debated the same question when conducting the evaluations in our paper. Unfortunately, both evaluation methods have limitations when the conversation topics are different. We acknowledge that comparing independently-evaluated scores may seem to provide a more isolated assessment of each dialogues, but this method can also be noisy and unreliable. As there is no direct comparison, the evaluators can hold different standards or expectations for different topics, making it complicated to compare them directly [3, 4, 5]. Given that our primary object was to directly compare the dialogue quality, we opted for a head-to-head comparison. We would like to note that we also ask workers to provide rationales for their judgments to minimize hasty comparisons during evaluation.\n\n\n**5. Is grounded information leveraged in evaluations when comparing COSMO with other models?**\nYes, we provide the persona information to BlenderBot when conducting comparison for BlendedSkillTalk (Table 6), but not the ground-truth external knowledge sentences. As COSMO is not trained using the BlendedSkillTalk-formatted information, we only provide the dialogue history to COSMO. Since DailyDialog does not provide grounding information, we only feed the dialogue history to the models for the experiments in Table 5.\n\n\n**6. Why does COSMO-11B surpass its teacher model InstructGPT and ChatGPT in several dimensions? What if these LLMs are prompted in the CO3 framework to generate responses?**\nThis is because COSMO is specifically trained on specific natural dialogues, yet ChatGPT etc. are general-purpose LMs. While CO_3 is primarily a data generation framework rather than a prompting method for response generation, those LLMs should perform as well as (or even better than) COSMO when given the same prompt containing all the information from the CO_3 framework, such as the ground-truth narrative and speakers.\n\n\n**7. How are the dialogue contexts used for evaluation (in Section 5.1, 5.2, 5.3) sampled?**\nWe randomly sample 100 contexts from the test set.\n\n[1] Liu, Yang, et al. \"G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment\" arXiv preprint arXiv:2303.16634 (2023).\n\n[2] Matthew Marge, João Miranda, Alan Black, and Alexander Rudnicky. 2010. Towards Improving the Naturalness of Social Conversations with Dialogue Systems. In Proceedings of the SIGDIAL 2010 Conference, pages 91–94, Tokyo, Japan. Association for Computational Linguistics.\n\n[3] Liang, Weixin, James Zou, and Zhou Yu. \"Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\n[4] Anja Belz and Eric Kow. 2010. Comparing Rating Scales and Preference Judgements in Language Evaluation. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics.\n\n[5] Sashank Santhanam and Samira Shaikh. 2019. Towards Best Experiment Design for Evaluating Dialogue System Output. In Proceedings of the 12th International Conference on Natural Language Generation, pages 88–94, Tokyo, Japan. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The scarcity of large-scale open-domain dialogue has been a long-standing problem. Recently, generating high-quality data by prompting LLMs has been widely adopted in various domains (e.g., knowledge graph, instruction-following data, tool-use data, emotional support dialogue data). This paper extends this framework to the task of open-domain dialogue.\n\n2. This paper leverages commonsense knowledge triples to prompt LLMs. Experiment results in Section 3.3 verify the effectiveness of this design.", "reasons_to_reject": "1. Lack of automatic evaluation results. I understand the difficulty of automatically evaluating open-domain dialogue models with previous metrics (e.g., BLEU, ROUGE, BERTScore).  I would recommend conducting evaluations using ChatGPT or GPT-4, which has been widely adopted in recent works, including open-domain NLG.\n\n2. Lack of human evaluation details. While this paper only consists of human evaluations, it lacks lots of human evaluation details. For example, how many annotators are recruited to annotate per instance, and what is the inter-annotator agreement. Additionally, based on the annotation page shown in Figure 4 and Figure 5, some evaluation dimensions are not well-defined. For example, naturalness, which is claimed as the key difference between COSMO and other baseline models, is simply illustrated as \"Which dialog has a more natural flow\" or \"Which response is more natural?\". I would recommend further clarifying these definitions, not only to researchers but also to annotators.\n\n3. The training setting for COSMO is confusing. ProsocialDialogue is augmented into the original SODA dataset, which is claimed to assist models in handling sensitive contexts. However, the safety of COSMO is not evaluated in experiments. I recommend conducting evaluations on the model solely trained on SODA.\n\n4. The evaluation setting between SODA and the previous dataset is unreasonable. Considering they do not share the same prefix or the same topic, it is hard to tell which one is better. Therefore, I recommend independently conducting evaluations on each dataset instead of pair-wise comparisons.", "questions_to_authors": "1. Is grounded information (e.g., persona and knowledge in BST) leveraged in evaluations when comparing COSMO with other models?\n\n2. Why does COSMO-11B surpass its teacher model InstructGPT and ChatGPT in several dimensions? What if these LLMs are prompted in the CO3 framework to generate responses?\n\n3. How are the dialogue contexts used for evaluation (in Section 5.1, 5.2, 5.3) sampled?", "paper_topic_and_main_contributions": "This paper proposes CO3, a framework for automatically generating high-quality conversations based on LLMs (text-davinci-002 in this paper). Specifically, CO3 first sample commonsense knowledge triples from a knowledge graph (Atomic) and then uses the sampled commonsense knowledge to guide conversation generation. Based on CO3, this paper constructs SODA, a million-scale commonsense grounded English dialogue dataset. Furthermore, the authors use SODA to train a dialogue model COSMO. Experiment results demonstrate the effectiveness of the automatically constructed dialogue data.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents SODA, a framework for generating social chit-chat dialogue data using pre-trained LLMs (GPT3.5 in this work). This framework is used to produce a 1.5M dialogue dataset and train a T5-based LLM (COSMO) which is evaluated. Both the code and model will be released as open-source. All reviewers appreciated the need for such a framework and the fact that the authors are willing to open-source both data and code. They also liked the fact that the authors used commonsense triplets for dialogue generation. Most concerns centered around cost and bias, given that GPT3.5 was used for data generation, and one reviewer pointed out lack of details and evaluations. All concerns, however, were properly addressed by the authors and I am happy to accept this paper.
Decision: Accept-Main