Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents
Authors: Hyungjoo Chae
Yongho Song
Kai Tzu-iunn Ong
Taeyoon Kwon
Minjin Kim
Youngjae Yu
Dongha Lee
Dongyeop Kang
Jinyoung Yeo
Venue: EMNLP 2023 Main (2023)
Keywords: Commonsense Reasoning
Dialogue
Large Language Model

Abstract: Human-like chatbots necessitate the use of commonsense reasoning in order to effectively comprehend and respond to implicit information present within conversations. Achieving such coherence and informativeness in responses, however, is a non-trivial task. Even for large language models (LLMs), the task of identifying and aggregating key evidence within a single hop presents a substantial challenge. This complexity arises because such evidence is scattered across multiple turns in a conversation, thus necessitating integration over multiple hops. Hence, our focus is to facilitate such multi-hop reasoning over a dialogue context, namely dialogue chain-of-thought (CoT) reasoning. 
To this end, we propose a knowledge distillation framework that leverages LLMs as unreliable teachers and selectively distills consistent and helpful rationales via alignment filters. 
We further present DOCTOR, a DialOgue Chain-of-ThOught Reasoner that provides reliable CoT rationales for response generation.  
We conduct extensive experiments to show that enhancing dialogue agents with high-quality rationales from DOCTOR significantly improves the quality of their responses.

Dialogue Chain-of-Thought Distillation for Commonsense-aware
Conversational Agents
Hyungjoo Chae Taeyoon Kwon'
Yongho Song
Kai Tzu-iunn Ong'
Minjin Kim²
Youngjae Yu'
Dongha Lee"
Dongyeop Kang²
Jinyoung Yeo!
University of Minnesota (mapoout, _hs, yjy, donalee, jinyeo]@yonsei. .cc
Yonsei University'
dongyeopêumn. edu
Abstract
I've lest my handbag. Or perhaps someane's taken it.
Oh, no! Was there anything important it? My checkbook and all the papers need for
Human-like chatbots necessitate the use of com- monsense reasoning in order to effectively com-
+
Oh, that's terrible. But how did you it?
prehend and respond to implicit information present within conversations. Achieving such coherence and informativeness responses, however, a non-trivial task. Even large language models (LLMs), the task of iden- tifying and aggregating key evidence within a single hop presents substantial challenge.
QU spent the morning with friend and then went shopping after lunch. could't find my checkbook
purchase
and realized it was in my handbag, which was in my car.
you
your car.
But didn't find it there.
And you went the police station right?
Not immediately. Baforal went to the police, called friend's office. But no luck.
This complexity arises because such evidence is scattered across multiple turns in a conversa- tion, thus necessitating integration over multi- ple hops. Hence, our focus is to facilitate such multi-hop reasoning over dialogue context, namely dialogue chain-of -othouhtt (CoT) soning, To this end, we propose knowledge distillation framework that leverages LLMs as unreliable teachers and selectively distills con- sistent and helpful rationales via alignment fil- ters. We further present DOCTOR, DialOgue Chain-of- Chain-of-ThOught Reasoner that provides re- liable CoT rationales for response generation'. We conduct extensive experiments to show that enhancing dialogue agents with high-quality ra- tionales from DOCTOR significantly improves the quality of their responses".
Single-hop Reasoning
Multi-hop Reasoning (Ours)
A needs find her hand bag. which contains important Items.
COMET
wants to affer some help.
PersonX call the police.
O should go to the restaurant
where she had lunch today and see if her handbag there.
Generated Response Call the police. X
You should go to where you
had lunch and look for it.
Figure 1: Comparison between responses generated via
single-hop reasoning and multi-hop reasoning.
they still suffer from limited or incorrect reason- ing that leads dull and incoherent responses, shown on the left in Figure
The problem lies in that commonsense reason- ing in a conversation involves multi-hop reasoning. Since key evidence is scattered across and implic- itly stated in multiple dialogue turns (Liu et al., 2021a la; Zhao al., 2022), it is challenging cap- ture all necessary information in single hop. For example, the response on the right in Figure (e.g.. "You should go to ..."" can only be obtained by in- tegrating multiple implicit evidence (e.g., E1, E2, and E3) from the dialogue. The process of find- ing and aggregating such scattered evidence takes multiple steps, highlighting the need for multi-hop reasoning for coherent and informative responses. Inspired by the success of Large Language Mod- els (L.LMs) (Brown al., 2020), we formulate multi-hop commonsense reasoning as Chain-off Thought (CoT) reasoning (Wei et al., 2022) for dia-
1 Introduction
Commonsense reasoning is crucial in human con- versation (Richardson and Heck, 2023). However, most conversational agents still lack commonsense reasoning, limiting their capability to engage in rich conversations with users (Arabshahi et al., 2021). Recent studies (Gao al., 2022; Zhou et al., 2022a.b) aim to tackle this issue by generating com- monsense knowledge (Hwang al., 2021; Bosselut et al., 2019) relevant to the dialogue context, but
"Equal contribution
'We release our source code on https://gitth:h com/ kyle8581 1yle@581/DialogueCon
"We release demonstrations of dialogue CoT reasoning in web. app/.

---

logue response generation, namely Dialogue COT. Our goal to facilitate dialogue CoT reasoning by enabling language models to decompose common- sense reasoning into multiple steps and generate rationale as a sequence of inferred commonsense knowledge required for response generation.
In these approaches, commonsense knowledge Z either retrieved from symbolic knowledge bases (KBs) (Zhou et al., 2022b; Gao et al., 2022) such as ATOMIC (Hwang et al., 2021), 1), or generated from neural KBs such as COMET (Bosselut et al., 2019). These methods, however, tend to miss subtle yet implicit details in a conversation (Shwartz et al., 2020; Schlegel et al., 2022), leading to dull and incoherent responses. We posit that commonsense reasoning conversation requires multiple hops to capture such implicit details scattered across multiple turns (Liu et al., 2021b; Zhao et al., 2022).
Despite their potential effectiveness, we observe two limitations of prompting LLMs for common- sense reasoning in conversations: (1) LLMs tend rely much on explicit cues, necessitating task- specific constraints to seek implicit knowledge. (2) LLMs exhibit poor alignment between rationales and dialogues, resulting in inconsistent and unhelp- ful rationales. These challenges motivate the need for robust symbolic distillation mechanism (West et al., 2022; Kim et al., 2022a) that selectively trans- fers CoT capabilities from LLMs train a reliable CoT reasoner.
2.2
Formulating Chain-of-Thought Reasoning in Dialogues
Inspired by the success of rationale augnnnle- LLMs on multiple reasoning tasks (Wei et al., 2022; Zhou et al., 2023), we formulate multi-hop rea- soning in conversation dialogue CoT reasoning, which decomposes reasoning into multiple steps and combines inferred commonsense knowledge into rationale that supports response generation. With dialogue CoT reasoning, dialogue agents can generate coherent responses by identifying relevant contextual cues in a dialogue and making use of implicit information underlying the context.
Our contributions are threefold: (1) We propose a dialogue chain chain-of-thougee thought distillation framework that extracts plausible rationales from unreliable LLMs and collects high-quality CoT rationales via iterative question answering and alignment fil- tering. (2) Using our framework, collect DONUT, dialogue dataset annotated with high- quality CoT rationales. Our qualitative analysis of the collected rationales shows the effectiveness of our method for controlling the reliability of LLMs in extracting rationales. (3) With DONUT, we train DOCTOR, a DialOgue Chain-of- Thought cOmmonsense Reasoner that integrates implicit in- formation in dialogue into rationale for generating responses². We conduct experiments on response generation tasks show that augmenting dialogue models with high-quality rationales from DOC- TOR significantly improves their performance.
A naive approach to facilitate dialogue CoT rea- soning is apply CoT prompting LLMs. How- ever, we find this approach is suboptimal due to the following limitations: (1) LLMs attend more to ex- plicit cues (e.g. lexical overlap) in dialogues for rea- soning, requiring task-specific constraints to guide the model to infer implicit information; (2) The rationales are often misaligned with the dialogues, i.e., inconsistent with the contexts (Peng et al., 2023) or unhelpful in response generation (Jung et al., 2022). Based on these insights, we aim construct a reliable CoT reasoner that generates high-quality rationales for dialogue agents.
2 Dialogue Chain-of-Thouge Reasoning
2.1 Preliminaries
Recent work (Wu et al., 2020; Zhou et al., 2022a,b; Gao et al., 2022; Zhong et al., 2021) aims to enrich dialogue modeling by augmenting dialogue agents with commonsense knowledge to infer implicit in- formation in conversations. Specifically, a dialogue model 0 given commonsense knowledge Z as an additional input to predict the next response u, for the dialogue context Usr of t - turns:
3 Dialogue Chain-of-Thought Distillation
In this section, we propose robust knowledge distillation framework that extracts plausible CoT rationales from unreliable LLM ($3.1) and selec- tively distills high -uality rationales via alignment filters ($3.2) to (re- )raain a reliable CoT reasoner. Figure 2 presents an overview of our framework.
uy ~ Po
(1) 3.1 QA-driven Rationalization
Our framework is designed to augment existing large-scale dialogue corpora with dialogue CoT
*Wo release the model checkpoint on https:// huggingface.co// Co.DLI-Lab/DOCTOR.

---

QA-driven Rationalization
Alignment Filtering
Training Dialogue CoT Reasoner
02. What's B's reaction to
Rationale-to toe-too--ooettteee Alignment
We're having our term. wish you good luck.
A2. expresses concern...
Q3. What advice might might suggest to to the restaurant they hac
Thanks! But I'm little nervous. Why? You used to study well.
I've my handbag-...
Consistent
DOCTOR
Inconsistent
Reliable reasoner
U1 Us
01
Why is A little nervous?
2. Rationale-to iettonaleto-resporee resposs Alignment
Q2
A2
Because they have an exam.
Ut-1
Us
What's E's opinion of A's study
uE
Q3 A3
used to study well.
Why nervous despite their
Source Dialogue
Oversampled
Call the police.
Holpfistness
DONUT
previous study efforts?
Rationales
You should to...
Dataset
-...
due the passage of time.
Figure 2: Overview of our framework. We leverage LLM to collect CoT rationales and apply filters to selectively annotate them. The same dialogue from Figure I is used to showcase rationale generation (left) and alignment filtering (middle). The dotted square shows the training of the critic model with counterfactual rationales.
rationales by leveraging the capability of LLMs to rationalize. We first prompt an LLM to generate a plausible rationale 7" for a dialogue context Uct and the ground-truth response u, such that the next response uy induced from the rationale Z":
text (Peng et al., 2023), which can often lead to rationales that are misaligned with the dialogue context. Inspired by West et al. (2022), we mini- mize such inconsistent rationales from LLMs by employing critic model detect counterfactual rationales generated without correctly grounding on the dialogue context+. We ask the LLM to gener- ate a counterfactual rationale Ž from a counterfac- tual context Üet, containing only the last utterance:
z argmax PLLM (Z)ee, Uct)
(2)
Specifically, represent rationale Z with se- quence of k question- pairs {(q; =-1- where qi is an iifformmtioonsseeiiie question about implicit information a, in Uei By instructing an LLM to iteratively generate questions and answers, we ask the model to pinpoint relevant contextual cues and infer underlying knowledge that supports response generation.
Ż = argmax PLLM (u Üet)
(3)
The critic model is trained to distinguish between Z" and 7 for given dialogue contexts. We sample 6K dialogues from SODA (Kim et al., 2022a) and collect 6K (U<1, Z²) pairs by manually choosing consistent Z* from the set of generated rationales. We then construct 6K (U<r, Z) pairs for the col- lected samples, resulting in 5k training instances of (U<t, Z*) and (Ü<t, Z) pairs for our critic model.
In practice, we choose set of commonsense re- lations from ATOMIC (Hwang et al., 2021) that are commonly used in dialogue domains. We prompt LLMs to construct questions q based on the rela- tion type to guide the model to construct questions pertaining to conversations. We further include demonstrations of dialogue CoT, each of which con- tains human- -athored question-anse answer pairs for dialogue U = [U<x u.]. We present the list of com- monsense relations along with the example prompt used for rationale collection in Appendix A.
Rationale-to-respon alignment. We consider rationale to be aligned with response if aug- menting dialogue models with the rationale helps predicting the ground-truth response. Hence, we introduce an indicator function helpful(-) to de- termine a dialogue model benefits from a ratio- nale Z when predicting the ground-truth response uy given a context Uzr Formally, we define a boolean function helpful (.) as:
3.2 Alignment Filtering
To ensure the quality of the annotated rationales, we further introduce two alignment filters that filter out rationales based on their alignment with the dialogue contexts and the ground-truth responses.
Po Po Pu
helpful(Z)
> >T]
(4)
"We implement the critic model with RoBERTa-large (Liu al.. 2019). See Appendix A.3 more details.
Rationale-to-context alignment. LLMs tend to hallucinate facts without attending to the con-
*We use Cosmo-3B (Kim et al., 2022a) trained on a large- scale dialogue dataset covering diverse social interactions.

---

where 1[:] is binary indicator and is hy- perparameter'. Intuitively, higher probability Po(ue/Z, Z Uct) indicates that the rationale is more helpful in predicting the response ut.-
Dialogue Context:
A: Hi, Viggo. How are you doing today?
B: Hey, Yovani. I'm doing right. Thanks for asking.
A: No problem. saw that you left your coffee mug on the counter this morning. Did you forget take it with you?
3.3 Training DOCTOR
B: Yeah, did, Thanks for grabbing it for me.
A: No problem at all. I know how busy you are and didn't
Using the annotated dialogue corpus, we train a DialOgue Chain-of-ThOught Reasoner, namely DOCTOR. We train our model with a causal lan- guage modeling objective to predict the probability of generating the rationale Z* given the dialogue history U'ci- Essentially, the training objective can be formulated as next-token prediction over se- quence of question-answer pairs (g: a;) in a ratio- nale, where the model iteratively predicts qi and Og following previously generated -nnswr pairs. We posit that by learning to generate and answer subquestions, the model can identify all im- plicit information required to infer the correspond- ing commonsense knowledge from the dialogue.
want you to have to come back for later.
B: You're re a lifesaver, Yovani. Seriously, thank you so much.
Dialogue Chain-of- Thoughh Rationale:
Q1: What did Person A do for Person B? (oReact)
A1: Person A grabbed Person B's coffee mug for him when he forgot it.
Q2: What is Person B's reaction to Person A's help? (xRe- act)
A2: Person B is thankful and expresses gratitude to Person A for helping him out.
Q3: What might Person want to convey Person B, based their previous interactions? (xIntent)
A3: Based on their previous interactions, Person A might
want convey that he understands how important coffee is Person and that he always willing to help him out.
DOCTOR is built on top of OPT-1.3B (Zhang et al., 2022) and is trained using 80% of the anno- tated data with a constant learning rate of 5e-4 for 5 epochs. See Appendix A for details.
Ground-truth Response:
A: Any time. know how much you love your coffee in the morning.
Table 1: A sample from DONUT
4 DONUT
Description
DONUT
CICERO
Roffcet-9K
ComFact
# Dialouges
1OK
5.6K
600
769
Alongside DOCTOR, we present its training cor- pus, DONUT, a DialOgue chaiN-of-thhoUhh d with annotated rationales for dialogue COT reasoning?. We choose three human-collected dialogue datasets, DailyDialog (Li et al., 2017), DREAM (Sun et al., 2019), and MuTual (Cui et al., 2020), to sample source dialogues for an- notation We also include 5% of the dialogues in SODA (Kim al., 2022a), million-scale social dialogue dataset, for scalability. In total, we obtain 10K dialogues for annotation. For each utterance in a dialogue (except for the first one), we instruct ChatGPT to generate 10 rationale candidates.
A Tums with Inf.
45K
28K
600
6K
A Inferences
367K
S3K
52K
Avg. words per Inf.
78.6
12.0
3.4
Table 2: Statistics of DONUT vs. human- authored dia- logue datasets with annotated commonsense knowledge.
DONUT. Further analyses on the alignment filter- ing and generated rationales are in Appendix D.
4.1 DONUT vs. Human-annotated Datasets
Here we summarize the advantages of DONUT over three dialogue datasets: CICERO (Ghosal et al., 2022a), Reflect-9K (Zhou et al., 2022a), and ComFact (Gao et al., 2022). These datasets pro- vide high-quality human-annotated commonsense knowledge for dialogues.
Using the two aligmment filters from $3.2, we fil- ter out 122,319 candidates (24.98%) that are either inconsistent with the dialogue context or not help- ful in predicting the next response. The resulting dataset, DONUT, consists of 10K dialogues with 367K COT rationales. Table shows sample from
Large scale. As shown in Table 2, DONUT con- tains a larger amount of annotated dialogue sam- ples compared to existing dialogue datasets with human-annotated commonsense knowledge.
"We use 0.95 for We provide the distribution of the ratio in Appendix D.I.
"DONUT is available on https://hugginngaaceu co/ Batasets/OLLLLLab/DONN.. *Wo use subset of dialogue samples from the three datascts as curated by Ghosal et al. (2022a).
Cost & time-efficiency. Unlike human-authored datasets, DONUT is automatically annotated via ChatGPT in a time and cost-efficient manner, With

---

Win Percentage (86)
DONUT CICERO
DONUT
Reflect
60
Percantage
66
40
40
20
20
Falthlnness
Helpfuiness
Relevance
Specificity
Falthfulness
Specifiityy
Overall
Figure 3: Results of head-to-head comparison between rationales from DONUT, commonsense annotation from CICERO (Ghosal et al., 2022a), and Reflect (Zhou et al., 2022a) via human judgment. The y y-axis represents the win
percentage against other datasets. The differences in all of the categories are statistically significant 0.05).
ChatGPT, the annotation process takes 0.1 seconds per sample and costs 0.003 USD with total of 1,200 USD. This significantly reduces the time and cost for data annotation.
R-to-Context
R-to-Response
Win Percentage
Pass
vs.
Fail
vs.
Fail
Consistency Helpfulness Specificity Overall 60%
71%
29
59%*
41%
53%
47%
74%
26 %6%
64%%
36%*
63%
37%:
High quality. We conduct human evaluation via Amazon Mechanical Turk to compare the qual- ity of CoT rationales from DONUT with Reflect- 9K and CICERO. At each voting stage, three hu- man judges are given two dialogues, one from DONUT and one from CICERO or Reflect-9K, and asked to choose a sample with better commonsense annotation based five criteria: (1) faithfulness,
40%*
61%
39% *
Table 3: Human evaluation results for head-to-head comparison between passed and filtered rationales from each filter (R Rationale). Win percentages with indicate significance (p < 0.05).
the aationale-to-conten alignment filter. The same applies the sitooall-to-responss filter, where judges tend to consider a rationale that passed the filter to be more hclpful. These findings are line with our intuition that aligning rationales with dialogue contexts and ground-truth responses im- proves consistency and helpfulness of the generated rationales, respectively.
(2) helpfulness, (3) relevance, (4) specificity, and (5) overall. To avoid potential bias from different annotation formats (i.e., commonsense knowledge vs. rationales), we only use the last answer in the QA pairs from DONUT. Figure 3 presents human evaluation results on 100 randomly sampled dia- logues. Judges deem commonsense inferences (i.e., rationales) from DONUT superior in quality to the two
Experiments
ones across all aspects, vali-
dating the outstanding quality of the dataset.
Our work builds upon previous efforts (Zhou al., 2022b; Shen et al., 2022; Zhou et al., 2022a) enrich dialogue models by injecting external com- monsense knowledge. Hence, we conduct exten- sive experiments on response generation to exam- ine how dialogue CoT reasoning from DOCTOR provides commonsense knowledge to assist dia- logue agents in generating high-quality responses.
4.2 Effects of Rationale Alignment
To investigate the effect of rationale alignment, we conduct additional human evaluations rationales that have passed and failed each alignment filter via Amazon Mechanical Turk (AMT). From DONUT, randomly sample 100 dialogues that contain rationales, one that has passed the alignment filter (i.e., pass) and another that has been filtered out (i.e., fail). For each dialogue sample, human judges are given the two rationales and asked to choose the better one based on the same criteria used for quality assessment ($4.1).
5.1 Datasets
In-domain. We evaluate DOCTOR on held-out test sets of the three datasets used for knowl- edge distillation: DailyDialog (Li al., 2017). DREAM (Sun al., 2019), and MuTual (Cui al., 2020). These datasets contain open-domain dia- logues that require commonsense knowledge thoroughly understand dialogue contexts (Ghosal et al., 2022a). Note that DREAM and MuTual are
Table compares the win percentages between the two sets of rationales (i.e., pass vs. fail) for each alignment filter. We observe that judges tend to find a rationale to be more consistent passes

---

In-Domain
Out-af-Domain
DuilyDialog
DREAM
MuTual
Redlect-9K
Method
B-1
B-2
B-4
R-1.
B-1
B-2
B-4
R-L.
B-1
B-2
B-4
R-L
B-1
B-2
B-4
R-L
Cosmo [38]
20.04
8.12
2.38
13.69
20.98
2.28
17.63
6.00
12.03
14.97
4.24
0.70
11.8
COMET w/ ComFact
19.54
7.82
2.25
14.05
20.90
8.26
2.17
14.09
17.56
6.24
1.37
12.41
15.80
4.60
0.87
12.09
DIALeCT
19.63
8.05
2.36
14.31
20.69
8.18
2.17
13.82
18.01
6.35
1.31
12.23
15.54
4.55
0.85
11.82
Reflect
19.44
7.62
2.11
13.58
18.23
7.10
1.91
13.30
18.57
6.13
1.31
12.01
15.33
4.20
0.71
12.05
DOCTOR (Ours)
20.43
2,63
14.68
6.56
12.35
16.66
12.11
ChatGPT [1758]
17.25
7.18
2.11
14.72
18.90
7.81
2.29
14.86
17.92
7:03
14.83
17.28
5.29
1.12
12.77
COMET w/ ComFact
18.24
7,50
2.19
14.56
20.09
8.06
2.33
14.44
19:32
7.74
2.18
15.46
17.38
5.30
1.03
13.28
DIALeCT
16.61
6.49
1.82
13.55
18.00
6.85
1.88
13.16
19:15
7.67
1.85
15.55
17.48
5.29
1.09
12.96
Reflect
17.47
2,05
13.80
19.02
6.87
1.91
18.24
5.46
1.15
12.54
Self-CoT
18.16
7.24
2.22
12.62
7.32
13.71
14.53
4.28
0.83
11.56
DOCTOR (Ours)
19.61
8,44
2.69
15,63
21.20
8.71
2,56
14,93
20.19
8.35
2.52
15.55
18.54
5.59
1.16
12.85
Table 4: Automatic evaluation results on DailyDialog, DREAM, and MuTual using BLEU (Papineni et al., 2002)
and ROUGE (Lin, 2004). We use B-1/2/4, and R-L to denote BLEU-1/2/4, and ROUGE-L for simplicity.
designed for dialogue comprehension, but we adapt these datasets into response generation setting to fully leverage their high-quality conversations"
5.3 Baselines
To assess whether and how different knowledge sources affect the quality of generated responses. we compare DOCTOR with the following base- lines. (1) Without commonsense knowledge: We first adopt the standard response generation baseline, where the dialogue agents predict re- sponses conditioned on the dialogue history only. (2) General-purposs commonsense knowledge model: We then consider dialogue agents aug- mented with general-purpose commonsense knowledge model COMET (Hwang al., 2021). To align knowledge from COMET with dialogues, we implemented a retriever using ComFact (Gao et al., 2022) that retrieves relevant triplets to the dialogue context. (3) Dialogue-focused com- monsense knowledge model: Finally, we con- struct task-specific baselines with knowledge mod- els tailored for dialogue understanding. Specifi- cally. we implement two knowledge models, DI- ALeCT (Shen et al., 2022) and Reflect (Zhou al., 2022a), trained on dialogue datasets with qualified commonsense knowledge annotations. DIALeCT is trained on DailyDialog, DREAM, and MuTual, which are also used to train DOCTOR. Reflect, on the other hand, is trained on Reflect-9K which is tested out-of-domain dataset for DOCTOR. When tested on Reflect-9K, the model produces commonsense knowledge conditioned on oracle information, which is not given to DOCTOR, See Appendix A.6 for more details.
Out-of-domain. To assess the generalizability of DOCTOR, we consider Reflect-9K (Zhou et al., 2022a) as an additional dialogue dataset. Reflect- 9K contains dialogues annotated with common knowledge between speakers. Note that we label this dialogue dataset as out-of-domain since it is an unseen dataset that has not been used to train DOCTOR, posing challenge its generalization.
5.2 Dialogue Agents
We consider two large-scale dialogue agents, Chat- GPT and Cosmo (Kim al., 2022a). ChatGPT is an LLM with 175B parameters, trained to follow instructions. Cosmo is a general dialogue model trained with a million-scale dialogue corpus on top of T5 (Raffel et al., 2020; Lester et al., 2021). For our experiments, we use the 3B version of 10 10
Specifically,, augment both dialogue mod- els with commonsense knowledge in a zero-shot manner to assess whether knowledge sources can be readily used to assist state-of oftthe--rte-tt dialogue models. To incorporate commonsense knowledge into dialogue agents, we simply prepend common- sense inference to the dialogue history as string con- catenation (Zhou et al., 2022b; Kim et al., 2022b), where commonsense knowledge and dialogue his- tory are separated using indicators (e.g., <SEP>). We include the details on dialogue models in Ap- pendix& A.8 and our ChatGPT prompt Table 13.
Note that both general-purpose and dialogue- focused knowledge models are single-hop ap- proaches, as they are not designed to handle multi- hop reasoning in conversations.
PFor MuTual, we retain the original dataset of dialogue context and ground truth response pairs to maintain the in- tegrity of the original setup. 10 whttps:/huuginggacc.oom htwph//gacccc.gllllaaiaci colallcnaiiccssmmo

---

Model
Natural
Consistent
Specific
Engaging
Model
Consistent
Helpful
Specific
Overall
w/o CS
33%
47%
42%
ComFact
33%
42%
38%
48 %
DOCTOR
67% %
53%
58%*
52%
DOCTOR
58%
62%
52%
ComFact DOCTOR
32%
40%*
44%
49%
DIALeCT
29%
39%
48%
44 %*
%
60%*
51%
DOCTOR
71 %
61 61%
52%
56%*
DIALeCT 18%
42%*
43%*
%.
Reffect
18%
29%*
28%
33%
DOCTOR
82%
58%*
57
63%
DOCTOR
82%
71 71%*
72%
67%
Reflect 33%
41 1%*
48%
53%
Self-CoT
27%
44%
52%
46%
DOCTOR 67%
59%
52%
47%
DOCTOR
73%
56%
48%
54%
Self-CoT 33%*
46%
42%"
43%*
Table Human evaluation results of the quality of commonsense knowledge from DOCTOR baseline knowledge modes. Win percentages with indicate significance (p 0.05).
DOCTOR
67% %
54%
58%
57%
Table 5: Human evaluation results of responses from ChatGPT on DailyDialgg when paired with DOCTOR vs. baseline models. "w/o CS" denotes direct response generation without commonsense knowledge. Win per- centages with indicate significance (p < 0.05).
with limited knowledge of implicit contexts. Overall, CoT rationales from DOCTOR lead to a larger improvement over the baselines. In partic- ular, we find that dialogue models augmented with DOCTOR outperform the models paired with Re- flect, which serves an oracle in the unseen bench- mark Reflect-9K. Furthermore, human evaluation results in Table show that responses grounded dialogue CoT rationales tend to be more natural and helpful compared those grounded to baseline knowledge models such as DIALeCT, which is trained using the same corpora.
5.4 Main Results
We report the results of the automatic evaluation in Table and human evaluation via Amazon Mechan- ical Turk in Table 5. For examples of generated responses, see Appendix E.
Helpfulness of dialogue CoT rationales. On both automatic (Table 4) and human evaluations (Table 5), we observe that integrating dialogue CoT into dialogue models improves their performance over the vanilla dialogue models without common- sense. Table shows that responses conditioned on dialogue CoT particularly more natural and spe- cific than those from vanilla ChatGPT. We also ob- serve from Table 4 that DOCTOR generates helpful rationales for the dialogue models on Reflect-9K. which is not used to train DOCTOR. While these dialogue models are trained on significantly large- scale datasets, they still benefit from DOCTOR in capturing implicit information in conversations.
Comparison with self- ceff-generated COT. To ex- amine the validity LLMs as dialogue CoT rea- soners, we compare the performance of dialogue agents augmented with CoT rationales from DOC- TOR and the teacher LLM (i.e. ChatGPT), Specifi- cally, we instruct ChatGPT with the same demon- strations used in DONUT construction to gener- ate dialogue CoT rationales and predict the next response conditioned them. Surprisingly, we observe in Table 4 that augmenting dialogue mod- els with CoT rationales from ChatGPT, denoted Self-CoT, does not lead to better response quality over DOCTOR. This result shows that LLMs do not reliably produce helpful rationales for response generation, suggesting the need for aligmment filter to control the quality of the rationales.
Comparison with single-hop approaches. Ta- ble 4 compares the performance of dialogue models paired with the single-hop baselines, i.e. general-purpose and dialogue- focused common- sense knowledge models. We find that augmenting dialogue models with baseline knowledge models show only slight improvement and sometimes even a subtle decrease in performance compared to the vanilla model. These results suggest that the baselines struggle to produce correct reasoning
thauueee-focused
5.5 Analysis
Better knowledge leads to better responses. To better understand the effect of knowledge on re- sponse generation, we conduct a human evaluation

---

Training
B-1
B-2
B-4
R-L
Filtering
Natural
Consistent
Specific
Engaging
DONUT (full)
19.61
8.44
2.69
15.63
w/o R-to-C
30%*
42%"
53%
44%*
DONUT (answer-only)
18.45
7.85
2.49
15.28
w/ R-to-C
70%*
58 %
47%
56%*
Table 7:
Results of ablation on generating question using the response quality of ChatGPT on DailyDialog.
w/o R-to-R 36%*
4,%%
49%
44%*
w/ R-to-R
67 %
57 %
51 %
56 %
Table 8: Human evaluation results of responses from ChatGPT on DailyDialog when paired with the models trained with different alignment filter settings. "R-to- C" and "R-to-R" denote cationale-to-conext/eeee contex//rrsssssseeeeeeeee alignment filters respectively. Win percentages with" indicate significance (p < 0.05).
of the quality of knowledge. We randomly sample 100 knowledge inferences the test set of Daily- Dialog and ask three different human judges for each sample to compare the knowledge generated by DOCTOR and the baseline model on the same aspects used in $. For the evaluation details, see Appendix C. The results are shown in Table 6. While the baselines produce knowledge relevant to the dialogue contexts, the knowledge lacks con- sistency and is usually unhelpful in predicting the responses. Since DOCTOR generates CoT ratio- nales by grounding on implicit evidence aggregated over multiple turns, knowledge from DOCTOR far superior in terms of specificity and helpfulness, which in turn leads to better response quality.
Rationale allgnment
Respones coherence
Falled --to-r Successful r-t-
100
Cohorent
incoterant
81.2%
1.5%
60
Percenage
80
440
12.3%
4.0%
Successful ntooi Failed moo.c
Aligrwd rationale
Not aligned rationale
Iterative QA helps response generation. To an- alyze the role of questions, we conduct an ablation on the generation of questions. We train an ab- lated model under the same setting DOCTOR to generate only answers. Specifically, we explicitly remove questions from the rationale annotations in DONUT and train the model with the reconstructed data. We use ChatGPT for the dialogue model and compare the response quality on DailyDialog. The results are shown in Table 7. Without questions, the response quality drops significantly, suggesting the importance of the questions in rationalization. We posit that the role of questions in guiding the an- swers is crucial, the answers are poorly aligned with the dialogues without guidance.
Figure 4: Results of qualitative analysis. Left shows the proportions of rationales that are aligned with context (r-to-c) or response (r-to-r). Right shows the percentage of coherent responses by rationale alignment.
response decreases, the generated rationales are not helpful in predicting the next responses.
To gain deeper understanding on the effect of well-aligned rationales on the response quality, we perform in-depth analysis on the 600 evalua- tion examples randomly sampled from DailyDia log. For each sample, we present human annota- tors from AMT with the dialogue context, ratio- nales, reference response and predicted response and ask them to answer (1) whether the knowledge is aligned with the dialogue context, (2) whether the knowledge is aligned with the reference response, and (3) whether the predicted response is coherent with the dialogue context.
Applying filters improves response quality. To analyze the impact of the aligmment filters, we train two reasoning models with only passed and filtered rationales respectively for each filter. For fair com- parisons, in training, we use the same amount of ra- tionale labels that are aligned with the same context. We show the results in Table 8. The performance of the dialogue model drops significantly when the reasoning model is trained without the rationale-to- context alignment filter, suggesting the importance of the alignment between rationales and contexts. Also, when the model trained with the rationales not aligned with the responses, the quality of the
In Figure 4, we find that 81.2% of the annotated rationales are considered to be aligned with both di- alogue contexts and gold responses, suggesting that DOCTOR trained using filtered data from DONUT learns generate rationales aligned with both the dialogue contexts and the responses. We also ob- serve that only 2.1 % out of 81.1% of samples with aligned rationales are deemed incoherent, indicat- ing that the generated response tends to coherent

---

if the provided rationale is well aligned. We pro- vide an error analysis on the generated rationales and responses in Appendix B.
Limitations
We test DOCTOR 4 diverse dialogue datasets, but our experiments are limited open-domain and dyadic dialogues. Further study can apply DOC- TOR to task-oriented or multi-party dialouges. Fur- ther studies could adjust this variable dynamically, potentially allowing for deeper levels of dialogue reasoning. The rationale annotations in DONUT are fully machine-generatel Caution must be exer- cised when training models with them.
6 Related Work
Commonsense-aware dialogue models. Recent studies incorporate commonsense knowledge into dialogue models to facilitate engaging interac- tions between humans. These approaches lever- age knowledge from general-purpose knowledge model (Zhou et al., 2022b; Wu et al., 2022; Liu et al., 2022b; Li et al., 2023) or mallogue-focuser knowledge model trained with human-annotated dataset (Ghosal et al., 2022a; Gao et al., 2022). On the other hand, focus building knowledge model for multi-hop commonsense reasoning in di- alogues, where desired knowledge is induced from implicit evidence scattered in dialogue contexts.
Ethical Considerations
Texts generated by large language model contain harmful, biased, or offensive content. How- ever, we argue that this risk is mostly mitigated in our work, as we focused on the knowledge within widely-used popular dialogue datasets. The four source datasets: Daily Dialo.. MuTual, DREAM, and SODA are either high-quality datasets authored by humans or examined via safety filtering mecha- nisms (both models and web-based API) targeting crimes, violence, hate, sexuality, etc. We also exam- ine the generated rationales and manually eliminate toxic and offensive uses of language. We guarantee fair compensation for judges we hire on Amazon Mechanical Turk. We ensure an effective pay rate higher than $15 per hour based on the estimated time required to complete the tasks. The presented DONUT dataset does not contain personal data or information that provides clues for the identifica- tion of any individual group.
Chain-of-Thought reasoning. LLMs have shown an emergent capability reasoning by eliciting rationales as explanations via CoT prompting (Wei et al., 2022; Wang et al., 2023b; Zhou et al., 2023). Despite their promising ability, we find that applying CoT prompting in dialogues is non-trivial challenge even for LLMs.
Mcanwhile, recent work proposes distillation frameworks for transferring the reasoning ability of LLMS to small language models (Wang et al., 2023a; Kang et al., 2023). However, these ap- proaches focus on generating rationales for answer- ing factoid questions and are suboptimal for com- monsense reasoning dialogues. This motivates the need to selectively transfer CoT rationales from LLMs conversations.
A Acknowledgements
This work was supported by Institute of Informa- tion & Communications Technology Planning & Evaluation (ITP) grant funded by the Korean gov- ermment (MSIT) (No. 2020-0-01361, Artificial In- telligence Graduate School Program (Yonsei Uni- versity)) and (No.2021-0-02068, Artificial Intel- ligence Innovation Hub) and 2N0.2022-0-00077, AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Hetero- geneous Data). Jinyoung Yeo, Dongha Lee, and Youngjae Yu are co-corresponding authors.
7 Conclusion
Commonsense reasoning in conversations involves multi-hop reasoning, which poses challenges even for LLMs. To address this, we present dialogue chain-of-thought distillation framework that se- lectively annotates high-quality rationales using LLMs. Our contributions are as follows: (1) With our framework, we collect DONUT, a large-scale dataset for dialogue CoT reasoning.
(2) We present DOCTOR, dialogue chain-of-thougie reasoner trained on DONUT. (3) Through extensive exper- iments, we show the efficacy of DOCTOR, espe- cially in the human evaluation, where 67% of the responses generated using DOCTOR are preferred over the responses using knowledge from LLMs.
References
Forough Arabshahi, Jennifer Lee, Antoine Bosselut, Yejin Choi, and Tom Mitchell. 2021. Conversa- tional multi-hop reasoning with neural commonsense

---

knowledge and symbolic logic rules. In Proceedings of EMNLP.
Seungone Kim, Se June Joo, Hyungjoo Chae, Chae- hyeong Kim, Seung-wor monn-woo Hwang, and Jinyoung Yeo. 2022b, Mind the gap! injecting commonsense knowl- edge for abstractive dialogue summarization. In Proceedings of the 29th International Conference Computational Linguistics, pages 6285-6300 Gyeongju, Republic of Korea. International Com- mittee on Computational Linguistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of ACL.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of NeurIPS.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic, Association for Computational Lin- guistics.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: dataset for multi-turn dia- logue reasoning. In Proceedings of ACL.
Siheng Li, Wangjie Jiang, Pengda Si, Cheng Yang. Qiu Yao, Jinchao Zhang. Jie Zhou, and Yujiu Yang. 2023. Enhancing dialogue generation with conversational concept flows. In Findings of the Association for Computational Linguistics: EACI. 2023, pages 1484- 1495, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Silin Gao, Jena D. Hwang, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, and Antoine Bosselut. 2022. Com- Fact: A benchmark for linking contextual common- sense knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 1656-1675, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Yanran Li. Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao. and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of IJCNPP.
Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022a. CI- CERO: A dataset for contextualized commonsense inference in dialogues. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5010- 5028, Dublin, Ireland. Association for Computational Linguistics.
Chin-Yeu Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- rion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras. Yejin Choi, and Hannaneh Hajishirzi. 2022a. Generated knowledge prompting for commonsense reasoning, In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3154-3169, Dublin, Ireland. Association for Compu- tational Linguistics.
Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022b. Cicero: A dataset for contextualized commonsense inference in dialogues. In Proceedings of ACI..
Jena D Hwang. Chandra Bhagavatula, Ronan Le Bras,
Jeff Da. Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6384-6392.
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021a. Topic-aware contrastive learning for abstrac- tive dialogue summarization. Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pages 1229-1243, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of EMNLP.
Junpeng Liu, Yamyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021b. Topic-aware contrastive learning for abstrac- tive dialogue summarization. In Findings of EMNLP.
Minki Kang, Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. 2023. Knowledge- augmented reasoning distillation for small language models in knowledge-imtensive tasks. arXiv preprint.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint.
Hyunwoo Kim. Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhami, Gunhee Kim, Maarten Sap, et al. 2022a. Soda: Million-scale dialogue distillation with social commonsense contextualization. arXiv preprint.
Yiting Liu, Liang Li, Beichen Zhang, and Qingming Huang. 2022b. Think beyond words: Exploring

---

context-relevant visual commonsense for diverse di- alogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3 3106-3117, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery. and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In Proceedings of ICLR.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. In Proceedings of NeurlPS.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le. and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Proceedings of NeurlPS.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang. Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of NAACL-HLT.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Sixing Wu, Ying Li, Ping Xue, Dawei Zhang, and Zhonghai Wu. 2022. Section-aware commonsense knowledge-grounded grounded dialogue generation with pre- trained language model. In Proceedings of the 29th International Conference on Computational Linguis- rics, pages 521-531, Gyeongju. Republic of Korea. International Committee on Computational Linguis- tics.
Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang. Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. arXiv preprint.
Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020. Diverse and informative dia- logue generation with context-specific commonsense knowledge awareness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 5811-5820 Online. Association for Computational Linguistics.
Colin Raffel. Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research.
Christopher Richardson and Larry Heck. 2023, Com- monsense reasoning for conversational ai: A survey of the state of the art. Knowledge Augmented Meth- ods for Natural Language Processing Workshop at AAAL
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint.
Viktor Schlegel, Kamen Pavloy, and Ian Pratt-Hartmann. 2022. Can transformers reason in fragments of natu- ral language? In Proceedings of EMNLP.
Chao Zhao, Wenlin Yao, Dian Yu, Kaiqiang Song, Dong Yu, and Jianshu Chen. 2022. Learning-by-ie narratig: Narrative pre- trainin for zero-shot dialogue compre- hension. Proceedings of ACI..
Siqi Shen, Deepanway Ghosal, Navonil Majumder, Henry Lim, Rada Mihalcea, and Soujanya Poria. 2022. Multiview contextual commonsense inference: A new dataset and task.
Peixiang Zhong, Di Wang, Pengfei Li. Chen Zhang. Hao Wang. and Chunyan Miao. 2021. Care: Commonsense-aware emotional response generation with latent concepts.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of EMNLP.
Shanshan Zhong, Jinghui Qin, Zhongzhan Huang, and Daifeng Li. 2022. CEM: Machine-human chatting handoff via causal-enhance module. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 3242-3253, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Kai Sun, Dian Yu, Jianshu Chen. Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue- based reading compre- hension. Transactions of the Association for Compu- tational Linguistics. 7:217-231.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Claire Cui, Olivier Bousquet, Quac V Le, and Fdd H. Chi. 2023, Least-to-most prompting enables complex reasoning large language models, In Proceedings of ICLR.
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023a. Scott: Self- consistent chain--f-thouggee distillation. In Proceed- ings of ACL.

---

Pei Zhou, Hyundong Cho, Pegah Jandaghi. Dong-Ho Lee. Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022a. Reflect, not reflex: Inference-based common ground improves dialogue response quality, In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 10450- 10468, Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Scokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkami-Tur. 2022b. Think before you speak: Explicitly generating implicit commonsense knowledge for response generation. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pages 1237-1252, Dublin, Ireland. Association for Computational Linguistics.

---

A Experimental Details
k
Consistency
Helpfulness
Specificity
Overall
A.1 Rationale Candidate Generation
78.1
77.9
83.4
79.9
2
87.5
78.4
81.1
80.7
We prompt ChatGPT to annotate CoT rationales for a given dialogue context (i.e., history) and the ground boound-truth response. We set temperature to 0.5, and max tokens to 300. The generation is led by subquestions based on several selected types of commonsense relations. Following West et al. (2022) and Gao et al. (2022), carefully choose 11 relation types which are crucial in conversa- tions (Zhong al., 2022; Ghosal al., 2022b; Zhou et al., 2022a) from ATOMIC (Hwang et al.. 2021) as presented in Table 9. The prompt used for rationale generation is shown in Table 12.
91.2
81.4
86.9
87.1
4
88.5
78.1
88.0
83.6
5
86.9
76.5
83.2
82.6
Table 10: Human evaluation results on rationales with
different k.
The workers prefer the rationales with k = 3 most in terms of consistency, helpfulness, and overall. The Krippendorff alpha (0.82, 0.58, 0.74, 0.71) scores show moderate agreement among the raters.
Relation type
Example question
A.3 Rationale-to-connext Alignment Filter
xintent
What is the plan that speaker and listener have made?
Data collection. To collect training data for the rationale-to-context alignment filter, we randomly sample 6K dialogues from SODA (Kim et al.. 2022a) that do not overlap with those used as source dialogues for our DONUT. For each di- alogue (context and ground-truth response), we first remove all utterances in the context except for the last one obtain an incomplete context, and prompt an LLM generate rationales based on the original and incomplete contexts respectively. For the rationales generated with the original con- texts, we manually select one rationale that is well- aligned with the context. We, therefore, acquire rationale that is grounded on the whole context along with a counterfaciual rationale, which ought to be inconsistent with the dialogue as it merely considers the last utterance.
xNeed
What does speaker need to pass the final exam?
xReact
How might speaker react the breaking news from listener?
xWant
What does speaker want to know from listener?
xAttr
What is speaker's role?
oEffect
What is the result listener's inquiry about George Hatton?
oRcact
What will listener react after confirming the meeting time and place?
oWant
What does listener want convey to speaker about the prices?
isAfter
What might listener request from speaker after the agreement?
isBefore
What happened before speaker's first trip abroad?
This results in 6K dialogues aligned with their rationales and counterfactual rationales. We dupli- cate the dialogues and align them with either type of rationale for the binary classification of rationale- to-context alignment. The data (12K) is split into training (10K), validation (1K), and test set (IK) without the overlap of dialogues across them.
Causes
What causes listener concemed about being late?
Table 9: Relation types and example questions.
A.2 Ablation on number of reasoning steps
To better understand the effect of le on the qual- ity of rationale, we conduct human evaluation us- ing 100 random dialogue samples from DailyDia- log, DREAM, and MuTual. For each dialogue, we prompt ChatGPT to generate five CoT rationales with k [1, 3,4,,), respectively, as do in $3.1. Using the same criteria from Table we ask 3 different workers from Amazon Mechanical Turk to evaluate the quality of the rationale from each dialogue. The results are shown in Table 10.
Training. To train this alignment filter, we use the aforementioned data to finetune the RoBERTa- large (Liu et al., 2019) model for 3 epochs, with batch size of 40 and learning rate of le-5'1, The training is run on one NVIDIA RTX A5000 GPU. The classification performance of this filter on the test set in terms of accuracy is 93.38%.
"htns///huginngfeee.oooooetttttp

---

A.4 Rationaleto--rsppose Alignment Filter Following Liu et al. (2022a), we use the perplex- ity of the ground-truth response for calculating Po(w)l)) For efficiency, we implement the fil- ter with 4-bit quantization. The filtering process on whole rationale candidates with rationale-to- response alignment filter takes about 8 GPU hours with NVIDIA RTX A5000 GPUs.
inferences with the newline characters and prepend the dialogue history for response generation.
Reflect. We finetune the OPT-1.3B with the Reflect-9K dataset (Zhou et al., 2022a). It is a human-authored dialogue dataset with aligned in- ference sentences that approximate common be- liefs between interlocutors. Specifically, we con- catenate the annotated question and the dialogue history as inputs and train a knowledge model to predict the paired inference. The training de- tails are identical to DOCTOR. When inference, we generate all types of inference dimensions de- fined in Reflect99K (i.e.. "How would you describe Speaker?", "What might have happened before?", "What might happen after?", "What Speaker feel- ing now?", and "What Responder feeling now?"). For response generation, we concatenate inferences generated by this knowledge model using the new- line characters and prepend to the dialogue history.
A.5 DOCTOR
We train DOCTOR with DONUT for 5 epochs using a constant learning rate of 5e-4 on NVIDIA RTX A5000 GPUs. We use a batch size of 8, and the whole training process takes about 12 GPU hours. For efficiency, we adopt 16-bit quantization for training.
A.6 Baseline Knowledge Models
ComFact. Following the setting of Gao et al. (2022), we use COMET and the same relation types (i.e., xReact, xIntent, xNeed, xEffect, and x xWnn).. We use COMET-BART to implement COMET12 We use the same decoding strategy used by Gao et al. (2022) to generate inference with COMET, We implement the knowledge retriever using the source code on the official GitHub repository us- ing DeBERTa-large. Among the inferences gener- ated by COMET, we apply the retriever and choose the inferences predicted as relevant to the dialogue context.
A.7 Self Chain-of-Thought
We prompt ChatGPT to generate CoT rationales and predict the target response based on the dia- logue context and self-generated rationales. We use the same demonstrations as the prompt used in our framework and instruct ChatGPT to generate dialogue CoT rationale and the following response.
A.8 Dialogue Agents for Response Generation
Cosmo. Cosmo is dialogue model trained with million-scale dialogue corpus on top of T5 (Lester et al., 2021). We use the 3B version of !5 To generate the response from Cosmo, we use greedy decoding, and 4-bit quantization for effi- ciency. When the length of the input context ex- ceeds 512 tokens, we truncate the sequence from left to ensure the last utterance is not removed from the input. We use a special token <SEP> to sepa- rate commonsense knowledge and dialogue history. Cosmo is trained to generate responses conditioned on narrative expanded from commonsense sen- tences, which are separated by a dialogue context using the <SEP>.
DIALeCT. We take the CICERO v1 dataset from Ghosal et al. (2022a) and convert the data format following DONUT. We then fine-tuning the OPT- 1.3B (Zhang et al., 2022) with the converted data 14 The training details are identical to DOCTOR. Tak- ing the name, DIALeCT, from Shen et al. (2022), we re-implemented the model to generate common- sense inference with CICERO v1. When we gener- ate inferences with DIALeCT, use all the ques- tion types defined in CICERO v1 (i.e., "What is could be the cause of target?", "What subse- quent event happens or could happen following the target?", "What or could be the prerequisite of target?", "What is the possible emotional reaction of the listener in response to target?", and "What is or could be the motivation of target?") and we set the target of inference as the last utterance in the dialogue history. We concatenate all generated
ChatGPT. ChatGPT is an LLM with 175B pa- rameters based on InstructGPT (Ouyang al.. 2022)16. ChatGPT is trained to follow instructions given by users and return requested information in
http:////hhtuuboooommll/m//oeeee 19 http://///giuubbooom/iiiiee 59/omfffc "hatpss/hugginggtacc co/faachooo//opp-11.3.3 https////uuuunnggoococcacccoooop
I https://huugiiggfaeecccollllllee "happs//oopmma.ccmmmmo///aaggggp

---

rationales and the responses are as follows: (1) 33% of the error cases contain plausible rationales from the dialogue context that lead responses different based on the dialogue context (i.e., history) and the augmented knowledge The prompt used for response generation is in Table 13. We append speaker tags (e.g., "A:") to enforce the model to predict the next response and not confuse which speaker takes the next turn.
conversational manner. We use langchain' to send API calls OpenAI API.
We prompt ChatGPT to predict the next response
from the reference since the openness of dialogue allows for multiple possible responses for single dialogue context. (2) for 31% of the error cases, DOCTOR generates sophisticated rationales to de- scribe its reasoning even in scenarios where simple conversations enough. e.g., daily greetings. As discussed Section 5.3, we observe Fig- ure 4 that few samples with aligned rationales lead to incoherent responses from the dialogue model. One possible reason behind these few failure cases is that rationales from DOCTOR might be too com- plex and lengthy due to the complex nature of dia- logue. In such cases, chat LLMs sometimes fail fully reflect the rationales in their responses, lead- ing to incoherent responses.
B Error Analysis
For error analysis, we collect 600 random samples from the test set of DiilyDialog. We present work- ers from AMT with the dialogue context, rationales, the reference response and the predicted response and ask them to evaluate generated rationales and predicted responses by answering the following yes-no questions:
Do you agree that knowledge is well aligned with the dialogue context?
C Details for the Human Evaluation
C.1 Annotated Knowledge Quality
Do you agree that knowledge is well aligned with the reference response?
We outsource a human evaluation comparing our DONUT and human-authored datasets on Amazon Mechanical Turk (AMT). We show the interface for the evaluation in Figure 7. We ask the human judges compare the annotated knowledge from each dataset based the following five criteria:
Do you agree that the predicted response is coherent with the dialogue context?
Each sample is evaluated by 3 different workers to reduce variance and improve the reliability of the evaluation. To collect error cases, we manually inspect samples where at least 2 workers disagree with the statement in each question. Refer to Figure 4 for the statistics of our evaluation.
Faithfulness: Which knowledge statement is less contradictory to aligned dialogue con- text and target response?
Among all test examples, DOCTOR generates rationales that are not aligned with the dialogue contexts for only 5.5% of the cases. We observe two major error types behind such misalignment with the dialogue contexts: (1) for 49% of the crror cases, DOCTOR struggles to follow the complex di- alogue flow and does not answer the questions cor- rectly, failing to aggregate enough evidence even for the correct subquestions; (2) for 38% of the er- ror cases, DOCTOR concludes the rationale with statement that cannot be induced from the dialogue context, mostly because it is either too short or not specific enough to contain necessary evidence for coherent reasoning.
Helpfulness: Which knowledge statement is more helpful in predicting the target response?
Relevance: Which knowledge statement is more relevant to its aligned dialogue context?
Specificity: Which knowledge statement is more specific and focused on the target re- sponse?
Overall: Overall, which knowledge statement is more useful and valuable?
At cach voting stage, human judges are given two dialogues with aligned commonsense infer- ences and asked to select a better knowledge state- ment according the above criteria. We show answers in our rationales without the prefix (i.c., "Subquestion:"" to match the format.
We also find only 16.3% of the test samples where DOCTOR generates rationales that are not aligned with the reference responses. The two ma- jor reasons behind the misalignment between the
Whttps: htwptttwww..omm//ccc/ee/////aeea

---

C.2 Knowledge Inference Quality
D
75.02%
We conduct human evaluation of the quality of inferred knowledge via AMT. The interface for evaluation is shown Figure 8. We ask human judges to compare the knowledge from DOCTOR and the baseline knowledge models. We focus on these four criteria:
1.90%
15.78%
refionale-to-contex filter.
Filtered resoonaleto-respoosee Fittered by
Consistency: Which inference sentence is more consistent with the dialogue context?
Filtered rationales B 24.98%
Figure 6: Percentage of filtered rationales.
Specificity: Which inference sentence is more specific and focused on the target response?
Top
Step 1
Step
Step
xAttr 39.7%
oReact $4.9%
xWant 24.1%
Helpfulness: Which inference sentence is more helpful in predicting the response?
xintent 28.4%
xÄttr 13.6%
oWant 18.5%
xReact 8.9%
xlntent 5.5%
oReact 17.4%
oReact 8.8%
xintent 5.5%
xintent 11.6%
x Want 5.8%
oWant 5.0%
oEffect 6.8%
Overall, which inference sentence is more useful and valuable?
Overall:
Table 11: Top 5 relation types at each generation step.
C.3 Response Quality
D Statistical Study on Generated Rationales and Alignment Filters
We also outsource human evaluation for com- paring the responses from ChatGPT when paired with DOCTOR and the baseline knowledge models on AMT. We ask human judges to compare the re- sponses based on these four criteria following Kim et al. (2022a) and Zhou et al. (2022a):
D.1 Distribution of the Helpfulness Ratio In Figure 5, we provide the distribution of the helpfulness ratio H, from the rationale-to-response alignment filter:
Naturalness: Which response more natural (human-like)?
H, Po(ue L,U))
(5)
(o(ue(Ue))
The mean of all I, is 1.076 with standard deviation of 0.496.
Consistency:
Which response is more consis- tent (well aligned) with the dialogue context?
D.2 Why Two Alignment Filters?
Specificity: Which response more specific?
Figure illustrates the percentage of rationales filtered out by the two alignment filters target- ing rationall-to-comtex and rationaleeto-respone aligmment, respectively. The two filters together filter out 24.98% of the rationales in total, while the rationales being simultaneously filtered out by both filters only accounted for 1.9% of all the sam- ples. This manifests the necessity of implementing filters that target different types of alignment. We include examples that have gone through the two filters in Table 14 to 16.
Engagingness: Which response is more en- gaging?
365
X0X
2000
D.3 Relation Types in Rationales
As formal framework for CoT reasoning, we in- vestigate questions the QA sequence evolve from more generic types of commonsense relation to more specific ones. Table 11 the distribution
Figure 5: Distribution of the helpfulness ratio.

---

of commonsense relation types in each generation step. We present the top five most used relation types. the first step, a large portion (39.7%) of questions focused on acquiring information about the participants of the dialogue (xAttr), facilitat- ing better alignment between the rationales and context. In the last step, the most questions are related to the intention of the speaker (xWant). By inferring the communicative intent of speakers, the rationale could be more helpful in predicting the next response.
E Examples of Response Generation with Different Knowledge Sources
We show some examples of response generation using ChatGPT with different knowledge sources in Table 17, Table 18, and Table 19.

---

We studying meaningful evaluation metrics for the qualities knowledge. The knowledge aquired by answering consequent questions.
Specifically, you'll given piece dislog response and two knowledgee you'll asked to compare which knowledge better terms espects, specify which aspect important judging write your rationales
Guidelines:
[Q1- First, choose knowledge better regarding the given aspect. There are choices: Definitely
Slightly A/B
Please your instincts and choose Definitely you would confident giving response, other
Try focus quality over quantity Contentful/high quality inowledge doesn't need to be lengthy.
[Q6] Second, choose which espect influenced you the most when judging the overall quality.
factor other than
Question had bigeest influence, select "Other" and specify.
3. [Q7] Third, please describe detail your option for the questions.
would be helpful to describe both reasons you (ike the better knowledgge and reasons why you did not other knowledge.
specific
your rationale.
Dialog Context
Dialog Context
$ccontett)
$contoxt2)
Response
Response
Siresponse1)
Siresponse2)
Knowledge
Knowledge
$jinawlddee1)
$jenowledge2]
Question Which knowledge fnithful dielog response)?
Defintaly A Slightly A
Slightyy BI
Delinitbly B
Question 2. Which knowledge is more relevant to the dialogue context?
Sighily
Sligelly
Delinillly
knowledge specific?
Defimeely A
Slightly
Slightly B
Definnely B
Question 4, Which knowledge is more helpful predicting the response?
Delinielyy
Slignlly
Sightly
Defrrttdy
Question Which knowledge you like more overall?
Delinnely
Skgiily A
Sightly &
Definitlyy
Question Which affected you the when judging the overall quality?
O Faihfulnsss
Relevance
O Specificity
Helplulness
O
Other:
Figure 7: Interface for human evaluation on annotated knowledge quality.

---

We studying meaningful evalustion metrics for the qualities of knowledge. The knowledge aquired by answering consequent questions.
dialeg, response inowledge, and you'll asked compare which knowledge is better specific aspects, specify which aspect was most important for judging and write down your rationales in free-text.
Specificall, be
Guidelines:
[Q1-5] First, choose which knowledge better regardlng the given aspect. There are four choices Definitely A/B and Slightly A/B
your
Definitaly you
giving response,
Try focus quality over quantity. Contentful/tigh-en quality knowledge doesn't need to be lengihy.
aspoct influenced you
judging
quality.
some factor other than the onas in Qusstion 1-5 had the biggest influence, please select "Other" and specify.
[Q7) Third, please describe in detail your option
questions.
would be helpful to describe both you better knowledge and
dil not like the other knowledge.
Please be specific and detailed in your rationale.
Diolog
S/oonted))
Response
Stresponse)
Knowledge
Knowledge
()onoweedgea)
$0enwwedgghh)
Question Which knowledgee more faithfuul (based on the dialog and the response)?
Dwiinttll
Slightly A
Slightly
Definbely Bi
Question 2. Which knouledge more consistent (well aligned) with the dialogue context?
Sligtly
Sägitly
Defineely
Question Which knovllegge specific?
Defirately A
Slihtly A
sligtly 8
Defintel BI
Question knowledge more helpful predicting the response?
Delinbely
Skiliiy A
Skidily B
Definlady
Question 5. Which knowledge do you like more everall??
Skihily
Deinttay
Question Which aspect affected you the most when judging overall quality?
Consistenyy Specifitty
O Helpfulness
Overall
O Other
Figure 8: Interface for human evaluation on knowledge inference quality.

---

We are studying meaningful evaluation metrics for the qualities of response. The response aquired by answering consequent questions.
Specifically, you'll be given piece dialog history, and response, and you'll asked compare which response is better terms of specific aspects, specify which aspect was most important for judging, and write down your rationales in free- text.
Guidelines:
[Q]+4] First, choose which response better regarding given aspect. There choices: Definitely and Slightly A/B
Please trust your instincts and choose Definitely you would feel more confident giving response, versus the other
Try to focus on quality over quantity. taitentfullhihhqquaiiee response doesn't need to be lengthy.
[Q5] Second, choose which aspect influenced you the most when judging the overall quality.
If some factor other than ones in Question 1-4 the biggest influence, please select "Other" specify,
3. [Q6] Third, please describe in detail your option for the questions.
It would be interesting describe both reasons you like better response and why you did like the
other response.
Please be specific and detailed in your rationale.
Dialog Context
S(conlex()
Response
Response B
S(responsea)
$(responsebs)
Question Which response (human-like)
Delinitely A
Slightly
Slightly
Delinitely
Question 2. Which response is more engaging?
Definitely
Slightly
Slightly
Definitely
Question Which response
aligned) with dialogue
Dafinitely
Slightly
Slightly
Dafinitaly
Question 4, Which response is more specific?
Definitely A
Slightly A
Slightly
Definitely
Question 5. Which aspect affected you the most when judging the overall quality?
Natualness
Engagingness
Consistency
Specificity
Other:
Figure 9; Interface for human evaluation on response quality.

---

Prompt
Generate rationales for generating the target utterance ("Target:"). The rationale consists of 3-hop subquestion- subanswer pairs.
Each question should contain commonsense relation [oEfffett oReact, oWant, xAttr, xIntent, xNeed, xReact, xWant, isAfter, isBefore, Causes]. These rationales should be the crucial cue for generating the turget utterunce, but you should include the target utterance and also pretend you don't 't know the target utterance. Subquestion 3 and Subanswer should be about guessing the target utterance, Subanswer should he closely related to the target utterance but don't mention it directly. If you think generating the target utterance doesn't need commonsense, then generate None for the rationale.
Example 1
A: The mosquitoes are biting me!
B: Me loo, can't I stop scratching. They are everywhere! Sneaky little jerks.
A: Do you have any bug spray?
B: No, forgot to buy some.
A: Then we'll ll have to put up with it.
B: We can cover ourselves with beer! That way if they bite us, they"l get drunk and fall asleep.
A: That's without a doubt, the best idea you't 've had! Let's do it!
Ground-truth Response:
B: Run! They are thirsty for more!
Rationale:
Suhquestion I: What is the intent of Person B when suggesting the use of heer to ward off mosquitos? (xIntent)
Subanswer I: Person B's intention is to make the mosquitos 'drunk" and cause them to fall asleep, reducing the amount of bites.
Subquestion 2: What is Person A's reaction to Person B's unique idea to use beer? (xReact)
Subanswer 2: Person A finds the idea amusing and agreeable, and shows enthusiasm in trying it out.
Subquestion 3: What might be the effect on the mosquitos after Person A and B use beer to ward them off? (oEffect)
Subanswer 3: Unexpectedly, the mosquitos might be attracted to the beer, causing them to swarn more intensively, creating the need for Person B to warn Person A about the increased mosquito activity.
Example 2
A: Did you check the internet for next week's s weather forecast ?
B: sure did, You're in luck! It's supposed to snow all week in the mountains!
A: Yes! Somebody up there loves me! knew it wasn't I too late for snow.
B: It is kind of strange though, to have snow in April, and much of it.
A: There have been so many dry winters lately that it's about time, don't you think?
B: When you put it that way. guess the skies can't 't hold
forever.
A: This will be the best ski trip I've taken. can't 't wait my new board.
Ground-truth Response: B: It will the best as long as the weatherman is right.
Rationale:
Subquestion 1: What Person A's reaction to the snow forecast? (xReact)
Subanswer Person is excited about snow forecast because he plans to use his new board on ski trip. Subquestion 2: What does Person B think about the reliability of weather forecasts? (xAttr)
Subanswer Person might believe that weather forecasts are not always accurate, given the unpredictability of weather patterns.
Subquestion 3: What might Person B want to communicate to Person A, given Person A's excitement and the uncertainty of weather forecasts? (oWant)
Subanswer 3: Person might want to remind Person A that his ski trip being the best is contingent on the accuracy of the weather forecast.
Example 3
A:
Table 12: The Prompt for Generating Rationales. We prompt ChatGPT to generate rationales in five-shot setting (Example 3, 4, and 5 are omitted in this table). We concatenate the source dialogue at the end of the prompt.

---

Prompt
Generate the most plausible response considering the dialogue history. You refer rationale, you should ignore the rationale misleads the next response. Do not try to put too much information next response. You should follow the style of the history.
Rationale:
(rationale]
History:
[history]
Next Response:
|name_tag
Table 13: An Example Prompt for Response Generation. We prompt ChatGPT to generate response based on the dialogue context (history) and its aligned rationales zero-shot setting.
Context
A: How may help you?
would like to return an item.
A: What are you returning?
B: want to return this cellphone.
A: Is there problem?
B: It's broken.
Response
A: What exactly is wrong with it?
Probability from Rationale-to-context Alignment Filter Passed Rationale
0.99
Subquestion 1: Why did Person B come to the store? (xIntent)
Subanswer 1: Person B wants to return cellphone.
Subquestion 2: What is the reason for Person B to return the cellphone? (xAttr)
Subanswer 2: The cellphone is broken.
Subquestion 3: What information does Person A need process the return? (xNeed)
Subanswer Person A needs know what is wrong with the cellphone order process the return.
Probability from Rationale-to-conteet Alignment Filter Filtered Rationale
0.48
Subquestion 1: What is the reason for Person B's request to return the cellphone? (xIntent)
Subanswer 1: Person B wants return the cellphone because is broken.
Subquestion 2: What does Person A want know about the broken cellphone? (oWant)
Subanswer Person A wants to know what specifically wrong with the cellphone.
Subquestion 3: What information might Person B provide to answer Person A's question? (xAttr)
Subanswer 3: Person B might provide details about the specific issue with the cellphone,
such as cracked screen malfunctioning buttons.
Table 14: Examples of rationales filtered out and passed the rationale-to-conter alignment filter. The consistency scores are the outputted probability of the classifier.

---

Context
A: How much for a bus pass?
B: Well, for monthly pass, it'll cost you $65.
A: Is there anything else that doesn't cost as much?
If you're a student, you can get student bus pass. How much does a student pass cost?
That actual bus pass free.
A: It doesn't 't cost anything?
B: The only thing you'le 'Il have to pay for is the monthly sticker.
Response
A: Can you tell me how much that'll 'll cost?
Probability from Rationale-to-Context Alignment Filter
0.99
Passed Rationale
Subquestion What is the initial question of Person A to Person B? (xIntent)
Subanswer 1: Person A wants to know the cost of bus pass.
Subquestion 2: What alternative option does Person B suggest to Person A? (oReact)
Subanswer 2: Person B suggests that if Person A is student, they can get student bus pass.
Subquestion 3: What additional cost is associated with the student bus pass? (xAttr)
Subanswer 3: Person A wants know the cost of the monthly sticker associated with the
student bus pass.
Probability from Rationale-to-Contex Alignment Filter Filtered Rationale
0.10
Subquestion 1: What is Person A's question after learning about the student bus pass?
(xIntent)
Subanswer 1: Person A wants to know the cost of the monthly sticker for the student bus pass.
Subquestion 2: What is the only cost associated with the student bus pass according to Person
B? (xAttr)
Subanswer 2: According to Person B, the only cost associated with the student bus pass the monthly sticker.
Subquestion 3: What might be the reason for Person A to ask about the cost of the monthly sticker? (oWant)
Subanswer 3: Person A might want to compare the cost of the monthly sticker for the student
bus pass with the cost of regular monthly pass determine the best option for them.
Table 15: Examples of rationales filtered out and passing the rationale-to-contey, alignment filter. The consistency scores are the outputted probability of the classifier.

---

Context
A: Hey, Mom.
B: Hi, sweetie. How was your day?
A: It was good. went the library and got some books.
B: That sounds like fun. What did you get?
A: got history book, a science fiction book, and a mystery book.
B: That sounds like a great mix! I'm sure you'll enjoy all of them.
A: think so too. can't 't wait to start reading them.
B: I'm glad you're enjoying your books. Reading is such great way to learn new things and escape
into new worlds.
A: Yeah, it is. I'm really looking forward to reading these books. Thanks for getting them for me, Mom.
Response
B: You're welcome, sweetie. I'm happy help you find books that you'l 'll enjoy.
Passed Rationale
Subquestion 1: What is Person A's reason for going to the library? (xIntent)
Subanswer 1: Person A went to the library to get some books.
Subquestion 2: How does Person B feel about Person A's book choices? (oReact)
Subanswer 2: Person B thinks that Person A's book choices are great mix and that they will enjoy all of them.
Subquestion 3: What might Person B want to communicate to Person A about her role helping Person find enjoyable books? (oWant)
Subanswer 3: Person might want to communicate to Person A that she is happy to help find books that Person A will enjoy.
Filtered Rationale
Subquestion 1: What is the intent of Person A in telling Person B about the books they got from the library? (xIntent)
Subanswer 1: Person wants share with Person B the books they got from the library.
Subquestion 2: How does Person react to Person A's book choices? (oReact)
Subanswer Person B is positive about Person A's book choices and encourages them to enjoy reading.
Subquestion 3: What might be Person B's motivation for helping Person A find books to enjoy?
(x (antt))
Subanswer 3: Person B might want to foster love of reading in Person A and help them find
enjoyment in learning and exploring worlds through books.
Table 16:
Examples of rationales filtered and passing the rationale-to aatosaal-to-respponee alignment filter. The filtered rationale, although being related to the context, does not provide evidence specific to the response.

---

Hella.
there? No. I'm Anna
Dialogue History
B: Oh, well, could you please leave Anna mossage for me?
home. Would you ask her she my French dictionary? If she does, tell her to bring with her.
You know, Anna comnnng for
Ground-truth Respouse
All right. And where are you calling from?
Knowledge Source
Commonsense Knowledge & Generated Response
Nooe
Generated Response:
Sure,
check your French dictionary bring with her finds
Comfact
to help.
to have problem.
somzone.
asks for help.
Generated Response:
have problem and hoping you could with
DIALaCT
The listener asked the speaker to leave anna message on her behalf.
The listener noted down the order of the speaker. friend
speaker.
The speaker is curious to know whether anna coming for dinner or not.
The listener is happy to know that the speaker is sending her favourite book to her.
Generated Response:
Sure, I'll leave message for her. By the way, do you know if Anna is planning to come for dinner tonight?
Reflect
Riley responsible.
Tracy might have gone shopping for groceries.
Alex might ask Anna to leave her house.
Riley a good friend.
Sydney single
Generated Respocse:
chock your French dictionary when gets back.
Self CoT
Subquestion What is message that Person B wants leave Auna? (xintent)
Subanswer Person B wants to ask Anna check if she has their French dictionary and bring it with her hecause Anna coming tea Person
Subquestion What is reason for Person to usk. Anna
the French dictionary? (aReason)
Subumswer 2: Person B needs the French dictionary for some reason and wants Auza to bring with her if she has it.
potential impact Subanswer Anna forgets hring the French dicrionary: Person might have to it and might not he able to complete whatever task they needed it for.
Subquestion might
forgets bring
dictionary? (oRffcct)
Generated Response:
A: Okay. I'll let her know to check for the French dictionary and hring with her if she has it.
DOCTOR
Subquestion What is the request that Person B makes to Porson A? (xIntent)
Subunswer Person requests Person
message
she informed about
Subquestion What the specific item that Person B asks Person
Anna bring with her? (xAit))
Subanswer 2: Person B asks Person A to ask Anna to bring his French dictionary with her.
Subquestion might Person A
clarify order Subunswer 3: Person A might need clurify the address of Person B's home order to deliver the message Anna.
request? (xNeed)
Generated Respease:
I'll leave message for Anna bring French dictionary with her when she for your home. Can plense remind me of your address so that can make sure the message to her?
Table 17: An example of response generation using ChatGPT paired with different knowledge sources.

---

A: Do want to go the movies tonight?
B: I can't t. I have to review my lessons.
A: Are you having test tomerrow?
Dialogue History
B: Yes. We're having mid-lemm exam.
A: wish you good luck.
B: Thanks. But I'm little nervous.
A: Nervous? You used study very well.
Ground-truth Response
B: But haven't studied long time.
Knowledge Source
Commonsense Knowledge & Generated Response
None
Generated Response:
B: know, but this exam is really important. need do well keep my grades up.
what do. find see
get's it.
Generated Response:
B: Great, thank you for the reccipt. Let me take look radio to see what issue might be.
DIALeCT
The speaker's 's friend has to appear for a mid-term exumination.
The listener booked movic show tickets for the speaker and himself.
The speaker has already prepared for his mid term examination.
The speaker is cncouraging the listener to study hard in the mid-term examination.
The listener is happy to hear that the speaker is going to watch him.
Generated Response: B: I know, but this mid-term exam is really important. I don't want to take any chances. Riley well-read.
Reflect
Sydney might have been told by their teacher that they needed review their lessons. Sydney might go
Carson is careless.
Curious.
Generated Response:
B: know, but like haven't studied enough this time. Plus, Carson keeps distracting me with curiosity about everything. Subquestion 1:
Self-CoT
What is the reason for Person B's nervousness? (xReason)
Suhanswer Person B nervous hecause they have a mid-term exam the next day.
Subquestion 2: What is Person A's opinion of Person B's studying abilities? (xAttr) Subanswer 2: Person A believes that Person B used to study well.
Subquestion 3: What might be Person A's intention in reassuring Person B? (xIntent)
Suhanswer 3: Person A might want to boost Person B's confidence and help them feel less nervous
about the upcoming exam.
Generated Response:
B: Don't 't worry, you'l 'l do great. Just remember take deep breaths and stay focused. Subquestion 1: What is the reason for Person B's nervousness? (xAttr)
DOCTOR
hecause they have mid-term Subquestion What is Person A's observation about Person B's previous academic performance? (oReact) Subanswer 2: Person observes that Person B used study very well.
Suhanswer Person B is
review.
Subquestion What might the reason Person B's difficulty understanding material
despite previous studying efforts? (oCause)
Suhanswer 3: The reason for Person B's difficulty understanding the material might be due the passage time; as may have affected understanding the material.
Generated Response: B: know, but it's been a while since I reviewed this material. I'm afraid might have forgotten some important details.
Table 18: An example of response generation using ChatGPT paired with different knowledge sources.

---

Taxis.
Hello. I'd like book taxi, please.
Certainly. For what time?
Five o' e'elock tomorrow morning.
Dialogue History
Where from, madam?
From Qilu Hotel.
Oh, Where you want go?
the railway station. How much will cost?
About 20 dollars.
Ground-truth Response
How long will
get there?
Knowledge Source Nonc
Commonsense Knowledge & Generated Response
Gencrated Response:
B: Grear, please book the taxi for am lomorrow from Qilu Hotel the railway station. Thank you.
Confact
he touch.
have phone.
talk to someune.
talk someone.
Generated Response:
B: We usually talk on the phone once week, but she's been really busy with work lately. The speaker asked the laxi driver about the cust of her train ticket.
DIALeCT
The listener charged the speaker fine the time he'd like travel.
The speaker has reach the railway station time. The speaker curious know about the tuxi fare. The listener happy to help.
Generated Response:
B: Great, thank you for letting know. See you komorow morning
Reflect
Sydney is responsible.
business trip.
Sydney might hire tuxi.
Sydney is imeeponsible.
Cameron cureless.
Gencrated Response:
Great, please book the taxi for me. Thank you.
Sclf-CoT
Subquestion 1: What the intent of Person B when calling CFC Taxis? (xintent)
Subanswer Person
book taxi for specific time and location.
Subquestion 2: What information does Person need know order book the taxi? (oNoed)
Subanswer Person needs know time and location of the pickup, well the destination.
Subquestion 3; What is the estimated cost taxi ride from Qilu Hotel the railway station? (xAttr)
Subanswer The estimated of the taxi ride about 20 dollars.
Gencrated Response:
Would you like book for you now?
DOCTOR
Subquestion 1: What the information that Person
know after booking taxi? (xWant)
Subanswer Person wants know arrival time
Subquestion 2: Why might Person B
know the arrival time taxi? (oEffect)
Suhanswer Person might want to plan the of the day accordingly based the arrival time taxi.
Subquestion 3; What might be the reason for Person ask arrival time of the taxi? (xIntent) Subanswer Person B might
make that arrives the time and doesn't arrive
too early late.
Generated Response:
Great, thank you. you please me know the estimated arrival time the taxi?
Table 19: An example of response generation using CharGPT paired with different knowledge sources.


Reviews:
[{"review_id": "pD1vMGW5KC", "reviewer": "Reviewer_FRrp", "summary": "", "questions": "A. In eq. 4, line 207, if Z is a helpful rationale, P(u_t|Z, U_{<t}) should be greater than P(u_t| U_{<t}) hence \\tau = 1 should be the case. Why is \\tau taken to be 0.95?\n\nB. In lines 188 to 192, (a critic model to detect counterfactual rationales generated without conditioning on the dialogue context. We ask the LLM to generate a counterfactual rationale \\tilde{Z} from a counterfactual context \\tilde{U}_{<t} containing only the last utterance),  \"without conditioning on the dialogue context\" and \"from a counterfactual context tilde{U}_{<t} containing only the last utterance\" is not clear and seems contradictory. Can you please explain?\n\nC.  What are the cases where the LLM fails to generate an aligned rationale and the cases where despite generating an aligned rationale, the chat LLM fails to produce a coherent response? Can you find some broad classes of conversations where your framework fails?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. We address the following important questions.\n\n **Question A.** Why is $\\tau$ taken to be 0.95 rather than 1?\n\nWe strongly agree that $\\tau = 1$ might be the reasonable choice for our hyperparameter in the rationale-to-response alignment filter. However, in the filtering process, we observed a gray area around $\\tau=1$ where a considerable amount of samples show only subtle differences in the helpfulness ratio, implying that their detrimental effects might be trivial. Our motivation was that adding such samples would result in a larger dataset with more variation in annotated rationales, thus enhancing the performance and robustness of the commonsense reasoner.\n\nWe tested various values around 1 for $\\tau$ (0.9, 0.95, 1, and 1.05) to study this effect and find a balance between the quality and quantity of the annotated rationales. We ran experiments using ChatGPT as a dialogue agent tested on DailyDialog. The results are shown in the table below:\n\n| $\\\\tau$     | B-1   | B-2  | B-4  | R-L   | \\|\\| | Relative size compared to DONUT |\n| ----------- | ----- | ---- | ---- | ----- | ------ | ------------------------------- |\n| 0.9         | 18.75 | 7.73 | 2.26 | 14.69 | \\|\\| | 1.1                             |\n| 0.95 (Ours) | 19.61 | 8.44 | 2.69 | 15.63 | \\|\\| | 1                               |\n| 1           | 19.58 | 8.34 | 2.6  | 15.52 | \\|\\| | 0.47                            |\n| 1.05        | 19.15 | 8.13 | 2.51 | 15.19 | \\|\\| | 0.19                            |\n\nWe find that $\\tau = 0.95$ shows the best results, thus we chose this value as a hyperparameter for $\\tau$. We should’ve included these results in our paper to show the readers the reason behind the choice of $\\tau$. Thank you again for pointing it out, we will include them in the camera-ready version.\n\n**Question B.** \n\nCounterfactual rationales are generated while being conditioned only on the last utterance ($u_{t-1}$) preceding the target response rather than the entire dialogue context $U_{<t}$. The sentence in line 189-190 should be revised as “a critic model to detect counterfactual rationales generated without correctly grounding on the dialogue context.”. We will clarify this in the camera-ready.\n\n**Question C.**\n\nThank you again for your valuable question. We believe that analyzing these cases will definitely help us to gain insights regarding improving our work.\nTo find failure cases, we randomly select 600 samples from DailyDialog and ask workers from AMT to evaluate rationales and generated responses by answering the following yes-no questions:\n\n- Do you agree that knowledge is well-aligned with the dialogue context?\n- Do you agree that knowledge is well-aligned with the reference response?\n- Do you agree that the predicted response is coherent with the dialogue context?\n\nEach sample is evaluated by 3 different workers to reduce variance and improve the reliability of the evaluation. Please refer to the table in C-3. Statistics for the statistics from the workers’ evaluation.\n\n**C-1. When does DOCTOR fail at generating aligned rationales?**\n\nWe manually inspect samples that at least 2 workers consider not aligned with the dialogue context or the response. We categorize the type of failure and describe the cases where DOCTOR fails at generating aligned rationales.\n\n**C-1-1. Failed cases of rationale-to-context alignment:**\n\nWe observe two major types of cases where DOCTOR hallucinates and generates rationales that are not aligned with the dialogue contexts.\n\n- *Complex dialogue flow (49%):* We find that the questions in CoT rationale are often well generated. However, DOCTOR struggles to follow the dialogue flow and thus fails to answer the questions correctly. This results in failure to aggregate enough evidence.\n\n- *Lack of evidence in dialogue context (38%):* DOCTOR concludes with a statement that is unrelated to what is mentioned in the dialogue context or cannot be induced based on the dialogue context when generating the rationales. This is mainly due to the absence of necessary evidence for reasoning, e.g. when a dialogue context is too short to contain enough evidence that supports coherent reasoning.\n \n**C-1-2. Failed cases of rationale-to-response alignment:**\n\nWe find two major reasons that lead to rationales misaligned with the reference response.\n\n- *Openness of dialogue (33%):* Due to the open-ended nature of dialogues, many possible responses exist for a single dialogue context. Therefore, while the rationales are plausible enough, they lead to responses different from the reference responses.\n\n- *Small talks (31%):* DOCTOR often generates sophisticated rationales to describe its reasoning even in scenarios where simple conversations are enough. e.g., daily greetings.\n \n**C-2. When does the chat LLM fail to produce a coherent response even if it is provided with an aligned rationale by DOCTOR (given the dialogue context)?**\n\nWe observe that it is highly unlikely that the response will be incoherent if the provided rationale is well aligned, considering that out of 81.1% of samples with aligned rationales, only 2.1% are deemed incoherent.\n\nOne possible reason behind these few failure cases is that rationales generated by DOCTOR might be too complex and lengthy due to the complex nature of dialogue. In such cases, chat LLMs sometimes fail to fully reflect the rationales in their responses, leading to incoherent responses.\n\n\n\n\n\n\n**C-3. Statistics:**\n|                             | Successful r-to-c alignment | Failed r-to-c alignment |\n| --------------------------- | :---------------------------: | :-----------------------: |\n| Successful r-to-r alignment | 81.2%                       | 1.5%                    |\n| Failed r-to-r alignment     | 12.3%                       | 4.0%                    |\n\n\n|                     | Aligned rationale | Not aligned rationale |\n| ------------------- | :-----------------: | :---------------------: |\n| Coherent response   | 79.0%             | 11.8%                 |\n| Incoherent response | 2.1%              | 7.0%                  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a very important problem of common sense reasoning over multiple turns in a conversation and provides a reliable dialogue rationale dataset that might be useful to the community. \n\n2. The paper is very well written, precise, and easy to follow.", "reasons_to_reject": "Although the paper proposes an overall novel and robust framework, the individual components have been already explored by the community.", "questions_to_authors": "A. In eq. 4, line 207, if Z is a helpful rationale, P(u_t|Z, U_{<t}) should be greater than P(u_t| U_{<t}) hence \\tau = 1 should be the case. Why is \\tau taken to be 0.95?\n\nB. In lines 188 to 192, (a critic model to detect counterfactual rationales generated without conditioning on the dialogue context. We ask the LLM to generate a counterfactual rationale \\tilde{Z} from a counterfactual context \\tilde{U}_{<t} containing only the last utterance),  \"without conditioning on the dialogue context\" and \"from a counterfactual context tilde{U}_{<t} containing only the last utterance\" is not clear and seems contradictory. Can you please explain?\n\nC.  What are the cases where the LLM fails to generate an aligned rationale and the cases where despite generating an aligned rationale, the chat LLM fails to produce a coherent response? Can you find some broad classes of conversations where your framework fails?", "paper_topic_and_main_contributions": "The paper tries to improve common sense reasoning over multiple turns in conversational agents by augmenting with a conversation rationale produced by a large language model. They propose to overcome the limitations of LLMs to generate reliable rationales by training a reliable COT reasoner on selective rationales.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "l3jfeAp7gO", "reviewer": "Reviewer_cf7H", "summary": "", "questions": "See the weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for acknowledging the importance of our work and providing helpful suggestions.\n\n**Q1.** For our dataset, we sampled 6K dialogues from SODA. From these, we constructed 6K $(U_{<t}, U_t, \\tilde{Z})$ pairs and another 6K $(\\tilde{U}_{<t}, U_t, \\tilde{Z})$ pairs, resulting in a total of 12K samples. We then randomly divided these samples into training, validation, and test sets, with 10K, 1K, and 1K samples respectively. We detailed the training of our critic model in Appendix A.2. Thanks to your questions, we recognized the importance of this information and believe it would be beneficial to include it within the main 8 pages of the paper. Again, we appreciate your keen observation.\n\n**Q2.** We agree with the reviewer that the number of reasoning steps $k$ in the rationales could be a factor that affects the quality of the generated rationales and the responses. From a qualitative analysis on the generated rationales, we discover that rationales shorter than 3 steps often lack sufficient evidence to support the reasoning while rationales longer than 3 steps tend to aggregate excessive evidence and contain spurious reasoning. Thus, we initially select the $k$ as 3.\n\nTo better understand the effect of $k$ on the quality of rationale, we conduct human evaluation using 100 random dialogue samples from DailyDialog, DREAM, and MuTual. For each dialogue, we prompt ChatGPT to generate five CoT rationales with $k=\\\\{1,2,3,4,5\\\\}$, respectively, as we do in Section 3.1. Using the same criteria from Table 3, we asked 3 different workers from Amazon Mechanical Turk to evaluate the quality of the rationale from each dialogue. \n\n| $k$ | Consistency | Helpfulness | Specificity | Overall |\n| --- | :-----------: | :-----------: | :-----------: | :-------: |\n| 1   | 78.1        | 77.9        | 83.4        | 79.9    |\n| 2   | 87.5        | 78.4        | 81.1        | 80.7    |\n| 3   | 91.2        | 81.4        | 86.9        | 87.1    |\n| 4   | 88.5        | 78.1        | 88          | 83.6    |\n| 5   | 86.9        | 76.5        | 83.2        | 82.6    |\n\nIn line with our initial motivation on choosing $k$, the workers preferred the rationales with $k=3$ most in terms of consistency, helpfulness, and overall. The Krippendorff alpha (0.82, 0.58, 0.74, 0.71) scores show a moderate agreement among the raters. We will include these results in the camera-ready version to provide empirical evidence for our choice of $k$. We once again thank you for your insight.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, easy to read and understand.\n2. I appreciate the authors for doing a robust evaluation of the Donut dataset as this would be helpful for other researchers working on dialogue system.\n3. The simple yet novel use of rationales for generating relevant responses using the Doctor framework is appreciable and is evidence through strange empirical performance on multiple dialogue dataset.", "reasons_to_reject": "1. The critic model is used to determine if a rationale is valid or not. It would be good to have an understanding of the number of training instances required for training the critic model.\n2. An analysis on the number of rationales required for generating a relevant response would be helpful.", "questions_to_authors": "See the weakness.", "paper_topic_and_main_contributions": "The paper proposes the use of CoT methodology for generating appropriate responses to a dialogue. First, the authors propose the Donut dataset that contains multiple CoT rationales for generating appropriate responses. The CoT rationales come in the form of QA pairs which an LLM can utilise to generate appropriate and relevant response. Then, the authors propose the Doctor framework, that utilises this large-scale Doctor dataset for supplementing the LLM with CoT rationales and thus generation improved responses.\n\nResearchers working on dialogue system can benefit from the utilisation of Donut dataset for different dialogue tasks. Additionally, the authors have showcased the strong performance of Doctor on multiple dialogue dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ydLq84XWs", "reviewer": "Reviewer_kVfw", "summary": "", "questions": "Token based scores, Bleu or Rogue seems quite low, which makes sense for this challenging setting.  Wondering if you tried semantic based scoring approaches, such as BleuBERT?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your time and the fact that you recognize the value of our research.\n\nFollowing your suggestion, we conduct an evaluation using semantic-based metrics, BLEURT and BERTScore. We show the results in the following table. Thanks to your suggestion, we can provide more diverse criteria for evaluation not only based on lexical similarity but also semantic similarity. We will include these results in the camera-ready. \n\n\n| Method⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀| \\| |       |       | DailyDialog |          | \\| |       |       | DREAM     |          | \\| |       |       | MuTual    |          | \\| |       |       | Reflect-9K |          |\n| ---------------- | --- | ----- | ----- | ----------- | -------- | --- | ----- | ----- | --------- | -------- | --- | ----- | ----- | --------- | -------- | --- | ----- | ----- | ---------- | -------- |\n|                  | \\| | B-1   | R-L   | BERTScore   | BLEURT   | \\| | B-1   | R-L   | BERTScore | BLEURT   | \\| | B-1   | R-L   | BERTScore | BLEURT   | \\| | B-1   | R-L   | BERTScore  | BLEURT   |\n| Cosmo            | \\| | 20.04 | 13.69 | 0.8539      | \\-1.0439 | \\| | 20.98 | 14.03 | 0.8629    | \\-1.0414 | \\| | 17.63 | 12.03 | 0.8529    | \\-0.9728 | \\| | 14.97 | 11.80 | 0.8567     | \\-1.0875 |\n| &plus;COMET w/ ComFact | \\| | 19.54 | 14.05 | 0.8553      | \\-0.9846 | \\| | 20.90 | 14.09 | 0.8621    | \\-1.0405 | \\| | 17.56 | 12.41 | 0.8524    | \\-0.9585 | \\| | 15.80 | 12.09 | 0.8558     | \\-1.084  |\n|&plus; DIALeCT          | \\| | 19.63 | 14.31 | 0.8565      | \\-0.9767 | \\| | 20.69 | 13.82 | 0.8623    | \\-1.0271 | \\| | 18.01 | 12.23  | 0.854   | \\-0.9738  | \\| | 15.54 | 11.82 | 0.8559     | \\-1.1007 |\n| &plus;Reflect          | \\| | 19.44 | 13.58 | 0.8529      | \\-1.0515 | \\| | 18.23 | 13.30 | 0.8569    | \\-1.0969 | \\| | 18.57 | 12.01 | 0.8438    | \\-1.0441 | \\| | 15.33 | 12.05 | 0.8565     | \\-1.099  |\n| &plus;DOCTOR           | \\| | 20.43 | 14.68 | 0.8572      | \\-0.9593 | \\| | 21.26 | 14.26 | 0.8632    | \\-1.0215 | \\| | 17.90 | 12.35 | 0.8539    | \\-0.9737 | \\| | 16.66 | 12.11 | 0.8569     | \\-1.0749 |\n|                  | \\| |       |       |             |          | \\| |       |       |           |          | \\| |       |       |           |          | \\| |       |       |            |          |\n| ChatGPT          | \\| | 17.25 | 14.72 | 0.8493      | \\-0.9676 | \\| | 18.90 | 14.86 | 0.8642    | \\-0.9983 | \\| | 17.92 | 14.83 | 0.8556    | \\-0.8589 | \\| | 17.28 | 12.77 |      0.8619      |   -0.9737       |\n| &plus;COMET w/ ComFact | \\| | 18.24 | 14.56 | 0.8501      | \\-0.9341 | \\| | 20.09 | 14.44 | 0.8642    | \\-0.9805 | \\| | 19.32 | 15.46 | 0.8567    | \\-0.8325 | \\| | 17.38 | 13.28 |   0.8611         |    -0.9694      |\n| &plus;DIALeCT          | \\| | 16.61 | 13.55 | 0.8471      | \\-1.0129 | \\| | 18.00 | 13.16 | 0.8615    | \\-1.0492 | \\| | 19.15 | 15.55 | 0.8574    | \\-0.8398 | \\| | 17.48 | 12.96 |       0.8602     |  -1.0101       |\n| &plus;Reflect          | \\| | 17.47 | 13.80 | 0.8474      | \\-1.0365 | \\| | 19.02 | 13.37 | 0.8612    | \\-1.0728 | \\| | 18.14 | 14.27 | 0.8579    | \\-0.8325 | \\| | 18.24 | 12.54 |    0.8600        |   -0.9702       |\n| &plus;Self-CoT         | \\| | 18.16 | 12.62 | 0.8461      | \\-0.9953 | \\| | 18.88 | 12.17 | 0.859     | \\-1.0544 | \\| | 19.97 | 13.71 | 0.8549    | \\-0.8502 | \\| | 14.53 | 11.56 |      0.8579      |  -1.0274        |\n| &plus;DOCTOR           | \\| | 19.61 | 15.63 | 0.8512      | \\-0.9224 | \\| | 21.2  | 14.93 | 0.8654    | \\-0.965  | \\| | 20.19 | 15.55 | 0.8583    | \\-0.7906 | \\| | 18.54 | 12.85 |     0.8621       |-0.9649|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The framework is novel and can be widely applied to nuanced conversations, or in general any text requiring deeper inference to understand the narrative.  Their method improves other competitive baseline approaches.", "reasons_to_reject": "N/A", "questions_to_authors": "Token based scores, Bleu or Rogue seems quite low, which makes sense for this challenging setting.  Wondering if you tried semantic based scoring approaches, such as BleuBERT?", "paper_topic_and_main_contributions": "This paper introduces a novel framework to model conversations, through multi-step reasoning using LLMs.  The challenge in modeling \"deep\" conversations is using commonsense reasoning at multiple turns of the conversations and collectively combine the reasoning steps to infer the main intent of the conversation.  This work demonstrates how \"shallow\" (single-step) approaches, such as integrating commonsense knowledge from per-trained KBs/models, and using single shot LLM prompting are not sufficient to tackle the task.\nThe authors contribute a 3 step process, first step is to extract implicit pairs of <question, answer> useful to model the conversation, using CoT approach.  The second step is to filter these conversations to ensure the sequence is \"consistent\".  The first 2 steps also allow to build a dataset to train the final conversational model in the third step, to generate the final dialogue, using implicit information from the previous steps.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EUgsPBnNKw", "reviewer": "Reviewer_dhkb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback.\n\n**Q1 and Q3.**\n> Abstract, last line, summarise the percentage improvement.\n\n> Conclusion should provide the main contributions and percentage improvements.\n\n\nThank you for your valuable comments on improving our paper. We will clarify our contribution in the camera-ready as you suggested:\n- We will add the following statement in the last line of the abstract: “We conduct extensive experiments to show that enhancing dialogue agents with high-quality rationales from DOCTOR significantly improves the quality of their responses. The results of the human evaluation show that 67% of responses using DOCTOR are preferred over the responses using knowledge from LLMs.”\n \n- We will revise the conclusion as follows: “Commonsense reasoning in conversations involves multi-hop reasoning, which poses challenges even for LLMs. To address this, we present a dialogue chain-of-thought distillation framework that selectively annotates high-quality rationales using LLMs. Our contributions are as follows: (1) With our framework, we collect DONUT, a large-scale dataset for dialogue CoT reasoning. (2) We present DOCTOR, a dialogue chain-of-thought reasoner trained on DONUT. (3)Through extensive experiments, we show the efficacy of DOCTOR, especially in the human evaluation, 67% of the responses using DOCTOR are preferred over the responses using knowledge from LLMs.”\nThe revised version would be definitely clearer for readers to understand our contributions.\n\n\n**Q2. Pg 8, right col, line no 529, why are these approaches not applicable for commonsense reasoning?**\n\nWe would like to clarify that while these approaches can be applied to commonsense reasoning, they focus on answering factoid questions and thus are suboptimal for commonsense reasoning in dialogues. We kindly note that reasoning in dialogues differs from reasoning in question answering in that the former involves a deeper understanding of the dialogue context (e.g., emotional state and situation of speakers) [1]. We apologize for any misunderstandings, and we will revise the sentence accordingly in our final draft to avoid any confusion.\n\n\n\n\n**Q4. How does this approach compare with ‘Augmenting end-to-end dialogue systems with commonsense knowledge’?**\n\nWe first appreciate the reviewer for mentioning this paper, which is a foundational work on integrating commonsense knowledge into dialogue agents using commonsense knowledge graphs [2]. We will make sure that we include this reference in our final draft.\n\nWe would also like to point out that while commonsense knowledge graphs provide valuable knowledge, integrating static knowledge graphs for conversations with vast world knowledge has always been a challenge [2, 3]. More recent studies thus rely on neural commonsense models [4], which offer more flexibility in representing and utilizing vast knowledge. Taking a step further, our work introduces a multi-hop commonsense reasoning model for dialogues that learns to represent more complex and diverse knowledge in dialogues using the reasoning ability of LLMs.\n\n[1] Sahand Sabour, Chujie Zheng, Minlie Huang, CEM: Commonsense-aware Empathetic Response Generation, AAAI 2022\n\n[2] Young, Tom, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang, Augmenting end-to-end dialogue systems with commonsense knowledge, AAAI 2018\n\n[3] Christopher Richardson and Larry Heck. 2023, Commonsense reasoning for conversational ai: A survey of the state of the art, KnowledgeNLP-AAAI 2023\n\n[4] Bosselut, Antoine, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi, COMET: Commonsense Transformers for Automatic Knowledge Graph Construction, ACL 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Figure 3 that it performs much better than common-sense annotation. \n\nTable 5 also shows that the response of ChatGPT combined with the proposed DOCTOR is significantly better than baselines. \n\n\n\n", "reasons_to_reject": "Abstract, last line, summarise the percentage improvement. \n\nPg 8, right col, line no 529, why are these approaches not applicable for commonsense reasoning?\n\nConclusion should provide the main contributions and percentage improvements. \n\nHow does this approach compare with ‘Augmenting end-to-end dialogue systems with commonsense knowledge’ AAAI 2018\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Identifying and aggregating evidence in chatbots is very challenging. In this paper they use counterfactual rationales to train the model. \nThey also use dialogue comprehension datasets such as DREAM to extract responses. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors tackle the problem of multi-hop common sense reasoning in dialogue and propose a dataset and a model. The reviewers appreciated the dataset, the clarity of the work, and the results achieved. They did raise a few concerns such as the fact that the novelty lies in the combination of known methods or some missing details but the authors adequately address them. Therefore I recommend accepting this paper.
Decision: Accept-Main