CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs
Authors: Taha Aksu
Devamanyu Hazarika
Shikib Mehri
Seokhwan Kim
Dilek Hakkani-Tur
Yang Liu
Mahdi Namazifar
Venue: EMNLP 2023 Main (2023)
Keywords: Instruction Tuning
Open Domain Dialog
Controlled text generation
Task unification
Unified grounding
Compositional learning
Compositional instructions
CESAR
TL;DR: We set a unified ground for a large set of open-domain dialog tasks and automate integration of atomic tasks into more complex compositional tasks.

Abstract: Instruction-based multitasking has played a critical role in the success of large language models (LLMs) in multi-turn dialog applications. While publicly available LLMs have shown promising performance, when exposed to complex instructions with multiple constraints, they lag against state-of-the-art models like ChatGPT. In this work, we hypothesize that the availability of large-scale complex demonstrations is crucial in bridging this gap. Focusing on dialog applications, we propose a novel framework, CESAR, that unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without any manual effort.

We apply CESAR on InstructDial, a benchmark for instruction-based dialog tasks. We further enhance InstructDial with new datasets and tasks and utilize CESAR to induce complex tasks with compositional instructions. This results in a new benchmark called InstructDial++, which includes 63 datasets with 86 basic tasks and 68 composite tasks. Through rigorous experiments, we demonstrate the scalability of CESAR in providing rich instructions. Models trained on InstructDial++ can follow compositional prompts, such as prompts that ask for multiple stylistic constraints.

CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs
Taha Aksu Devamanyu Hazarika²) Shikib Mehri', Seokhwan Kim², Dilek Hakkani-Tür', Yang Liu?, Mahdi Namazifar National University of Singapore
Amazon Alexa Al taksuêu. nus edu, dvhaz@amazon, com
Abstract
Instruction:
Generate coherent response based provided actions.
Instruction- based multitasking has played crit- ical role in the success of large language mod- els (LLMs) in multi-turn dialog applications. While publicly-availabl LLMs have shown promising performance. when exposed to com- plex instructions with multiple constraints, they lag against state-of of-the-art models like Chat- GPT. In this work, we hypothesize that the availability of large-scale complex demonstra- tions is crucial in bridging this gap. Focus- ing on dialog applications, we propose a novel framework, CESAR, that unifies large ber of dialog tasks in the same format and al- lows programmatic induction complex in- structions without any manual effort.
Dialog Context
Hey do you like music?
my ATOMIC
-PROPP
COMPOSITIONAL
Action:
Action:
The response
Action:
The respanse
Thh resparse should contain keywords: catchhy tune, playful.
should contain
+
should depicl diaing art:
followno
keywords: catchy tume, playful.
ank question
response depict dialog question
(1-D Groundingl
(1-0 Grounding!
22-0 Grounding
OUTPUT
Output: Ara you fan the catchy tune or do you anjoy the playtul
We apply CESAR on InstructDial, bench- mark for instruction-based dialog tasks. We further enhance InstructDial with new datasets and tasks and utilize CESAR to induce complex tasks with compositional instructions. This re- sults a new benchmark called InstructDial+++ which includes 63 datasets with 86 basic tasks and 68 composite tasks. Through rigorous ex- periments, we demonstrate the scalability of CESAR in providing rich instructions. Models trained on InstructDial++ can follow compo- sitional prompts, such prompts that ask for multiple stylistic constraints.
lyrics?
Figure An illustrative integration of compound tasks, namely keyword controlled generation and act grounded generation, into more complex compositional one. These tasks automatically merged-without without human oversight-using CESAR.
model's ''s response to meet two control dimensions, (i) incorporating three keywords in the response 'catchy', 'tune", and 'playful" and (ii) following the dialog act ask question. A large language model (LLM) may perform well these dimen- sions individually. However, it may struggle meet the requirements simultancously as it has not seen such a composition of constraints during the training process. Prior work has addressed this problem through novel architectures or prompting tricks (Peng et al., 2023; Ramakrishnan et al., 2022: Hu al., 2022). However, despite the proven effec- tiveness of scaling up the number of tasks (Chung et al., 2022a), the prior efforts, to the best of our knowledge, have yet to focus on scaling up compo- sitional data during training.
1 Introduction
Instruction tuning is popular multi-tasking method for fine-tuning large language models (LLMs). In this setup, LLMs are trained over range of tasks specified by instructions, which lets them generalize over new task descriptions at case (Wei et al., 2021). Although instruction tun- ing enables individual task performance at scale, a language model's practical usage often requires high performance compositions of these tasks. For example, in Fig. 1, the prompt requires the
One could handle complex instructions by intro- ducing compositional tasks as demonstrations at
Work done during internship at Amazon Alexa AL. First two authors contributed equally.

---

Collection
Size
Prompt
# Tasks
# Data Points
Objective
Zero
10M-
FLANN
Few
1836
15M
NI.P Tasks
540B
COT
Zero
30-
NLP Tasks
OPT-IML
Few
2067
18M
175B
Meta Tasks
COT
400M-
Zero
InstructDial
48
250k
Dialog Tasks
3B
Fow
Dialog Tasks
CESAR
7ero
154
450k
11B
Compositional Tasks
Table 1: CESAR compared against other recent instruction tuning benchmarks. Zero and Few stand for zero- and few-shot, respectively, whereas COT stand for chain of thought prompting.
training stages. However, getting compositional data at scale is non-trivial task because the num- ber of compositions grows exponentially with the number of atomic tasks. This introduces significant human labor in adding appropriate instructions for each new composition.
td) Finally, perform comprehensive experi- ments that reveal multi-faceted benefits of having compositional tasks in fine-tuning stages (c.f. $6): (a) They improve compositional task performance for both seen and unseen task compositions; (b) They improve atomic or non-compositional task performance under similar data budgets.
A naive solution to this challenge might be com- bining individual task prompts' instructions and control sequences, following the controlled genera- tion literature (Yang et al., 2022; Liu et al., 2022). However, for cross-task compositions with mul- tiple constraints, this could result in nonsensical tasks, in ways such as their composition is either infeasible, invalid or too idiosyncratic to be of any practical use (see Fig. 10 for example). Thus, reliable and scalable mechanisms compose tasks (and their instructions) without manual effort are highly desirable.
The CESAR framework, along with the Instruct- Dial++ benchmark, enables the automated genera- tion of complex tasks in dialog benchmarks, which we believe is one of the critical ingredients in bridg- ing the gap between publicly-availatee dialog mod- els compared to proprictary AI assistants.
2 Related Work
The term instruction-tuninn was popularized by Wei et al. (2021); Mishra et al. (2022) and has gained popularity among various researchers- for example, Ouyang et al. (2022) optimized outputs based user preferences by incorporating human feedback. Chung al. (2022a), on the other hand, demonstrated that scaling tasks and model size im- proved performance across benchmarks. Finally, InstructDial (Gupta et al., 2022) focused on instruc- tion tuning for downstream dialog tasks but lacked curated compositional tasks. To address this gap. CESAR enables large-scale training of composi- tional tasks. Please refer to Table for compari- son of CESAR with the other recent benchmarks
To address the above-mentioned challenge, we make the following contributions: First, propose an instruction-basse frame- work for dialog tasks named CESAR. CESAR modularizes dialog task prompts based on their in- put, constraints, and outputs. This enables an auto- matic combination of specific parts of different task prompts to generate compositional task prompts without any human intervention e.g. in Fig. CESAR combines two atomic tasks which only dif- fer in the constraint component of the prompt. We describe the complete framework in $4.
Contributions.
Unified Grounding. The research community has recently emphasized Unified Grounding as a solution for diverse datasets. UnifiedSKG (Xie et al., 2022) unifies 21 SKG tasks into text-to- text format, demonstrating improved performance through multitask learning. Convlab3 (Zhu et al., 2022) proposes a shared format for task-oriented
ùi) We introduce InstructDial++, an update over the original InstructDial benchmark (Gupta et al., 2022). It incorporates more atomic tasks and datasets and introduces composite tasks by utiliz- ing the CESAR framework. Overall, InstructDial++ consists of 86 basic and 68 composite tasks defined on 63 datasets. We detail the InstructDial++ bench- mark in $5.
"hile Flan and OPT also consider dialog datasets, its coverage is not as extensive as InstructDial and CESAR.

---

Instruction constraints)
Instruction
constraints)
Dialog Context
Generale reeporse that follows the dialog act: contimation givan dialog context. The reeporse should have keylorrd "Everes!"
Generato responsa
question
the given dialog context. The reeporse should start with the intial phrase "Is Everest Da not use the following words within the
Hey would you like road logelher??
Okay, but where??
Response
Response
Sure, just to sonfirm, you meein we're heeding somewhere North, like Evereet?
ChatGPT:
ChatGPT: is Everest the destination up North
Somewhere North
have
InstructDiat:
I'm not would lkk
InstructDial: Everest you want to camp ?
Investigating the disparity between instruction following capability for dialog response generation between closed-access (ChatGPT) and open-access (InstructDial) dialog models. ChatGPT comfortably satisfies two constraints in dialog response generation whereas showing signs of struggle in >> 3 constraint scenarios (more examples in Appendix B). In contrast, InstructDial lags behind ChatGPT since it is unable to satisfy >= 2 constraint scenarios (further evidence Fig. 3).
Figure 2:
dialog datasets. BlenderBot 3 (Shuster et al., 2022) adopts a modular approach, assigning specific tasks to different parts of prompt. However, none of these frameworks have explored task composition from their unification efforts.
compare to CESAR framework, which specifically addresses the scalability of compositional tasks in training data.
3 Motivation
Compositional Generalization via Structure. Incorporating structure into the prompts have con- sistently been shown to enhance the language model's ability to handle compositional tasks. Bursztyn et al. (2022) propose compositional fine- tuming, which involves breaking down the task into its constituent parts and combining these parts using decision templates. Keysers et al. (2019) use logical forms in training data enable rule- based generation of compound samples for com- positional generalization semantic parsing tasks. Finally, Chen et al. (2022b) design task-specific prompts that provide detailed information about each task to help the model understand commonal- ities across different tasks.
In this section, we first explore the performance disparity between closed-access and open-access models complex compositional tasks. We then investigate the impact of including compositions in the training data on task performance.
Closed- vs. Open-access Models in Complex Dialog Tasks. We begin by checking the dispar- ity between open and closed-access models. In Fig. 2, we notice that ChatGPT (a closed-access model) can produce satisfactory results for simple composite tasks whereas publicly available DIAL- TO (Gupta al., 2022) struggles. This demon- strates that open-access models require additional resources to improve on complex dialog tasks.
Can Compositional Demonstrations Improve Performance? Next, we want to verify whether the presence of compositional demonstrations can improve performance on complex dialog tasks. For this, we design preliminary experiment where select four dialog tasks: generating di- alog responses where control the beginning phrase, ii) the ending phrase, iii) the length (short, medium, and long), and iv) keyword be incorpo- rated the response. We manually create instruc- tions for all possible combinations of these tasks (e.g. combining begins with with ends with gen- eration, amongst others. We fine-tune the public
Dialog Control. Various approaches have been used to control multiple attributes, such as adhe- sive architectures prompting techniques. For instance, Chen al. (2022a) define task-specific prompts and concatenate them at runtime to tackle complex tasks. Alternatively, SHAO et al. (2023) learn a compositional codebook where each task corresponds to combination of codes, allowing controllability at inference time. Subramanian et al. (2018) embed each target attribute separately and average them as the seqqueee symbol dur- ing generation. Hu et al. (2022) propose two- stage decoder that imposes stylistic and word-level constraints separately within the seq2seq frame- work. However, none of these approaches directly
"We define closed vs. open-access hased on whether the model parameters and data are publicly available or not.

---

60
ing to dialog interaction, such as ullerances. speaker states, intents, personas, stylistic tributes of utterances, dialog summaries, ut- terance revisions, external knowledge snippets, amongst others.
Baseine
Compesitional Model
56
5.4
Dialog Components are logical categories of a dialog, which include: C + context (or di- alog context); S -> dialog state(s); E -> evi- dence(s); A - action(s); R - the dialog re- sponse. Any dialog item X, can be mapped to a dialog component using mapping function g(). i.e., ( -> {C, S, F,, A, R). Table 2 provides an overview of the dialog components along with some sample dialog items that are mapped to it.
50
13K 30K 75K
100K
Size
Figure 3: Compositional accuracy of both the base- line and compositional model over a varying number of training data sizes. Each datapoint is rum across three indenpendently sampled test sets to account for variabil- ity - c.f. Fig. for atomic performance comparison.
Dialog Components
Sample Dialog Items
4.1 CESAR Framework
Dialog context between
C
utterances
CESAR adopts a structured approach to dia- log processing, often seen in task-oriented di- alogs (Hosseini-Asl et al., 2020). It generalizes the rigid structure of task-oriented dialogs to ac- commodate natural dialogs. A task in the CESAR framework is essentially a text-to-text task where both the input and the output are linearized 3 representation of multiple dialog components for- matted as per a specified structure. A high-level representation of a CESAR task is defined as fol- lows:
the two speakers.
dialog summary, speaker intent, etc.
S State of the dialog context.
Evidences that could be relevant for response.
retrieved knowledge, persona, etc.
E
Actions/constraints that the response has follow.
utterance style, dialog act, etc.
The next response by the assistant.
response utterance
Table 2: Dialog Components of CESAR with sample Dialog Items.
ICA - 1b
Flan-T5-xl model (Chung et al., 2022b) on two dif- ferent sets of training data, each with the same size, create two models: i) Baseline trained solely on the four atomic tasks, and ii) Composi tional- trained on a mixture of atomic and compositional tasks. For a fair comparison, we keep the training steps the same for both models.
= (CC ((m))) v.
(1)
grounding
where, the symbol delineates the input from the output, i.c. prompt - [output].
The input prompt in a CESAR task con- tains three prime components. First, represents task instruction. Second, C' includes the dialog context, which are previous utterances by the two speakers. Finally. A ('(())....... (((n),.........)) } is the multiset of dialog components* that is provided in the input in addition to the instruction I and the dialog context C.
Input:
In Fig. 3, we observe that Composi tinaal model outperforms Baseline model, regardless of the training size. This indicates that including compositional tasks in the training data is crucial for better performance. Interestingly, we also find that the presence of compositional tasks in the train- ing data positively affects atomic task performance ( Fig. 8). This analysis reveals the need for a scal- able generation of compositional dialog tasks and instructions gap that we fill with CESAR.
v E (S, F, A, R} represents the task output described by the instruction I and context C. Note that is never empty, as each task needs an output.
Output:
4 CESAR
For any given dialog interaction, we first define the notions of dialog items and dialog components:
We aim achieve order invariance in this linearization, using random sampling process described in $5.2. "A multiset is a set allowing multiple instances of items. *We limit the scope of this work to tasks with only one component as output. Naturally, there could be tasks (or in-
Dialog items are units of information pertain-

---

Speaker
Utt.
State
Evidence
Action
Dialog Context
User
0
{an]
(ua)) The digg act the fial
Input
Assistant
font
{an
(ra] Heg, do you like muxes?
uttrance in is cuostonn.
User
f
(Pag) The summary of can on
(1a) You, am big of the Barbrie song. band plays
Outpot
Assistant
{
(ea
fas axp)
Lober to iste intermteed inowing the act's music entoveats
(a) For any dialog item æzj- refers to it's turn number in the dialog and refers to it's identification within the
(rg) you What other ganves do you like?
Agus.
same dialog component, i.c., S, E, A, or R. for that turn.
Evidence
Fig. 4 provides an example for this setup.
(ego) oo songs bond sostalicc
Response
CESAR Task
Input
Output
[ta) like musal los particsrerty the Metallica they play
Action
C
i-D, A
Pay) The Tiail should have
IC-S
(rs.ra)
0-D, ()
#21
IC-A
(ra.ra)
0-D. ()
431
Pta)) The length of the ahoull be short
IC-E
O-D. O
6a1
ICS-A
(m.ra)
1-D. {
431
Figure 4: An example dialog with dialog items: terances (71s P2, P3, 74), state ($31 $32), evidence (641). actions (0.1, 0.2) as described in Table 3a.
ICE-A
(2.77)
1-D. (e4t)
ICA-R
(n.ra)r))
1-D. faa))
ICEA-R or ICAE-R
(marggra)
2-D. (en)
ra
ICSE-R or ICSE-R
2-D. {
rs
(b) For the given input dialog context (ri, ra} and output dialog response {ra} in Table 3a, provide some
sitional task: Atomic Tasks are either 0-D or 1-D tasks;Composi tional Tasks are any n-D Task with > 2.
example tasks defined under CESAR framework.
Table 3
To create any compositional task in CESAR, we define the following composition operation.
Let us provide a concrete example of the CE- SAR framework. Imagine that a user interacting with an Al assistant, which is shown in Table 3a. This dialog includes different dialog items. For ex- ample, evidence item 641 that is useful to construct the utterance r4 state item $32 that either infers information about utterance r3 or the complete dia- log history F r2, T3- Given these dialog items, we can construct several CESAR tasks, some which are demonstrated Table 3b. We also illustrate an example dialog interaction, including dialog items with their dialog component mapping in Fig. 4.
Definition 2 (Task Composition): For two i-D Tasks,
IC IC ( v and IC (AU -
where, |A| = =i= and i 2 1. we combine the two tasks to form an (3 1)-D Task:
IC IC (2, -
This composition operation allows the creation of arbitrarily complex compositional tasks, i.e., m-D Tasks with m 2 2, subjected to the availability of relevant atomic tasks.
4.2
CESAR Tasks
We now define an n-D CESAR task:
Our proposed CESAR framework can also in- corporate dialog items as reasoning elements in chain of thought (Wei et al., 2022). We present this extended formulation in Appendix E, but keep its experimentation future work.
Definition (n-D Tork); For any CESAR task of the form ICAA n dialog items A. L.. |A|
call the task n-D Task if there
Table 3b illustrates multiple CESAR tasks that are 0-D, 1-D, and 2-D tasks framed from the dialog items in Table 3a. Note that a 0-D task does not mean the input is empty, since CESAR task al- ways assumes task instruction I and a dialog context C (which can be potentially empty if there is no previous interaction).
5 InstructDial++
InstructDial is an instruction tuning benchmark for dialogue, which consists of a repository of diverse dialogue tasks in unified text-to-text format created from openly available dialogue datasets (Gupta et al., 2022). In this section, we first discuss new tasks and datasets added to this benchmark, resulting in the updated version, In- structDial++.. We then discuss how we map cach task in InstructDial++ to the CESAR framework.
Atomic vs. Compositional Task. CESAR, we categorize every task as either atomic and compo-
structions) where user could ask to generate multiple outputs, such as generating dialog summary along with an appropri- ate response. We defer such a setup for future work.

---

nead you Hed cmerent
Wyytees procened eniead pretiess
preveess
preves
sac. procet encrzae
evernt mice
wight mosettttn
proriao Jaie
Figure 5: Chord Wheel showing all atomic and compositional tasks in CESAR. Tasks colored in red newly added compared to the InstructDial dataset. Each edge between a pair of tasks indicates a new compositional task that combines them together.
5.1 New Tasks and Datasets
format ICA - + (Eq. (1)).
Knowing that scaling data in training benchmarks has positive impact on performance (Longpre al., 2023), we expand the InstructDial benchmark by incorporating 15 additional datasets and 42 new atomic (i.e. 0-D & 1 1-D) tasks (c.f. Table 15).
Prompt Design. Unlike InstructDial's approach of providing unique instructions for each task, we adopt a more general approach in crafting our in- structions which cover two main aspects: i) iden- tifying the dialog components (c.f. Table 2) to fo- cus during generation, and ti) specifying which component the generation is aimed at, such the example instruction: Provide the correct value for action field given the dialog context, state, and ev- idence fields. Due to the structural form of the instruction, we can programmatically generate in- structions for each compositional task.
The majority of newly introduced atomic tasks are derived from the inherent structure of the CE- SAR framework, which allows for multiple tasks to be defined and performed on the same dataset. For example, we create novel tasks (not comprehen- sive) listed in Table 3b using the dialog depicted in Fig. 4. Following the inclusion of the newly added tasks, the Instructdial++ benchmark consists of 86 tasks across 65 datasets - c.f. Fig. 5 and Table 15. The next step is to explain how we have mapped each Instructdial task to the CESAR format.
Generative and Discriminative Tasks. Despite the generative nature of each CESAR task, is important to note that our framework enables us to specify discriminative tasks as well. As an il- lustration, in the imottion_lassiifcaee provide the candidate emotions by incorporating them within the state component, e.g. 'State: Can- didate emotions are sad, happy, and mad. The emotion of the last utterance is:".
5.2 Mapping InstructDial++ to CESAR
task, we
To map each InstructDial task into the CESAR format, begin by assigning them CESAR task based on the input constraints and output type. None of the tasks in Instructdial++ incorporate rea- soning or CoT, leading to the simplified CESAR
CESAR framework enables 4 0-D tasks, and 12

---

1-D tasks. InstructDial++ incorporates downstream tasks for each O-D grounding task and for 10 of all the I-D grounding tasks as depicted in Fig. Please find examples of the input-output structure these tasks in Table 3b
I0-3
ICE
IC-A
ICR
Thare
K25-5
ICA-
ICS-A
ICE-
O-D Tasks, The top of Fig. depicts all Ø-D tasks. Each of these CESAR tasks clusters down- stream tasks with similar output objective. For instance, IC-S tasks involve categorizing, format- ting, or organizing information within the dialog context in more succinct or structured way, e.g. sugllo--summrizzationn IC-E, on the other hand, involves generating external knowledge useful in the dialog's 's context; both persona generation and knowledge generation are downstream tasks under this category. We believe the persona information user is better fed as 'evidence' rather than an "action' because it is external information and not a strict constraint to follow during generation, c.f Table 2. IC-A tasks are responsible for generat- ing actions to be followed in the response, such as keyword or intent prediction. Finally, IC-R is the collection of tasks that generate/select a response for given dialog context.
200
ICAS5
ICES-A
ICSAU
Figure 6: Comparsion CESAR tasks in InstructDial++ vs. their counterparts in InstructDial.
nation does not result in an infeasible task o.f. Fig. 10. One sample rule allows compo- sitions where the generation incorporates 2 ac- tions (e.g. beginswith cttpvitt_coorooledd_.mm controlled__eeeeraioon and cevoordconnrooled leeeedggnneennnnn creating ICAA-R task. For comprehensive explanation of these rules please see Appendix C
This results in 68 new downstream tasks defined on 7 2-D Cesar tasks c.f. Table 16 and Fig. 11. Fig. 5 shows each of these compositions as edges between atomic tasks.
Order Invariance of Grounding Items As dia- log items in prompt must be linearized, we adopt randomizing process ensure that our models are invariant in ordering the items. We place each section randomly inside the prompt, with two spe- cific rules: i) the instruction is always placed at the beginning. which is a common practice; it) the target section, referred to as y in Eq. (1), is placed at the end. The rest of the sections (C and A from Eq. (1)) are randomly positioned within the prompt.
1-D Tasks. 1-D tasks have the same categoriza- tion because their generation is also aimed one of S, E, A, R components in 0-D tasks, except they ground on an additional component other than the dialog context. For example, for ICA-R, the re- sponse generation task is additionally conditioned on a provided action, e.g. begins-wied with controlled generation or slot-value grounded generation. We also include edit generation under this category be- cause, unlike an IC-R task, it grounds on both the context and the previous version of the response to be corrected. Another 1-D CESAR task exam- ple is ICS-A which involves generating action of the upcoming response conditioned on the state of the current dialog context. An illustrative example is length-groundee grounded keyword prediction, where the generated keywords (for the response) are condi- tioned the length of the final utterance in the di- alog context. As depicted in Fig. 6, InstructDial++ incorporates more 1-D tasks along with the 2 that we borrow from InstructDial.
Experiments
6.1 Setup
Models. Throughout experiments, we utilize ChatGPT model gpt-3.5-turbo-16kk22
and five public models, namely T0-3B (Sanh et al., 2021) which is trained on mixture of downstream tasks, úi) DIAL-TØ and idi) DIAL-BARTO (Gupta et al., 2022), fine-tuned on InstructDial dataset and based on TØ-3B and BART (Lin et al., 2022), re- spectively. We train another baseline model based on InstructDial dataset using FLAN-xxi (Chung et al., 2022a) and name it with the same conven- tion the authors as iv) DIAL-FLAN-XXI. Our main model, v) CESAR-FLAN-xXL. is also trained using the FLAN-xxi model but on the Instruct- dial++ dataset rich in compositional tasks - c.f. Appendix for training details.
2-D Tasks. "After manually mapping all tasks of O-D and I-D nature, the CESAR frame- work selects and organizes viable 2-D tasks au- tomatically according to a small subset of pre- defined rules. These rules ensure the combi-

---

6.2 Tasks and Metries
the test splits of the corresponding datasets.
Atomic Tasks. Throughout experiments, we test models on eight atomic tasks - either individ- ually or as part of some composition: Begins With Generation (BW): Generate a response that starts with given phrase. Ends With Genera- tion (EW): Generate response that ends with a given phrase. Keyword Controlled Generation (KC): Generate a response which incorporates given set of keywords. Length Controlled Gen- eration (LC): Generate response with a certain length (short/medium/long)) Persona Based Gener- ation (PB): Generate a response based on given speaker persona. Knowledge-Bod Based Generation (KB): Generate response based some exter- nal knowledge. Edit Generation (EG): Edit re- sponse to make it coherent with the dialog context. Emotion-Groune
Atomic Task Performance. Table 4 presents the atomic task performance of each model. Even though we do not claim any discrete advantage in atomic task performance, we observe that the atomic task performance of the CESAR model proves to be comparable and even better in many tasks compared to the baselines. This aligns with preliminary experiments insights, o.f. Fig. 8.
Compositional Task Performance. Table 5 presents compositional task experiments results. Our model outperforms the baselines on every task composition. This indicates that good performance on atomic tasks does not necessarily translate to good performance compositions of those tasks, as evidenced by the widening gap between CE- SAR and baselines from atomic compositional tasks. We add two qualitative examples depicting atomic and compositional generation by CESAR, interested readers can find them in Fig. 12.
Generation (EMG): Generate a response that depicts a certain emotion. To main- tain standardized evaluation, we utilize the atomic task metrics implemented by Gupta et al. (2022) for all of our atomic tasks.
Generalization Experiment. To evaluate the ro- bustness capabilities of CESAR, we design a sim- plc training setup where each model trains only limited set of tasks using the smaller FLAN-x1 model. We then evaluate these models with various task compositions, both seen and unseen.
Compositional Task Metrics. We evaluate com- positional task performance on nine tasks, which are binary compositions of the atomic tasks. In InstructDial++, these compositions are available in the test set. For InstructDial, we manually create instructions for each task composition following the same formatting the InstructDial paper and generate new test sets accordingly.
We employ three models for this experiment. i) Atomic Model, solely trained on four atomic tasks: BW, KC, LC, and EG, representing the lower bound without any compositional training, ii) Naive Composer, trained on the same four atomic tasks well as three compositional tasks: BW+ KC+LC, and BW+LC. These composi- tional tasks are created by concatenating instruc- tions and constraints from individual tasks within the same prompt using the conjunction 'and'. To avoid generating infeasible tasks due to the lack of structural information inherent in the CESAR framework, we manually select the tasks that the Naive Composer combines (as explained in Ap- pendix G). Finally, we train another model using the iii) CESAR format, incorporating the same four atomic tasks and three compositional tasks as used for the Naive Composer.
For each compositional task performance, we report accuracy scores if possible. If the combined accuracy is unclear, we provide multiple metrics evaluating each dimension separately. For example. for persona-based ends with generation, because it is difficult to quantify how well the persona was used in the final generation, report both the "ends with accuracy" and Rouge-L metrics.
6.3 Results
this section, we present the results three main experiments. Each experiment's 's test set prompts for each model are formatted according to their training data. For TO-3B and DIAL-FLAN-xxl models, the prompts included natural phrases that explain the task. For DIAI-BARTH and DIAL-TO, we use the special tokens defined by Gupta al. (2022), and for CESAR-FLAN-xxl the prompts are automatically generated CESAR format by the framework itself. Despite their format, each test set is composed of the same data instances from
The results, presented in Table 6, demonstrate that the CESAR structure outperforms the Naive Composer in all seen compositions and most un- seen tasks and compositions. For the unseen task composition at the right-most column, incor-
II order to check these tasks are truly unseen we ex-

---

BW
EW
KC
LC
PB
KB
Model
Training
Acc
Acc
Acc
Acc
Blew-2
R-L
Bleu-2
R-L
TO-3B
Zero-shot
13.12
20.14
3.34
DIAL-BARTO
InstructDial
84,02
61.46
87.24
44.92
4.73
14.5
10.83
21.9
DIAL-TO
InstructDial
86.46
60.84
74.38
50.56
4,43
14
10
20.3
InstruciDial
81.4
62.12
86.04
53.26
4.23
13,66
9.32
18.96
CESAR-FLAN-xI
Dial+
14.5
Table 4: Evaluation results on atomic tasks. R-L stands for Rouge-L metric, best results for each column are bold.
BW
BW KC
BW +
EW
EW+
+
KC+
PB
EG
EW
LC
KC
LC
LC
EW
EW
Model
Training Set
EW
EW
Ace
Acc
R-L.
Acc
Acc
TO-3B
Zero-shot
2.38
4.39
5.75
2.95
6,92
8.55
5,4
11.56
22.5
29.9
DIAL-BARTO
DistructDial
69.01
77.2
38
46.3
11.8
16.65
82.8
50.2
85.2
83.2
DIAL-TO
InstructDial
72.9
72.8
42.5
49.1
27.4
35.2
76.9
46.1
84.5
77.2
PLA
Instrucil
70.1
78.53
44.25
58.04
30.15
44.4
82.53
49.03
89.3
85.33
CESAR-FLANxxi
IistructDial++
75.18
83.08
70.43
62.7
47.2
68.6
88.1
51.9
94.8
93.10
Table 5: Evaluation results on compositional tasks. R-L stands for Rouge-L. metric, best results for each column are bold.
Seen Compositions
Unseen Compositions
Unseen Tasks
BW+
KC+
BW+
FW+
EW+
BW+
EW+
EW+
Model
KC
LC
LC
KC
EW
EMG
KC
EW
ChatGPT
Acc
Acc
Acc
R-L
Acc
Acc
Acc
R-L
Acc
Acc
E.C. I.QQ.
Atomic Model (FLAN-xI)
73.88
43.8
50.3
82.2
69.2
36.1
54.21
20.1
49.3
22.6
60.9
1.76
Naive Composer (FLAN-xI)
74.08
44.55
52.07
82.5
73.69
35.24
54.21
20.35
50.9
24.2
63.0
1.91
CESAR (FLAAN-I)
76.07
44.8
54.4
83.1
83.7
31.89
56.91
21.75
54.6
22.3
56.4
1.99
Table 6:
Evaluation results on seen/unseen compositions, and unseen tasks. R-L stands for Rouge-L metric, E.C. stands for emotion classification, and R.Q. stands for response quality. Best results for each column bold.
porate chatGPT to classify and evaluate the gener- ated response's s emotion and the response's quality -cff Appendix F for details on the evaluation prompts used.
allowing the programmaticaly--nnuued generation of complex composite instructions. Moreover, we create new benchmark building on top of Instruct- Dial, adding new tasks and datasets and utilizing CESAR populate composite tasks. The new, In- structDial++ includes 63 datasets with 86 atomic and 68 composite task definitions. Our frame- work's compositional and atomic task abilitics are demonstrated in extensive experiments. These ex- periments reveal that our framework significantly improves the model's ability to handle complex prompts compared to previous approaches. No- tably, we discover that including composite data in the training set enhances compositional perfor- mance and improves atomic performance. Addi- tionally, we conduct robustness experiment and find that the CESAR structure outperforms the base- lines in majority of compositions as well as in un- seen tasks and compositions.
is important note that the manual selection of tasks to be composed by the Naive Composer overlooks an essential contribution of our frame- work: the ability to detect viable compositions. Therefore, this experiment only demonstrates how the robustness is affected by the CESAR structure and is not direct comparison between the Naive Composer and our approach.
7 Conclusion
We propose CESAR fill the compositional ca- pability gap between public models and irrepro- ducible state-of-the-arr, ChatGPT. CESAR modu- larizes downstream dialog tasks under one format
amined the FLAN collection and saw that out of 23 datasets it incorporates we only use DailyDialog within the test set. Moreaver we saw that there is minimal intersection between the version of DailyDialog used by FLAN (NIv2 corpus) and the original version we used. Thus we conclude there is only minimal chance of contamination, where in the worst case 100 test instance are seen by the model.
8 Limitations
Given its large scope, our work has yet to be able to delve into many promising avenues. We dedicate this section to discussing these to benefit future

---

research:
Victor S. Bursztyn, David Demeter, Doug Downey, and Larry Bimbaum. 2022. Learning to perform complex tasks through compositional fine-tuning of language models, ArXiv, abs/2210. 12607.
1) Datasets in InstrueiDial+ are not comprehen- sive. Even though we tried to increase the dataset and task scale to our best ability there are cer- tainly more datasets that InstructDial++ would ben- efit from such as Contrack (Ruckert et al., 2022), PRESTO (Goel et al.. 2023), elc.
Bill Byme, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Youns Kim, and Andy Cedilnik. 2019. Taskmaster- I: To- ward realistic and diverse dialog dataset. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- pages 4516-4525. Hong Kong, China, Association for Computational Linguistics.
2) Multi-tasking with non-dialog data is not done. Due the large scope of our work, we lim- ited the scope of datasets we focused on dialog- specific ones. Previous work has shown that adding non-dialog data might help (Zeng and Nic, 2021). 3) We have not explored negative conditions. A true composition should incorporate all logical compositions. The challenging part of the negative condition is its evaluation.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop Natural Language Processing for Conversational Al, pages 38-45, On- line. Association for Computational Linguistics.
4) We have only experimented using the exist- ing grounding features available in the datasets. This limits the kinds of controls that could be done. 5) Automatic metries are not necessarily robust. We try to mitigate this by choosing control signals that can be automatically measured.
Kushal Chawla. Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. 2021. CaSiNo: A corpus of campsite negotiation dialogues for automatic negotiation systems. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3167-3185, Online. Association for Computational Linguistics.
6) Our action fields are mostly lexical, and some are semantic. A comprehensive set of actions would be better.
Hailin Chen, Amrita Saha, Shafiq R. Joty, and Steven Hoi. 2022a. Learning label modular prompts for text classification in the wild. In Conference on Empirical Methods in Natural Language Processing.
References
Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. Just say no: Analyzing the stance of neu- ral dialogue generation in offensive contexts. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 4846- 4862, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jifan Chen, Yuhao Zhang, Lan Liu, Rui Dong, Xinchi Chen, Patrick Ng. William Yang Wang, and Zhiheng Huang. 2022b. Improving cross-task generalization of unified table-to-text models with compositional task configurations. ArXiv, abs/2212.08780.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. 2022. Training helpful and harmless assistant with re- inforcement learning from human feedback. ArXiv, abs/2204.05862.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1962-5074, Online. Association for Computa- tional Linguistics.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018 QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 21 2174-2184, Brussels, Belgium. Association for Com- putational Linguistics.
Pawel Budzianowski, Tsung-Hsiee Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. MultiWOZ- a large- scale multi molti-domain Wizard-of-oe dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gauray Mishra, Adams Wei

---

Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022a. Scal- ing instruction-finetune language models. ArXiv, abs/2210. 14.122.
Yao Dou, Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2021. Multitalk: A branchin di- alog testbed for diverse conversations. ArXiv, abs/2102.01263.
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Anmual SIGdial Meeting Discourse and Dialogue, pages 207-219, Saar- brücken, Germany. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret 7oph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022b. Scaling cntruuction-inetunee language mod- els. arXiv preprint arXiv:2210.1 11416.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt- agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau, 2018, Snips voice platform: embedded spoken language understanding system for private- by-design voice interfaces. ArXiv, abs/1805. 101.800111999 10190.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task- oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Saarbrücken, Germany, Association for Computational Linguistics.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A dataset for multi-turn dia- logue reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1406-1416, Online. Association for Computational Linguistics.
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document- -ocument-grounded grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 81 18-8188, Online. Association for Computa- tional Linguistics.
Zhuyun Dai, Arun Tejasvi Chaganty. Vincent Zhao, Aida Amini, Mike Green, Qazi Rashid, and Kelvin Guu. 2022. Dialog inpainting: Turning documents to dialogs. In International Conference on Machine Learning (ICML). PMLR.
Sarik Ghazarian, Behnam Hedayatnia, Alexandros Pa- pangelis, Yang Liu, and Dilek Hakkani-Tur. 2022. What is wrong with you?: Leveraging user senti- ment for automatic dialog evaluation. In Findings of the Association for Computational Linguisties: ACL 2022. pages 4 1194,2204 Dublin, Ireland. Association for Computational Linguistics.
Dorottya Demszky, Dana Movshovitz-Attias Jeongwao Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine- -ne-ggained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages Online. Association for Computational Linguistics.
Deepanway Ghosal, Pengfei Hong. Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021, CIDER: Commonsense inference for dialogue expla- nation and reasoning. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 301-313, Singapore and Online. Association for Computational Linguistics.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing JEMNLP-IJCNLA pages 4537-4546, Hong Kong, China. Association for Com- putational Linguistics.
Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022. CICERO: A dataset for contextualized commonsense inference in dialogues. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5010-5028, Dublin, Ireland. Association for Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur D. Szlam, lulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexan- der I. Rudnicky, Jason Williams, Joelle Pincau, Mikhail S. Burtsev, and Jason Weston. 2019b. The second conversational intelligence challenge (con- vai2). ArXiv, abs/ 190.00098....
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang. HyunJeong Choe, David Greene, Kyle He, Rattima Nitisaroj. A. A. Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, and Zhou Yu, 2023. Presto: A multilin- gual dataset for parsing realistic task-oriented dialogs. ArXiv, abs/2303.0899 .08554.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powee conversational agents. ArXiv, abs/1811 110144...
Karthik Gopalakrishnan, Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek -ukk 2019. Topical-Chat: Towards Mhowledge-Groundde

---

Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Choi. 2022. Soda: Million-scai dialogue distillation with social commonsense contextualization. ArXiv, abs/221 22./2212.100 10465.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskénazi, and Jeffrey P. Bigham. 2022. In- structdial: Improving zero and few-shot generaliza- tion in dialogue through instruction tuning. In Con- ference on Empirical Methods in Natural Language Processing.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang. and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 131 1-1316, Hong Kong, China, Association for Computational Linguistics.
Prakhar Gupta, Chien Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021. Dialfact: A benchmark for fact-checking in dialogue. ArXiv, abs/21 10.08222.
Charles T. Hemphill. John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to- end learning of negotiation dialogues. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Ehsan Ass, Bryan McCann, Chien- Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:20179- 20191.
Jiaqi Li, Ming I Liu, Min-Yen Kan, Zihao Zheng. 7ekun Wang. Wenqiang Lei, Ting Liu, and Bing Qin. 2020. Molweni: A challenge multiparty dialogues-basec machine reading comprehension dataset with dis- course structure. Proceedings of the 28th Inter- national Conference on Computational Linguistics. pages 2642-2652, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Chao-Chun Hsu, Sheng- Yeh Chen, Chuan-Chun Kuo, Ting-Hao Huang, and Lun-Wei Ku. 2018. Emotion- Lines: An emotion corpus of multi-partyy conversa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LRFC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Zhe Hu, Zhiwei Cao, Hou Pong Chan, Jiachen Liu, Xinyan Xiao, Jinsong Su, and Hua Wu. 2022. Con- trollable dialogue generation with disentangled multi- grained style specification and attribute consistency reward. IEEE/ACM Transactions on Audio, Speech, and Language P Processing, 31:188-199 199.
Xiujun Li, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018. Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems, ArXiv, abs/1807.11122
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised cross- task generalivation via retrieval augmentation. ArXiv, abs/2204.07931.
Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. GRADE: Automatic graph- enhanced coherence metric for evaluating open- domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9230-9240, Online. Association for Computational Linguistics.
Guisheng I. Liu, Yi Li, Yanqing Guo, Xiangyang Luo, and Bo Wang. 2022. Multi-attribute controlled text generation with contrastive-generaa discriminator. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 5904 5913,, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
and external-
Hang Jiang. Xianzhe Zhang, and Jinho D. Choi. 2019. Automatic text-based personality recognition on monologues and multiparty dialogues using atten- tive networks and contextual embeddings. In AAAI Conference on Artificial Intelligence.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. ArXiv, abs/1993.005566
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. 2019. Measur- ing compositional generalization: A comprehensive method on realistic data. ArXiv, 1b1.199..09713.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay. Denny Zhou, Quoe V Le, Barret Zoph. Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. ArXiv.
1.5/17

---

Online Kaixin Ma, Tomasz Jurezyk, and Jinho D. Choi. 2018. Challenging reading comprehension on daily con- versation: Passage completion on multiparty dialog. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- fional Linguislic:: Humun Language Technologies, Volume 1 (Long Papers), pages 2039-2048, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng. Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023, Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsuper- vised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group Discourse and Dialogue, pages 225-222255 235. 1st virtual meeting, Association for Computational Linguistics.
Dinesh Raghu, Shantanu Agarwal, Sachindra Joshi, and Mausam. 2021. End-to-end learning of flowchart grounded task-oriented dialogs. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681 681-707, Online. Association for Computational Linguistics.
Ramya Ramakrishnan, Hashan Narangodage, Mauro Schilman, Kilian Weinberger, and Ryan McDon- ald. 2022. Long-term control for dialogue gener- ation: Methods and evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 738-753, Seattle, United States. Association for Computational Lin- guistics.
Erine Merdivan, Deepika Singh, Sten Hanke, Johannes Kropf, Andreas Holzinger, and Matthieu Geist, 2020. Human annotated dialogues dataset for natural con- versational agents. Applied Sciences, 10:762.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguistics.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Ra- jen Subba. 2019. OpenDialKG: Explainable conver- sational reasoning with attention-based walks over knowledge graphs. In Pruceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 845-832 854, Florence, Italy. Associa- tion for Computational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guee guided dialogue dataset. In AAAI Confer- ence Artificial Intelligence.
Nikola Mrksić Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neu- ral belief tracker: Data-driven dialogue state tracking. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1777-1788, Vancouver, Canada. Association for Computational Linguisties.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. like fish, espe- cially dolphins: Addressing contradictions in dia- logue modeling, In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 1699-1713, Online. Association for Computational Linguistics.
Pedro Rodriguez, Paul Crook, Seungwhan Moon, and Zhiguang Wang. 2020. Information seeking in the spirit of learning: A dataset for conversational cu- riosity. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8153-8172, Online. Association for Computational Linguistics
Ulrich Ruckert, Srinivas Sunkara, Abhinav Rastogi, Sushant Prakash, and Pranav Khaitan. 2022. A uni- fied approach to entity context tracking social conversations. ArXiv, abs/2201.12409.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong

---

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica. Sheng Shen. Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Rose Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multi- task prompted training enables zero-shot task gener- alization. ArXiv, abs/21 10.08207.
Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 6462- 6481, Dublin, Ireland. Association for Computational Linguistics.
Xuewei Wang. Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang. Jingwen Zhang, and Zhou Yu. 2019. Per- suasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages (635-5649, Florence, Italy. Association for Computational Linguisties.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester. Nan Du, Andrew M. Dai, and Quoe V. Le. 2021. Finetuned language mod- els are zero-shot learners. ArXiv, abs/2109. 01162..
Karin Sevegnani, David M. Howcroft, Ioannis Konstas, and Verena Rieser. 2021. OTTers: One-turn topic transitions for open open-domain dialogue. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 2504-2504, Online. Association for Computational Linguisties.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting clicits reasoning in large language models. ArXiv, abs/2201.11903.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguisties. pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
NAN SHAO, Zefan Cai, Hanwei xu, Chonghua Liao, Yanan Zheng, and Zhilin Yang. 2023. Compositional task representations for large language models. In The Eleventh International Conference on Learning Representations.
Ashish Sharma, Adam S Miner, David C Atkins, and Tim Althoff. 2020. A computational approach to un- derstanding empathy expressed in text-based mental health support. In EMNLP.
Tsung-Hsice Wen, David Vandyke, Nikola Mrkšić, Mil- ica Gašić, Lina M. Rojas- Barahona, Pei-Hao Su, Ste- fan Ultes, and Steve Young. 2017. A network based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449) Valeciia, Spain. Association for Computational Lin- guistics.
Kurt Shuster, Jing Xu. Mojtaba Komeili, Da Ju. Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora. Joshua Lane, Morteza Behrooz, W.K.F. Ngan, Speneer Poff, Naman Goyal, Arthur D. Szlam, Y-Lan Boureau, Melanie Kam- badur, and Jason Weston. 2022. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. ArXiv, abs/2208.031a
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang. Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 602-69 631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Sandeep Subramanian, Guillaume Lample. Eric Michael Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. 2018 Multiple- attribute text style transfer. ArXiv, abs/181 abs//811.0005/. 1.00552.
Hao Sun. Guangxuan Xu, Deng Jiawen Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2021. On the safety of con- versational models: Taxonomy, dataset, and bench- mark. ArXiv, abs/21 10.08466.
Jing Xu, Da Ju, Margaret Li. Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dia- logue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2950-2000 2968. Online. Association for Computational Linguistics.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge data set and models for dialogue-based reading compre- hension. Transactions of the A Association for Compu- tational Linguistics, 7:217-231.
Megan Ung. Jing Xu, and Y-Lan Boureau. 2022. SaFeR- Dialogues: Taking feedback gracefully after conver- sational safety failures. In Proceedings of the 60th
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun

---

Xie. 2022. Tailor: prompt-based approach to attribute-basse controlled text generation. ArXiv, abs/2204.1333
Meeting the Special Interest Group on Discourse and Dialogue, Singapore and Online. Association for Computational Linguistics.
Zhengzhe Yang and Jinho D. Choi. 2019. FriendsQA: Open-domain question answering on TV show tran- scripts. In Proceedings of the 20th Annual SIGdial Meeting Discourse and Dialogue, pages 188-197, Stockholm, Sweden. Association for Computational Linguistics.
Qi Zhu, Christian Geishauser, Hsien-Chin Lin, Carel van Nickerk, Baolin Peng. Zheng Zhang, Michael Heck, Nurul Lubis, Dazhen Wan, Xiaochen Zhu, Jian- feng Gao, Mílica Gavsi'c c, and Minlie Huang. 2022. Convlab-3: flexible dialogue system toolkit based on a unified data format, ArXiv, 00s//211.17148..
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue-basse baaoou relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4927-4940, Online. Association for Computational Linguisties.
Sayyed M. Zahiri and Jinho D. Choi. 2017. Emotion detection tv show transcripts with sequence-based convolutional neural networks. In AAAI Workshops.
Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi. and Yejin Choi. 2021. Turing Ad- vice: A generative and dynamic evaluation of lan- guage use. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4856 4880, Online. Association for Computational Linguistics.
Yan Zeng and Jian-Yun Nie. 2021. simple and effi- cient multi-task learning approach for conditioned dialogue generation. In Proceedings of the 2021 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, pages 4927-4939.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Pupers), pages 2204-2213, 221,, Melbourne, Australia. Association for Computational Linguistics.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing precise and robust dialogue response eval- uators. In Proceedings the 58th Anmual Meeting of the Association for Computational Linguistics, pages 26-33, Online. Association for Computational Lin- guistics.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu. Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, Ioan Cristian Sto- ica. 2023. Judging lim-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05683
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayat- nia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur 2021. Commonsense- focused dialogues for response generation: em- pirical study. In Proceedings of the 22nd Annual

---

A Training Details
generating an output that encapsulates dual ac- tions. For instance, the combined tasks could in- volve geteeratidngneeaatiooon.
In line with Gupta et al. (2022), we create the train- ing data by sampling 5000 instances per atomic task from both the InstructDial and Instructdial++ datasets for their respective trainings. For Instruct- dial++, we additionally sample 1000 instances per compositional task for each dataset generated by the CESAR framework. Input sequences are set to maximum length of 1024 tokens, and output se- quences are set maximum length of 128 tokens. Longer sequences are truncated to these lengths, and shorter sequences are padded. We trained both the DIAL-FLAN-xlI and CESAR-FLAN-xx mod- els on eight A A100 GPUs, with a batch size of 10 per device and gradient accumulation steps set to 4. Both models are trained for two epochs, with a learning rate of 5e-05, and using the Adam optimizer (Loshchilov and Hutter, 2017).
controlled_generaiioon and key-
cord__ontrooled
In the rule structure, the key "common fields" represent modules that are recurrent across each task involved in the composition. For this rule, the common modules include the "dialog context" and "response". These shared elements are essential as they get assimilated into the final compositional task. Furthermore, the "target field" is pivotal as it indicates the expected output from both tasks involved in the composition. Hence, in this rule, the "response" field, which is anticipated from both ICA-R tasks, becomes the output of the culminat- ing ICAA-R task. It's also important to note that due the consistent structure prevalent across all ICA-R tasks, the framework ensures a seamless composition. This means that irrespective of the variant of ICA-R task combined, as long as the rule's s constraints are satisfied, the resulting compo- sition should be impeccable.
B Compositional Capabilities of Closed Source Models
Fig. 7 shows 3 more examples depicting GPT-4 and Claude- v2's compositional task capabilities. It's evident that each of these models demonstrates certain degree of compositional capabilities, showcasing their aptitude in complex language understanding and generation. However, there are certain scenar- ios where each exhibits inaccuracies or nuances that deviate from expected outputs. Amongst the three, GPT-4 consistently delivers the most reliable results whereas Claude v2 exhibits comparable per- formance, although it occasionally makes minor errors. GPT-3.5, on the other hand, tends to fall slightly behind its successors.
D Benhmarking ChatGPT on Atomic and Compositional Tasks
In this section we provide additional results by ChatGPT model complementing out main results tables. We have chosen not to include these results in the main paper for two reasons: (1) ChatGPT is a closed-access model, limiting scientific insights, and (2) Replicating ChatGPT results is not guaran- teed due to inaccessible architecture, parameters. and the API's lack of consistency across different runs.
To query ChatGPT, we utilize the same naturally formatted prompts as those used for the DIAL- FLAN model. An example of a zero-shot prompt can be found in Table 11. The one-shot experiments follow similar procedure but include an additional in-context example, maintaining the same prompt structure.
C Composition Rules
As explained in section $4, CESAR does not only unify dialog tasks certain prompt structure but it utilizes this structure combine these dialog tasks as compositional tasks automatically. This automation is achieved by defining specific rules, 10 as of the current version, that constrain which CESAR tasks can be composed together. The com- prehensive list of these rules can be found in Ta- ble 7.
Table 8 and Table 9 present additional results for the ChatGPT model in the atomic and composi- tional task evaluations, respectively. Results depict that ChatGPT actually has relatively lower perfor- mance compared to CESAR and DIAL-FLAN-xxl and CESAR-FLAN-xX models. However, upon analyses of the results we saw that these results maybe somewhat misleading. Because both DIAL.- FLAN-xx and CESAR-FLANT are trained on the training splits of the tested data they learned
The first rule delineated in the Table provides an illustrative example of how the CESAR com- positional tasks are defined. This particular rule is formulated to create ICAA-R compositions, im- plying that combines two distinct ICA-R tasks,

---

COMPOSITIONAL TASK CAPABILITIES CLOSED SOURCE MODELS
PROMPT
ChatGPT
GPT
Claude V2
Instruction:
Generate masporsa in the forn of queston the glven daig context. The resporse should have the beyword Everent" and should tms rephrased version the given orig nut rasporiog ses Original Response: Okay, lits go to Alps then!
How heading Everest instedd of Alps?
Everest
going
to go
together?
you
thinking
North.
Instruction:
Ganorata shart answer to the glvan dialog
context. The responsa nse should start with the phrase:
MMy" and should not insluce the keyword:
Ttavoite".
My tasto in music quite divarse, so it's hard to plok: just one favortto song.
My choics changes oftan, but zurrnnty loving "mmagine" John Lannon.
My favorita song is always changi changinna onjoy Istanin to wide varityy of music.
Dielog Contect: St: Hey, do music:
Which favorte??
Instruction:
Generate response to following diatog contedt. genemted resporse should be wecied should phrees warry
Wow. that's amaxing! Having productive meeting such great finiing. you want to have productive day at work, the best tp can give you to tocus on one task at time. Studies have shown that this aporoach can significantty Improve the quality of work So, just take things one step at time. and you'l be amsooed how much you toin azzcmmpiieh. Dorll wony about it. you this
ganenating the resporiee: shown increase work quality sign flcantly"
Also procided
fosus signiiicanlly It's
absoluteyy One
hewing more
things
great teoling,
days. on
When you dedicate
huge
Dialog Context:
full atiention
teok,
getting
S1: HL how was your day at work
lend builer moallss. It'r amazinng how much more accomplish about It!
way quality goes
92: went great, had very productive meeding.
51: Great, what tipe would you give to get productive day st work?
Dont
though
steadily
building good
totally got
Figure Qualitative examples showing compositional capabilities of some closed source models.
Rule
Task
Task 2
Composed Task
Common Dialog Components
Target Field
ICA-R
ICA-R
ICAA-R
r dc,
2
ICE-R
ICE-R
ICEE-R
de,
r
3
ICE-R
ICA-R
ICEA-R
de.
4
ICS-R
ICE-R
ICSE-R
de,
5
ICS-R
ICA-R
ICSA-R
de.
ICS-R
ICS-R
ICSS-R
de,
7
ICE-A
ICS-A
ICAES-A
dc, a
a
ICS-S
ICA-S
ICAS-S
de
9
ICA-A
ICS-A
ICASA-A
de
a
10
ICA-A
ICE-A
ICAEA-A
dc
Table List compositional rules.

---

BW
EW
KC
LC
PB
KB
Model
Acc
Acc
Acc
Acc
Bleu-2
R-L.
Bleu-2
R-L
ChalGPT (Zero-shol)
75.2
31.3
79.3
62.9
0.95
7.3
6.65
15.22
ChatGPT (One-shot)
83.35
33.0
70.13
72.6
1.82
10
6.08
14.5
Table 8: Complementary evaluation results on atomic tasks
c.f. Table
BW EW
BW +
BW
EW +
EW LC
KC
PB
EG +
KC
LC
KC
LC
EW
EW
Model
EW
EW
Acc
Acc
Acc
Acc
Acc
Acc
R-L
R-L
Acc
Acc
ChatGPT (Zero-shot)
30.8
65.4
50.4
9.05
27.9
13.3
9.37
45.7
ChatGPT (One-shot)
29.7
69.7
$7.2
23.7
12.5
34.47
15.4
11.1
64.7
68.8
Table 9: Complementary evaluation results compositional tasks best results for each column are bolded.
c.f.Table R-L stands for Rouge-L metric,
Instruction: In this task you given dialog and initial phrase. You need generate response which begins with the initial phrase.
Compostional Modil Atomic
Context:
Speaker I: Have any plans weekend, Tom ?
Speaker 2: Yeah, I going for hike the southern
Perfommnce
70-
Rocky Mountains.
Speaker I: , you go hiking often
INITIAL PHRASE: Not really,
Given this context generate response starts with the given initial sentence Answer:
64
Table 11: Sample Prompt used for ChatGPTTbased benchmarking
12K
30K
75K
100K
Data Size
Figure Atomic accuracy both the baseline and com- positional model over varying number of training data sizes. Each datapoint is run across three indenpendently sampled test sets account for variability
this song,'), and thus indirectly 'miss meiis-generates' the correct response. Another simple mistake it does for endswith seterivttggneeettionnnnnoo is that generates sentence that actually ends with the given phrase, however, does not stop the generation and adds another sentence to the response. e.g. for the task: 'The response should end with: song.', ChatGPT might generate: 'I like this song. Where can listen to it?".
certain spurious traits in the datasets and in the way we preprocess our data, For example, for the beginswith_ge seeeerittnnreattionnnnnomo task, the evaluation checks if the given initial phrase and beginming of the re- sponse are exactly the same. During tokenization we split punctuation with an additional space e.g. ('The response should start with: Yes, I love this song ,') but ChatGPT omits the additional space while generating the response i.e. ("Yes, I love
To examine the impact of incorporating more examples in the context and employing enhanced proprietary models, we reassessed the outcomes for three tasks with a reduced test set, seen in c.f. Table 10. Generally, GPT-4 performs notably better than GPT-3.5. Additionally, adding more contex- tual examples considerably boosts performance.
BW
BW
EW
Model
# Shots
EW
KC
KC
GPT-3.5
0
24.13
67.85
37.93
GPT-3.5
3
31.03
78.57
44.82
E Chain of Thought Potential in CESAR
GPT-4
0
31.03
92.85
65.51
We can extend CESAR also incorporate reason- ing supervision to the dialog tasks. This can be done by including Chain of thought elements into the dialog tasks. The CESAR task from Eq. (1) is modified as follows:
GPT-4
3
62.06
86.71
75.86
Table 10:
Evaluation results on three compositional tasks for smaller test set using more incontext samples, and GPT-4 API.

---

Speaker 1: Great, Peter may join us too! How good is the response: "Nice! It has been so long since have seen Peter, would be great to see him."? "? Provide your evaluation as a score between and 5, where for answers had quality, such as dull or inrelevant You helpful assistant that helps evaluate response given user query. Your job is tell if the response answers the user's query.
ICA A' IC (g(Ai)...... g(mm)) (o(X),,..
Given the dialog: Speaker I: Hi Mark, am going to the pool tomorrow, would you like to join?
g((,)) 1h
Speaker Hi Alice, sure thing. was thinking my self go rocently.
grounding
reasoning or CuT
(2)
Speaker I: Great, Peter may join us too!
Speaker Nice! has been so long since I have seen Peter, would be great to see him.
where, = (o(())...... s(x,)) is interpreted as the multiset of dialog components that can be used to reason about the primary output vb. In the tradi- tional CoT framework, the reasoning precedes the main output, thus represent the output sequence N/b.
Which of the following emotion does the final turn de- pict: happiness, fear, surprise, disgust, sadness, anger or no emotion?
happiness
Given the dialog: Speaker 1: What are you working on Sam?
Speaker 2; am still trying to solve the math problem from last week, think am close.
Speaker 1: see, I was able to solve it last night before going to bed.
Definition 3 (CoT-Generationn Any i-D Task, ICA- vp with A and 1, can be converted to its Col counterpart by shifting subset of dialog items from input ouiput, Le., IC (A), \ (Na)) ((Aa)) vb.
Which of the following emotion does the final turn depict: fear, surprise, joy, neutral, disgust or sadness, anger? neutral Given the dialog:
<Dialog Context>
Which of the following emotion does the final turn depict: <cand_ _enonddom_oon>>
Note that any i-D Task can be converted to any j-D Task where j < using Definition 3 repeatedly. For example, CESAR Task ICSSEA- R can be converted to -CA-SSER after three iterations of Definition 3 in shifting S, S, E dialog components.
Table 12: Prompt template used for ChatGPT evaluate emotion-groundes generation
You helpful assistant that helps evaluate response given user query. Your job is tell if the response answers the user's query.
These operations ensure full task coverage of any arbitrary CESAR task as per Eq. (1).
Given the dialog: Speaker I: Hi Mark, am going to the pool tomorrow, would
you like to join?
F
ChatGPT Evaluation Prompts
Speaker 2: Hi Alice, sure thing. was thinking my self go recently.
Earlier studies have shows that ChatGPT does good job in evaluating the overall quality of gener- ated text by language models (Zheng al., 2023). We used ChatGPT to evaluate the EW+EMG task in Table 6. Since emotion grounded generation is hard to evaluate we utilize ChatGPT and classify the emotions generated by each model and then calculate the accuracy of each model. Moreover because models may tend to generate the name of the emotion directly rather than infusing the emo- tion into the response (e.g. for response genera- tion grounded on happiness the generated response can be 'I am very happy!") we further generate qualitative scores by ChatGPT for each of model's responses.
answers, and 5 is for answers that are relevant to the dialog and are not generic.
Score: 4/5.
Given the dialog:
Speaker 1: What are you working on Sam?
Speaker 2: am still trying to solve the math problem from last week, think am close.
How good is the response: "I am going to bed."?
Provide your evaluation as a score between and 5, where I is for answers of bad quality, such as dull irrelevant answers, and 5 is for answers that are relevant to the dialog and are not generic.
Score: 1/5.
Given the dialog:
<Dialog Context>
How good is the response: "<response>"" Provide your evaluation as score between I and 5, where 1 is for answers of bad quality, such as dull or irrelevant answers, and 5 is for generic
We use in-context examples set each of these evaluation prompts. Table 12 and Table 13 depict template prompts for emotion classification and quality evaluation respectively.
answers that are relevant to the dialog and
Table 13: Prompt template used for ChatGPT evaluate response quality.
"While CoT is accommodated in the CESAR framework, we do not keep it in the scope of our experiments. Thus, in our experiments, A' = (Ø), i.c., A' is an empty set.

---

Instruction: In this task you will generate utterance given dialogue context and emotion and this task, you will be shown conversation context and phrasc. You need generate response the conversa- tion based the context which ends with the provided phrase.
Instruction: Provide the correct value for respouse fields given the dialog context and action fields.
Actions:
The emotion of the next turn should be: surprise The response should end with this sentenee: planning on taking me
Input:
Dialog Context You'll never guess what I work
Dialog Context:
today! Tickets to tonight's final NBA game.
Speaker I: You'l 'll never guess what won at work today
Emotion: surprise
Tickets to tonight's final NBA game.
Final Phrase: planning laking me!
Response:
The response with the given emotion is and Given this context and final phrase, the response is:
(b) Compositional task by CESAR
Compositional by Naive Composer
Figure 9: Figure depicting naive composition vs CESAR composition.
ATOMIC TASKS
COMPOSITIONAL
TASK
Instruction:
Instruction:
Instruction:
Ganerate the connect acts for based dalog context.
Ganeriaa cotaeert seced determined dalog ast.
Ganeraaee the convec based deiing centeat.
Generate coherent response based given dalig
Dielog Contect:
Buyo pless..
resenastion
G1 mesevation Blue Label,
the booking
82 the booking for today??
E2. booking for tacay??
Question:
possi bi
Question.
Question:
The pussible dialog are Inform Queation. The best option
The acosrtle dieloo stto are Inform Question best option Inform
preszite acte Inform Question. The best opdoon is. folo
Output: Inform
Output:
Output: Intorm. Yea, the booking for tanight.
Yee, the booking in for tonight.
Figure 10: Infeasible Compositional Task example resulting from naive composition of atomic tasks.
G Can Naive Conjunctions provide Complex Instructions?
task the right-hand side. This task is infeasible because the answer first task is incorporated in the input of the second task.
To get additional data for complex instructions, we could simply append single-task instructions and create complex ones. We use this method at Table 6 named as the Naive Composer to see how this compares to CESAR prompts. Fig. 9 shows two example prompts for the same task generated by the Naive Composer vs CESAR.
H Structured vs. Natural Prompts
The structured form of prompts that CESAR frame- work employs makes it easy to scale both atomic and compositional tasks up. However, this comes with potential caveat. One could argue that a model trained on CESAR formatted prompts might struggle to process natural human queries. In this section we discuss that the benefits that the CESAR framework brings can possibly be enjoyed without the structured form it imposes because the conver- sion between two is made casy by large language models, e.g. ChatGPT. We show example of this conversion using ChatGPT where we provide an in-context example in Table 14. This example indicates that given an in-context example, Chat- GPT model is able to convert CESAR format- ted prompt to a natural format that can be used model training.
It is important to note that for these experiments manually feed Naive Composer which tasks it should compose together. However, can use this naive method provide complex instruc- tions at a scale? We claim that the answer to this question is No. In dialog tasks, such naive compositions of tasks could lead to either non- sensical/invalid tasks or create tasks that are very rare and not useful. Let us look at an example Fig. 10. Here have two atomic tasks the left hand side, cialoo_act_prediicii alog_act
and di- Lllo__act_gronnllee grounded generattionn If one were to implement naive composer that would blindly combine two tasks whenever they have the same dialog context this would result in the infeasible

---

Sections of the Prompt
Content
Convert the following language model prompts from their structured format natural
Instruction
format that is likely to be written by a human. Follow the exact structure in each example:
Original Prompt:
Instruction: Fill in the correct value for the action field given the dialog context.
Dialog Context:
A: Hi, how was your day at work
B: It went great, had very productive meeting.
A: Great, what tips would you give to get productive day at work?
Action:
The dialog of the next response should be:
In context example
Natural Prompt:
What should the dialog act of the next response be?
Dialog context:
Dialog Context:
A: Hi, how your day at work
B: went great, had a very productive meeting.
A: Great, what tips would you give to get productive day at work?
Original Prompt:
Instruction: Fill in the correct value for the action field given the dialog context.
Dialog Context:
A: Hi, what is your favorite activity for week nights?
Query
B: like listen music while walking the city.
A: Great, what hand is your favorito?
Action:
The length of the next response should be:
Natural Prompt:
What should be the length of the next response?
Dialog context:
Output
A: Hi, what is your favorite activity for week nights?
B: like listen to music while walking the city.
A: Great, what band your favorite?
Table 14: Example prompt and output pair, utilizing the ChatGPT model to convert CESAR formatted prompt natural format. The first three rows are sections of the prompt provided to ChatGPT whereas the last row is the output generated by ChatGPT.
35
35
I retse lcd
na5-- yoe. and
Rass
CESAR Task Types
Figure 1:: CESAR task distribution InstructDial++
dataset

---

Prompt:
Prompt:
Instruction: Provide correct value response tields
Instruction: Provide the correct value for response fields
given the dialog context and action fields.
given the dialog context and action fields.
Input:
Input:
Dialog Context:
Dialog Context:
Speaker 1: What kind of place shall we rent ?
Speaker I: That's rough day for it. had my heart broken once by girl because her sister was jealous of us. Her sister claimed that she and had thing and my giriflriend (at the time) believed the lie.
Speaker 2: It should be close to the university Neither of us are good at getting up in the mornings and closer it is the later we can get
Actions:
Speaker 2: That's like rough day, the song
The response should start with this initial phrase: "Abso- lutely That's"
Speaker 1: I don't 't know that song, who sings it?
Actions:
The response should contain the following keywords: "thing" and "flat"
The final sentence of the response should be: "guy, named Paulini."
Response:
Response:
Output:
Output:
Absolutely. That's the most important thing and flat should be furnished.
It's by guy, named Paulini.
(a) Atomic Task example
(b) Compositional Task example
Figure 12: Qualitative examples generated the CESAR framework.

---

Task Name
Datasets
CESAR Task
MSRE2E (Li et al., 2018), MultiWOZ (Budzianowski ct al., 2018). DailyDialgg (Zhan et al., 2020), Curiosity-
are_predit
IC-A
Dialogs* (Rodriguez et al., 2020).
MultWWOZZBudZzanoossk et al., 2018), KVRET (Eric et al.. 2017). WOZ (Mrkšić et al., 2017). CAM- REST676 (Wen et al., 2017), MSRE2E (Li et al., 2018). IC-A
beeief_stateppredictooon
Frames (EI Asri et al., 2017), TaskMaster (Byme et al.,
2019), DSTCS-SGD (Rastogi et al., 2019)
Emotionl (Hsu et al., 2018), GoEmotions (Dem- szky et al., 2020), DailyDialog (Zhao et al., 2020), IC-A FriendsED* (Zahiri and Choi, 2017)
emotion_ emotion_predictionn prediction*
ATIS (Hemmphll et al., 1990). SNIPS (Coucke et al.,
2018). CLINC-150 (Larson al., 2019), HWU64 (Liu al., 2019), Banking77 (Casanueva al., 2020). Di-
innent_predictton
IC-A
aSafcty" (Sun et al., 2021)
Dialogue NLI (Welleck et al., 2019), DECODE (Nie
ali_ predictiinon
IC-A
al., 2021)
CaSiNo (Chawla et al.. 2021), Persuasion (Wang et al., 2019)
persuasion prediction
IC-A
DailyDialg (Zhao et al., 2020), WoW (Dinan et al..
IC-A
2018), EmpatheticDiallopes (Rashkin et al., 2019)
slot-dstc8 sgd, ATIS (Hemphill et al., 1990),
slot present prediction
SNIPS (Coucke et al., 2018). TaskMaster (Byrne IC-A et al., 2019), MSRE2E (Li et al.. 2018)
3lot-dste8_sgd. ATIS (Hemphill et al., 1990),
slot value_
ha___allu_ppedicctooo prediction
SNIPS (Coucke et al., 2018). TaskMaster (Byrne et al., 2019), MSRE2E (Li et al., 2018)
IC-A
speaker_predictions
Molweni (Li al., 2020)
IC-A
empathy (Sharma et al., 2020), DailyDialog (Zhao al., 2020), CONVAI (Dinan et al., 2019b), Emotion- IC-A
keyword prediction
Lines (IIsu et al., 2018), WoW (Dinan et al., 2018)
woz. MSRE2E (Li et al., 2018). Multi- WOZ (Budzianowski et al., 2018). DailyDialog (Zhao al., 2020). Dialoss (Rodriguez et al., 2020)
act act__eneration*
ICA-R
advice generratnntttion
TurinAAvvice (Zellers et al., 2021)
ICA-R
empathy (Sharma al., 2020), DailyDialog (Zhao al., 2020), CONVAI (Dinan et al., 2019b), TuringAd- vice (7ellers et al., 2021). Emotionl lines (Hsu et al.,
ICA-R
2018), WoW (Dinan al., 2018)
db based generation
MulnWOZ (Budzianowski et al., 2018)
ICA-R
TopicalChat (Gopalakrishnan et al., 2019), Emotion- Lines (Hsu et al., 2018). Persuasion (Wang al., 2019), CaSiNo (Chawla et al., 2021), DialogSum (Chen al., 2021), empathy (Sharma al., 2020), DailyDia- ICA-R log (Zhan al., 2020), CONVAI (Dinan et al., 2019h), EmotionLines (Hsu et al., 2018), WoW (Dinan et al., 2018)
eddi__eneeration
GoEmotions (Demszky et al., 2020), EmotionLines (Hsu al., 2018), DailyDialog (Zhao al., 2020), ICA-R
gentrationn
FriendsED* (Zahiri and Choi. 2017)
empathy (Sharma et al., 2020), DailyDialog (Zhao al., 2020), CONVAI (Dinan et al., 2019b), TuringAd- vice (7ellers et al., 2021). Emotionl.ine iines (Hsu et al., 2018). WoW (Dinan al., 2018)
hiiswith_controlllet generaaton
ICA-R
ATTS (Hemphill et al., 1990). SNIPS (Coucke et al.. 2018). 155 (I arson et al., 2019). HWU64 (Liiu et al., 2019), Banking77 (Casanueva et al., 2020), Di- aSafety" (Sun et al., 2021)
intent generation
ICA-R
empathy (Sharma et al., 2020), DailyDialog (Zhao et al., 2020). CONVAI (Dinan et al., 2019b), Emotion- ICA-R
leyword_contoolled_ennnnnii
Lines (Hsu et al. 2018), WoW (Dinan et al., 2018)
Dialogue NLI (Welleck et al., 2019), DECODE (Nic
nli *
ICA-R
al., 2021)
nontoxic_ feetoxic_feedbackcccc feedback_generationn
SaFeRDialogues (Ung et al., 2022)
ICA-R
CaSiNo (Chawla et al., 2021), Persuasion (Wang et al..
persuasion_generatton
ICA-R
2019)
cccovrry_generalee
SaleeDialogues (Ung et al., 2022)
ICA-R
DailyDialog (Zhao et al., 2020), WoW (Dinan et al..
response gecrussennnerttionne length*
ICA-R
2018), Empatheiiccialloges (Rashkin et al., 2019)
schema based generation
FloDial (Raghu al., 2021)
ICA-R
DSTC&-SGD (Rastogi et al., 2019), ATIS (Memphill et al., 1990), SNIPS (Coucke et al., 2018), TaskMas- ter (Byme al., 2019). MSRE2E (Li et al., 2018)
slot_ Boo__preent_geeeration
ICA-R
slot-dste8_ sgd. ATIS (Hemphill et al., 1990). SNIPS (Coucke et al., 2018), TaskMaster (Byrne ICA-R et al., 2019). MSRE2E (Li et al., 2018)
slot. value grwundedddddgggee generation*

---

Task Name
Datasets
CESAR Task
speaker generation
Molweni (Li al., 2020)
ICA-R
target controlled generation
OTTERS (Sevegnani al., 2021)
ICA-R
dialfact classification
Dial (Gupta et al., 2021)
IC-E
CONVAI (Dinan et al., 2019b), PersonaChal (Zhang
persona_gencration"
IC-E
et al.. 2018). FriendsPD* (Jiang et al., 2019)
MuTual (Cui ct al., 2020), CoQA (Reddy et al., 2018),
suswer_evidence_eennrrtio
QuAC (Choi et al., 2018), CIDER (Ghosal et al., 2021),
IC-E
WikiDial (Dai ct al., 2022)
document generation" "neument_generation"
doc2dial (Fong ct al., 2020)
IC-E
graph generation*
OpenDialKG (Moon et al., 2019)
IC-E
TopicalChat (Gopalakrishnan et al., 2019), WoW (Dinan
knowledge_ "nowlcdge_generation"
IC-E
et al.. 2018)
Mulual (Cui et al., 2020), CoQA (Reddy et al., 2018),
answer_generation
QuAC (Choi et al., 2018), CIDER (Ghosal et al., 2021),
ICE-R
WikiDial (Dai et al.. 2022)
CoQA (Reddy ct al., 2018), QuAC (Choi et al., 2018),
answer_selection
ICE-R
MuTuual (Cui al., 2020), CIDER (Ghosal et al., 2021)
Soda (Kim et al., 2022), Commonsense" (Zhou al.,
commonsense_ggoonneee generationn
ICE-R
2021)
document grounded generation
doc2dial (Feng et al., 2020)
ICE-R
graph based generation
OpenDralKGG (Moon et al., 2019)
ICE-R
TopicalChat (Gopalakrishnan et al., 2019), WoW (Dinan
knowledge grounded generattogeneratiionn
ICE-R
et al., 2018)
CONVAI (Dinan et al., 2019b). PersonaChat (Zhang
gusona_groundde_gemeeeee
ICE-R
2019) USR (Mchri and Eskenazi, 2020b), FED(Mchri and
al.. 2018). FriendsPD* (Jiang
Eskenazi, 2020a). GRADE (Huang al., 2020), HU-
eval. ranking
IC-R
MOD (Merdivan al., 2020), Anthropic* (Bai et al., 2022)
DailyDialog (Zhao et al., 2020), CONVAI (Dinan et al., 2019b), WoW (Dinan al., 2018). EmpatheticDia-
"isponse_generation
logues (Rashkin et al., 2019), OpenDialKG (Moon al., 2019), Anthropic* (Bai et al., 2022), Multitalk* (Dou et al.. 2021)
IC-R
MSRE2E (I.i et al., 2018), MoltiWOZ (Budzianowski ct al., 2018), DailyDialog (Zhao et al., 2020), Curiosity- Dialogs (Rodriguez et al., 2020)
act classification
IC-S
adesene
TuringAdvice (Zellers et al., 2021)
IC-S
MultiWOZ. (Budzianowski et al., 2018), KVRET (Fric et al., 2017). WOZ (Mrk3ić et al., 2017), CAM- REST676 (Wen al., 2017), MSRE2E (Li et al.. 2018), Frames (EI Asri et al., 2017). TaskMaster (Byrne et al.,
IC-S
(Rastogi et al.. 2019) Deal (1 (l.wwis ct al., 2017)
DSTCS-
deal dcal_preeset present
IC-8
discourse "
CIDER (Ghosal al., 2021), Molweni* (Li et al., 2020)
IC-S
EmotionLines (Hsu et al., 2018), GoEmotions (Dem-
emotion_tagging
szky et al., 2020). DailyDialog (Zhao al., 2020),
IC-S
FriendsED* (Zahiri and Choi, 2017)
eval. eval_binary
USR (Mchri and Eskenazi, 2020b). FED(Mehri and
IC-S
Eskenazi. 2020a). GRADE (Huang et al., 2020), HU-
MOD (Merdivan et al., 2020)
USR (Mehri and Eskenazi, 20206), FED(Mehri and
Eskenazi. 2020a). GRADE (Huang al., 2020), HU-
eval wwal_rating
IC-S
MOD (Merdivan al., 2020), ConTurE* (Ghazarian
ct al., 2022)
DailyDialog (Zhao et al., 2020). WoW (Dinan et al., 2018), Finparheticeiallouuss (Rashkin et al., 2019), OpenDialKG (Moon et al., 2019)
find incoherent utterance
IC-S
DailyDialog (Zhao et al., 2020), CONVAI (Dinan al..
2019b). WoW (Dinan al.. 2018). EmpatheticDia-
find_ mod__msssnn_uuteeeeae utterance
IC-S
logues (Rushkin et al., 2019), OpenDialKG (Moon al., 2019)
DailyDialog (Zhao et al., 2020), CONVAI (Dinan et al., 2019b), WoW (Dinan et al., 2018), EmpatheticDia-
find sod__sw utterance
IC-S
logues (Rashkin ct al., 2019), OpenDialKG (Moon et al., 2019)
ATIS (Hemphill et al., 1990), SNIPS (Coucke et al., 2018), CLINC-150 (Larson el al., 2019). HWU66 (Liu et al., 2019), Banking77 (Casanueva et al., 2020), Di- aSafety* (Sun el al., 2021)
anten_classificationn
IC-S
ATIS (Hemphill et al., 1990). SNIPS (Coucke et al., 2018), CLINC-150 (Larson et al., 2019), HWU64 (Liu et al., 2019), Banking77 (Casanueva et al., 2020)
intent present
IC-S

---

Task Name
Datasets
CESAR
Task
Dialogue NLI (Welleck al., 2019), DECODE (Nie
nli "ileasssfficatoon**
IC-S
et al., 2021)
person identification
FriendsRC* (Ma et al., 2018)
IC-S
CaSiNo (Chawla et al., 2021), Persuasion (Wang et al.,
persuasion trrpeess
IC-S
2019)
FriendsQA (Yang and Choi, 2019), CICERO* (Ghosal
question_anme answering
IC-S
et al., 2022), Dream (Sun et al., 2019)
Dtation_classificatiinn relation celation_present
DialogRE (Yu et al., 2020)
IC-S
DialogRE (Yu ct al., 2020)
IC-S
DailyDiaog (Zhao et al., 2020), WoW (Dinan et al.,
response_ henposssJengttheellsifictooooono classification
IC-S
2018), Empatheticiiaalouues (Rushkin et al., 2019)
slot-dstc8_sgee ATIS (Hemphill et al., 1990),
slot present*
SNIPS (Coucke et al., 2018). TaskMaster (Byme et al., 2019). MSRE2E (Li et al., 2018)
IC-S
slot-dstc8_sgee ATIS (Hemphill
al., 1990),
slot_tagging
SNIPS (Coucke et 2018). TaskMaster (Byme et al.. 2019), MSRE2E (Li al., 2018)
IC-S
speaker selection*
Molweni (Li al., 2020)
IC-S
samsum, DualogSum (Chen al., 2021), Soda* (Kim et al., 2022)
summarization
IC-S
ToxiChat (Baheti al., 2021), BAD (Xu al., 2021). Build it Break it Fix (Dinan al., 2019a). Di-
sxic_cllssification
IC-S
aSafety (Sun et al., 2021)
DailyDialog (Zhao et al., 2020), CuriosityDialogs* (Ro-
act classified generation*
ICS-R
driguez al., 2020)
DailyDialg (Zhao al., 2020), WoW (Dinan et al..
response_eenth_cllase cwwngssidcessiiieed_ee generationn
ICS-R
2018), EmppatheticDialloues (Rashkin al., 2019)
CaSiNo (Chawla et al., 2021), Persuasion (Wang et al.,
persuasion caanussionclassiiiedddeee dociifiee_geneeatiionn
ICS-R
2019)
DailyDialog (Zhao et al., 2020), FriendsED* (Zahiri and Choi, 2017)
emotion classified generation*
ICS-R
CAMREST676 (Wen et al., 2017)
ICS
crounddd_emotion emotion "tggiong act grt__groddddd
DailyDialog (Zhao et al., 2020)
ICS-S
act grounduddd wtl.gruuddd_mmmtiinnnnnoo emotion www.eeommpeediiicccc prediction
Diilo (Zhao et al.. 2020)
ICS-A
MSRE2E (Li et al., 2018), MultiWOZ (Budzianowski et al.,
2018), DailyDialog (Zhao et al.. 2020), CuriosityDialogs* (Ro-
ICS-A
driguez et al., 2020)
CONVaI 2019b), PersonaChat (Zhung et al., 2018), FriendsPD* (Jiang et al., 2019), MultiWOZ. (Budzianowsski et al.,
length_grounded grounded keyword *
ICS-A
2018), CuriosityDialoge
(Rodriguez et al., 2020)
emotion predicted emotion__pedicited
act classification
Daily (Zhao et al., 2020)
ICA-S
empathy (Sharma et al., 2020), DailyDialog (Zhao et al., 2020), CONVAI (Dinan et al., 2019h), TuringAdvice (Zellers et al., 1021). EmotionLines (Hsu et al., 2018), WoW (Dinan et al.,
beginswith grauuddehhhgroundddddee endswith prediction
ICA-A
2018)
CDn et al., 2019b), PersonaChat (Zhang et al., 2018),
ICA-A
FriendsPD* (Jiang et al., 2019)
MONNAl et al., 90)99,, PersonaChat (Zhang et al., 2018),
keyword_ grounndddr persona ceasonna_ecerationn
ICA-E
FriendsPD* (Jiang et al., 2019)
CONVAI al., 2019b), PersonaChat (Zhung et al., 2018),
versona_grrundeer Leyword__peediceee
ICE-A
FriendsPD* (Jiang et al., 2019)
CONVAI (Dinan 2019h), PersonaChat (Zhang al., 2018),
persona length "rediction*
ICE-A
FriendsPD* (Jiang et al., 2019)
Table 15: CESAR downstream atomic tasks incorporating OD and ID grounding tasks. Tasks marked with *e* are novel compared the InstructDial benchmark (The table starts in the previous pages).

---

Task Name
Datasets
CESAR Task
act generation
DailyDialog (Zhao al., 2020)
ICAA-R
edit generation
act generation
DailyDialgg (Zhao et al., 2020)
ICAA-R
emotion_ u Let_generation
DuilyDialog (Zhao al., 2020)
ICAA-R
endswith_ c generation act generatienonnn
DailyDialog (Zhao al., 2020)
ICAA-R
keyword__ontrolleed controlled_ generratoon act generation
DailyDialog (Zhao al., 2020)
ICAA-R
response generation length act get_eennttiion
MSRE2E (Li et al., 2018)
ICAA-R
slot_ toreerreeett__e present_generation act act_generation
MSRE2R (Li et al., 2018)
ICAA-R
slot value grounded generation advice_generation
TuringAdvice (Zellers et al., 2021)
ICAA-R
beginswith_conttolled generation Savvice_generation
TuringAdvice (Zellers et al., 2021)
ICAA-R
endswith_ endswith_controolee
heginswwth_conttrolldd controlled genilaatonnn
DailyDialog (Zhao al., 2020)
ICAA-R
act generatiiion
DailyDialog (Zhao et al., 2020). Emotion- Lines (Hsu et al., 2018)
heginswith_conneen generation
ICAA-R
emotion__enerran generation
CONVAI (Dinan al., 2019h), DailyDia- log (Zhao et al., 2020), EmotionLines (IIsu al., 2018), empathy (Sharma et al., 2020), WoW (Di- al., 2018)
beginswith controlled generation
ICAA-R
gonertteeeetlleeeeetttinnnnnn keyword__cntroll generation
CONVAI (Dinan et al., 2019h), DailyDia- log (Zhao et al., 2020). EmotionLines eHsu al., 2018), empathy (Sharma et al., 2020), WoW (Di- nan et al., 2018)
edit_ geieertennt
ICAA-R
beginswith_ce conertlleddeedeeeeeeeeeeo generation gentralled_eeneeaiooon
DailyDialog (Zhao et al., 2020), Emotion- Lines (Hsu al., 2018)
edit generation
ICAA-R
emotion generation
CONVAI (Dinan al.. 2019b). DialyDia- log (Zhao et al., 2020). EmotionLines (Hsu et al., 2018), empathy (Sharma et al., 2020), WoW (Di- nan al., 2018)
edit generatnnttionn
ICAA-R
endswith_controllld generation
generaoled
CONVAI (Dinan et al., 2019b), DailyDia- Iog (Zhan et al., 2020), Emotionl ines (Hsu al., 2018), empathy (Sharma et al., 2020), WoW (Di- nan et al., 2018)
edit generation
ICAA-R
keyword controlled generation
CaSiNo (Chawla al., 2021), Persuasion (Wang et al., 2019)
edit_generation
ICAA-R
DailyDialog (Zhan et al., 20200), WoW (Dinan et al. 2018)
adit_generation
ICAA-R
response generation length emotion generation
DailyDialog (Zhao et al., 2020), Emotion- Lines (Hsu et al., 2018)
ICAA-R
endswith_comm www.hiileeoottoleeeeeoeoo -enerrationnnn
DailyDialog (Zhao et al., 2020). Emotion- Lines (Hsu et al., 2018)
ICAA-R
keyword keyword_conttroled generation

---

Task Name
Datasets
CESAR Task
CONVAI (Dinan al., 2019b), DailyDia- log (Zhan et al., 2020), Emotionl lines (Hsu et al., 2018), empathy (Sharma et al., 2020). TuringAd- vice (Zellers et al., 2021), WoW (Dinan et al., 2018) DailyDialog (Zhao et al., 2020)
endswith_ htwprith_cootoolee_eeeeeee generation
ICAA-R
wwwtrololoddeeeeeooc generation c
endswith_ ctwpriihhooooooeeddeeeeee controlled__enneation
ICAA-R
emotion_elasseeeeeeeeeeeeeeeeeee httpg_alassfeedeeeeeeeeee generation
CONVAL (Dinan et al., 2019h), DailyDi-- log (Zhao et al., 2020), Emotionl Lines (Hsu et al., 2018), empathy (Sharma et al., 2020), WoW (Di-
endswith controlled generation
ICAA-R
nan et al., 2018)
controlled generation nontoxic Jeedback generation
deyyoor_controlleed
SaFeRDialogues (Ung et al., 2022)
ICAA-R
recovery generation
DailyDialog (Zhao et al., 2020), WoW (Dinan et al., 2018)
susponse_eeneaationn
ICAA-R
beginswith_comee controlled generation response generation length
DailyDialog (Zhao al., 2020)
ICAA-R
emotion classified generation
response ceareseeggneattioneeee length
DailyDialog (Zhao al., 2020)
ICAA-R
emotion generation
DailyDialog (Zhao et al., 2020), WoW (Dinan et al., 2018)
Leannseeeeneration_ooo lenghh
ICAA-R
endswith_ badswith_conttrllld___oo generation controlled_eenerattoon
DailyDialog (Zhao 2020), WoW (Dinan et al., 2018)
ICAA-R
k generation slor_ prrppeet__eeerratiooottt
MSRE2E (Li et al., 2018), slot-dste8 sgd
ICAA-R
alor_valueeeee ceatluu_groundddgeeee generation act classified classified_gencratiion
DailyDialog (Zhao al., 2020)
ICSS-R
emotion classified generation classified generation
DailyDialog (Zhao 2020)
ICSS-R
response length classified generation
CONVAI (Dinan et al.. 2019b).
Per-
persona_groundet lew.gggdddddd_ennhhhhheee prenit_prediition
ICEA-A
sonaChat (Zhang al., 2018)
beginswith grounded length prediction act generation
MultWOO (Budzianwski 2018)
ICEA-R
db_ bb__base___u generation
TopicalChat (Gopalakrishnan al., 2019), WoW (Dinan et al., 2018)
edit_generation
ICEA-R
knowledge grounded generation edit generatteniin
CONVAI (Dinan et al., 2019h)
ICEA-R
persona grounded generation saddsith_conttrolledeggg generation
CONVAI (Dinan 2019b)
ICEA-R
persona_grounde generation g knowledge_graunddd generationnnnnn
WoW (Dinan et al., 2018)
ICEA-R
haginswith_conttllled,, controlled genellatinnnnrationnnn knowledge grounded generation
WoW (Dinan et al., 2018)
ICEA-R
endswith controlled generation
knowledge stowledge_groumded.oo groumded_gennraaion
WoW (Dinan et al., 2018)
ICEA-R
keyword__controlled controlled generation lisismaaggounded_eeeeeritt
CONVAI (Dinan et al., 2019b)
ICEA-R

---

Task Name
Datasets
CESAR Task ICEA-R
persona grounded generation
CONVAT (Dinan al., 2019b)
keyword_ controlled generation
WoW (Dinan et al., 2018)
ICEA-R
knowledge grounned generation
CONVAL (Dinan et al., 2019b), Per-
persona_ grronne_roundddd.mmm
ICES-A
sonaChat (Zhang et al., 2018)
length grounded keyword Lreucton_reuiccion act classified generation
DailyDialog (Zhao al., 2020)
ICSA-R
uct_classiiedd generation generaaion
DuilyDialog (Zhao al., 2020)
ICSA-R
endswith controlled coneralle_genecation act classified generation
DailyDialog (Zhao et ul., 2020)
ICSA-R
keyword controlled generation
DitoossyDialos*** (Rodriguez et al., 2020), Dai- lyDialog (Zhao al., 2020)
act_gencration
ICSA-R
Lel_lasiffed_geeneerti act generatiecnnn
DailyDialog (Zhao et al., 2020)
ICSA-R
emotion
emotion_classiioenn classified generation act generation
DailyDialog (Zhao et al., 2020)
ICSA-R
response_ Laapee__enttthhhheeeeeeeee cl beginswith__onnenn controlled cowttrolllld_geeeeettoo generation
DailyDialog (Zhao al., 2020)
ICSA-R
act classiffed generation beginswith_ legenswith_contollledddddd generation
DailyDialog (Zhao al., 2020)
ICSA-R
emotion classified generation geiieeeentttion
DailyDialog (Zhao al., 2020)
ICSA-R
_et_seasifeeddecc generation cenertiieed_ggeeaattoonn odit geieegggtttionn
DailyDialog (Zhao et al., 2020)
ICSA-R
emotion_elasee generration edit generation
DailyDialog (Zhao et al., 2020), WoW (Dinan et al., 2018)
ICSA-R
response length classified generation emotion_ geneeatton
DailyDialog (Zhao et FriendsED* (Zahiri and Choi, 2017)
2020),
ICSA-R
Phottion_llassffiee generation o
DailyDialog (hho et al., 2020), WoW (Dinan et et., 2018)
haddwith_controlledd generation g
ICSA-R
response jesponse_Jength classified glaertfie__enneattionn keyword controlled generation
DailyDialog (Zhao al., 2020)
ICSA-R
CaSiNo (Chaawlacttl, 2021), Persuasion (Wang et al., 2019)
Persuasion
ICSA-R
edit generation
CaSiNo (Chawlaetal. 2021), Persuasion (Wang et al., 2019)
Persuasion classified generation
ICSA-R
Persuasion_genern generattoo susposse_generratonnleeee
DailyDialog (Zhao al., 2020)
ICSA-R
act wwwssified.geneneeeeee generation

---

Task Name
Datasets
CESAR Task
DailyDialog (Zhao et al., 2020), EmpatheticDia- logues (Rashkin al., 2019), WoW (Dinan al., 2018)
suppase_generation_Looo length
ICSA-R
DailyDialog (Zhao et al., 2020), WoW (Dinan et al., 2018)
length
ICSA-R
beginswwtthcconttolleddd htwpaiseeeeegghhhasiieeeeeeeeeee
generation
DailyDialog (Zhao al., 2020)
ICSA-R
emotion claiiiiio__asssieeeeeeee generation
leaponss_length_llaeee genertiiiicnnnneniiioon
DailyDialog (Zhao al., 2020)
ICSA-R
emotion getrriatnnneettionnnn
DailyDialog (Zhao et al., 2020), WoW (Dinan et al., 2018)
ICSA-R
k wtwpeooooooddeeeeeeeeeee generation
cnowledg__gronndeeeennneee
WoW (Dinan 2018)
ICSE-R
response_ gagponss_eutttcclllsiieeeegeeeee
Table 16: CESAR downstream compositional 2D tasks (The table starts in the previous pages).


Reviews:
[{"review_id": "HcIaomlZql", "reviewer": "Reviewer_Lkwr", "summary": "", "questions": "A) The section starting from L184 is important to motivate the need for compositional tasks, however the details regarding this preliminary experimentation are not provided.\n\nB) In L401 is it mentioned that CESAR selects and organizes 2-D tasks automatically, however it is not clear how this is implemented and how are infeasible tasks identified\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough feedback from Reviewer \"Lkwr\". It's encouraging to know that they recognized **the critical gap** our paper addresses, viewed **CESAR as a well-defined framework**, and saw the **InstructDial++ dataset as a meaningful contribution**.\n\nCESAR centers on the generation of prompts and instructions within dialogues. Our principal innovation lies in its capacity to craft compositional task prompts, an aspect previously overlooked in prior studies in this domain. \n\nBelow, we address each of Reviewer's concerns:\n\n> One of the primary motivations of the paper is to bridge the gap between closed LLMs and open LLMs, hence there should have been more comparison and evaluation between CESAR and GPT-3.5-turbo in the main paper. The authors have tabulated some results in the appendix section and they have carefully analyzed why the numbers might be deceiving, however, it would be better to show several qualitative examples across CESAR and GPT-3.5-turbo and analyze trends in performance, it is not clear whether the gap has been bridged or not.\n\n- We agree with the reviewer that adding such qualitative examples could assist the reader in better comprehending the distinctions between CESAR and ChatGPT.\n- Following the reviewer’s suggestion In the final version we will add qualitative examples from GPT3.5, GPT4 and Claude models. \n- We will also enrich the quantitative results with more shots during prompting and also using GPT4 API. For a sample table with preliminary results please see the answer to Reviewer \"8BXP\".\n- We will also provide a fine-grained analysis of areas of vulnerability causing publicly available models (i.e. CESAR in the context of this work) to lag behind closed models (GPT models and Claude) in tasks involving composition. We add one such sample here to give a better idea of what to expect in the final version:\n\n**Prompt**: \n\n“Please generate a response to the following dialog context.\nThe response should start with this initial phrase: “Absolutely . That’s”.\nThe response should also contain the following keywords: “thing” and “flat”.\n\nDialog Context:\n\nSpeaker 1: What kind of place shall we rent ?\n\nSpeaker 2: It should be close to the university . Neither of us are good at getting up in the mornings and closer it is , the later we can get up.\n\nResponse: ”\n\n**CESAR**: Absolutely. That’s the most important thing and flat should\nbe furnished.\n\n**GPT 3.5**: “Absolutely. That’s a valid point. Being near the university is definitely a crucial thing to consider, especially given our morning habits. The closer we are, the more likely it is that we can avoid rushing in the early hours. Another thing to think about is the type of place itself. Would a flat be suitable for us, or are you leaning more towards a house? Let's make sure we find a place that aligns with both our proximity needs and our preferences for the kind of living space.”\n\n**GPT 4**: Absolutely. That's the smartest thing to consider. A flat near the university would be perfect for us.\n\n**Claude v2**:  Absolutely. That's a flat thing to consider when looking for a place near the university. Being close by means we can sleep in a bit later and still make it to class on time.\n\n- Note that this prompt incorporates a compositional task that CESAR has been trained on whereas other models are prompted in zero-shot. In the final version of the paper, we shall also try unseen tasks that CESAR has not seen before.\n\n> In L213 it is stated that scalable generation of compositional tasks is one aspect that CESAR aims to solve, however, it is not clear in the paper how the compositional tasks are generated in a scalable fashion using CESAR\n\n> In L401 is it mentioned that CESAR selects and organizes 2-D tasks automatically, however, it is not clear how this is implemented and how are infeasible tasks identified\n\n\n- **Scalable Task Composition via Automation**: As explained in section 4.2, CESAR does not only unify dialog tasks in a certain prompt structure but it utilizes this structure in order to combine these dialog tasks as compositional tasks in an automatic manner. \n- This automation is ensured by defining specific rules (10 in the current implementation) that limit what kind of CESAR tasks are composable together. Our framework utilizes these rules in the Task composition Operation (Definition 2 in the paper) in order to create only feasible compositional tasks. We shall share the complete list of these rules in the final version of the paper. One sample rule is as follows which defines the rule to create ICAA-R compositions where the generation incorporates 2 actions (e.g. beginswith_controlled_generation+keyword_controlled_generation)\n    - (\"ICA-R\",\"ICA-R\"):{\"composition\": \"ICAA-R\", \"common_fields\": [\"dialog context\",\"response\"], \"target_field\":\"resonse\", \"Instructions\": [AR_INST]},\n    - “common fields” stands for modules in each task that is common and should be incorporated in the final compositional task.\n    - “target field” represents the output expected in both tasks so it should also be the output of the final compositional task.\n    - “final instructions” represent a list of instructions that may be used for the composition. These instructions are manually written for each compositional CESAR task only once (there 7 such tasks in total).\n    - These are simplified instructions (prompts) that direct the model to utilize the “common fields” -> This ensures that the automated combination is grammatical and there is no need for fine-grained human intervention. \n    - Since we share the same structure in all ICA-R tasks our framework ensures that this composition should work flawlessly for any type of composition that abides by the rule.\n- The scalability proposition stems from CESAR's inherent capacity for automation. \n\n**Infeasible Tasks in Composition**: Without this automation, incorporating new tasks would entail manual identification by human annotators of potential compositional links with prior tasks, followed by the creation of new prompts, again carried out by human annotators. The manual identification would be necessary because composing every task together would result in an infeasible task (see Figure 11 for such an example). \n- Moreover even generating only feasible tasks with the assumption of an oracle, this naive composition results in worse results when compared to the CESAR framework (see Table 5 for results).\n\n**How does CESAR avoid Infeasible Tasks?**: In the section before, it was highlighted that CESAR sidesteps tasks that aren't feasible by leveraging 10 specific manually written rules. \n- Essentially, these rules specify which 1-D CESAR tasks can be combined. The primary advantage of CESAR lies in its ability to use these high-level rules to combine 86 distinct 1-D tasks, yielding 68 viable 2-D tasks. In the absence of this structured approach and rules, human annotators would be burdened with conducting 86x86 manual evaluations to ascertain the compositionality of these two tasks.\n- We acknowledge that in its current form, it is hard to understand from the paper how infeasible tasks are avoided by CESAR framework. We shall edit the relevant parts of the paper to ease understanding in the final version.\n\n> The section starting from L184 is important to motivate the need for compositional tasks, however, the details regarding this preliminary experimentation are not provided.\n\n- We understand the reviewers' concern that there are not enough experimentation details in this section. Our original intention was to only provide a brief section highlighting our motivation for the need for compositional prompts and delve into the actual experiments (with details) in Section 6. However, we agree that the section details can be made more enriched. We shall, thus, revise the section to make the experimental setup more explicit.\n\n- We shall add the details in the final version of the paper. This shall include but should not be limited to sample manual prompts used for compositions, and the amount of data used for training and testing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well written and well structured. Although the introduction and description of CESAR were somewhat challenging to follow due to many new terms being introduced, the accompanying tables and figures helped in following along. \n\nThe authors have identified a critical gap between performance of closed and open-source LLMs in task-oriented dialogues and clearly delineated their motivations with preliminary experiments to support their hypothesis that compositional instructions can help improve performance on complex dialog tasks.\n\nThe CESAR framework is well defined and designed to be scalable across different tasks within task-oriented dialogue.  They have explained how n-D tasks are defined, how discriminative tasks can be incorporated and how they ensure order-invariance while linearizing their inputs\n\nThe InstructDial++ dataset is a meaningful contribution (the authors have committed to sharing the dataset on Github) and results from Table 3 - 4 indicate that fine-tuning FLAN model on InstructDial++ shows performance gains over a FLAN model trained on InstructDial\n\nThe quantitative results indicate that CESAR-FLAN performs better than existing approaches on a good number of atomic tasks and seems to out-perform other approaches in compositional tasks. \n", "reasons_to_reject": "One of the primary motivations of the paper is to bridge the gap between closed LLMs and open LLMs, hence there should have been more comparison and evaluation between CESAR and GPT-3.5-turbo in the main paper. The authors have tabulated some results in the appendix section and they have carefully analyzed why the numbers might be deceiving, however it would be better to show several qualitative examples across CESAR and GPT-3.5-turbo and analyze trends in performance, it not clear whether the gap has been bridged or not\n\nIn L213 it is stated that scalable generation of compositional tasks is one aspect that CESAR aims to solve, however it is not clear in the paper how the compositional tasks are generated in a  scalable fashion using CESAR\n", "questions_to_authors": "A) The section starting from L184 is important to motivate the need for compositional tasks, however the details regarding this preliminary experimentation are not provided.\n\nB) In L401 is it mentioned that CESAR selects and organizes 2-D tasks automatically, however it is not clear how this is implemented and how are infeasible tasks identified\n\n\n", "paper_topic_and_main_contributions": "The paper notes that closed-access large language models such as the ones used in chatGPT are much more capable than the open-source LLMs and it is reasoned that this gap can be bridged by training the open LLMs on large-scale complex demonstrations, specifically compositional instructions: instructions where output is grounded on a set of two or more constraints\n\nThe paper proposes a framework called CESAR through which the authors clearly define a generalized representation of providing instruction and dialogue components as inputs to a language model. They further argue that improving the availability of demonstration data invariably leads to better instruction tuning performance therefore they augment the existing InstructDial dataset with more datasets and tasks. The resulting dataset is called InstructDial++\n\nThe paper finetunes a FLAN model on this augmented dataset and it is shown that when compared to benchmark models used in InstructDial, as well as FLAN model trained on InstructDial the CESAR approach performs better. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3 is not clear, what metric does performance indicate and what does data size indicate, are they number of utterances or dialogues as a whole. The authors could provide more details on this preliminary experimentation that they performed. \n"}}, {"review_id": "BTcyqGtLuL", "reviewer": "Reviewer_gUjh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed unified framework presents a novel idea.\n2. The motivation and details of the approach have been described in great detail.\n3. Abundant experiments have validated the authors' method.", "reasons_to_reject": "I haven't identified any significant shortcomings of the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new framework called CESAR for instruction-based multitasking in dialog applications. It addresses the challenge of handling complex instructions by automatically generating compositional tasks without manual intervention. The authors enhance the InstructDial benchmark with new datasets and tasks, utilizing CESAR to create compositional tasks with multiple constraints. The resulting benchmark, InstructDial++, comprises 86 basic tasks and 68 composite tasks across 63 datasets. Through experiments, the authors demonstrate that CESAR's compositional tasks enhance performance on both seen and unseen compositions, bridging the gap between publicly-available dialog models and proprietary AI assistants.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y7mNHijUIB", "reviewer": "Reviewer_dek5", "summary": "", "questions": "See 3. and 4.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback from Reviewer \"dek5\". We are happy that they thought the **motivation of the paper was clear and sound**.\n\nCESAR centers on the generation of prompts and instructions within dialogues. Our principal innovation lies in its capacity to craft compositional task prompts, an aspect previously overlooked in prior studies in this domain. \n\nBelow, we address each of the Reviewer's concerns:\n\n> While the paper introduces 68 new downstream tasks based on seven 2-D Cesar tasks, it mainly offers overall performance. This obscures the individual quality and contribution of each task. There's a possibility that only a few high-quality tasks drive the improvements, while the majority add little value.\n\n- **CESAR’s core contribution:** The central concept of the paper revolves around the incorporation of task composition rather than a mere increase in the number of tasks. Such compositions are not explored in any of the previous Instruction-tuning benchmarks, such as Flan, OPT-IML, etc., and particularly in dialog benchmarks, such as - InstructDial and DialogStudio.\n\n- **Data Mixing Strategy:** As our core proposal is in the automatic creation of compositional dialog instructions, we follow prior work in our data mixing strategy. The reviewer is correct in their understanding that certain datasets, and tasks could be overpowering the others. To alleviate this, previous work, such as [FLAN](https://arxiv.org/abs/2109.01652), [InstructDial](https://arxiv.org/abs/2205.12673), and [Super-NaturalInstructions](https://arxiv.org/abs/2204.07705), limit the overall quantity of samples per task. This restriction aims to ensure a balanced distribution of data. Moreover, in the [Flan-collection paper](https://arxiv.org/abs/2301.13688) authors highlight that while they performed ablations across prompt themes -- like CoT, templated, etc., they followed \"practitioner’s intuition\" and performed data balancing using max thresholds. We adopt the same strategy.\n\n- **Principled Data Mixing:** Identifying optimal task/data mixtures is a research area on its own and is not the primary focus of our work. For example [Chen et al.](https://arxiv.org/abs/2307.14430) show a systematic way to create a mixture in their recent paper.\n\n> The methodological clarity regarding CESAR's task combination into coherent instruction prompts remains ambiguous. It's essential to discern the reliability of these instructions and ensure task diversity, preventing overlapping or overly similar tasks.\n\n- **Reliability of Prompts:** The CESAR framework ensures reliable combined task instructions through its inherent structure. We emphasized this aspect throughout our work. Our main goal was to automate prompt generation for combined tasks. To achieve this, we developed a modular prompt design, which is shown in Figure 10-b in the appendix. This figure demonstrates a sample prompt created using the CESAR framework.\n\n- **Task/Prompt Diversity:** We agree with the reviewer that the paper lacks a specific analysis of task/prompt diversity. We value this feedback and intend to incorporate it in the paper's final version. Building on our previous efforts, we are eager to present an advanced diagram illustrating the task categories utilized by the CESAR framework, accompanied by the tasks showing the diversity of InstructDial++.\n\n> The paper doesn't sufficiently differentiate itself from FLAN-T5, which already encompasses several dialogue tasks and detailed task design. It would be great if the paper could discuss the differences.\n\n- The Flan-T5 introduces a variety of datasets and tasks in the flan collections, but only a handful pertains to conversational AI. Another such example is [OPT](https://arxiv.org/abs/2205.01068) which includes collections from multiple dialogue datasets, however, they are not publicly available.\n\n- Furthermore, Iyer et al. have pointed out that integrating dialog data negatively impacts the outcomes, leading them to exclude dialogue data when fine-tuning the final [OPT-IML model](https://arxiv.org/abs/2212.12017). This insinuates that dialog modality might benefit from a dedicated benchmark. \n\n- While some proponents have introduced public dialog benchmarks like InstructDial and DialogStudio, none of these datasets address compositions, which is the central focus of our paper.\n\n- All in all, we agree with the reviewer that these comparisons could be helpful. We will add a new table in the final version of the paper where we discuss differences between popular multi-tasking benchmarks --> Flan, OPT-IML, InstructDial, and ours.\n\n| Collection    | Model                       | Base        | Size       | Public | Prompt                  | # Tasks | # Data Points | Objective                                                         |\n|---------------|-----------------------------|-------------|------------|--------|-------------------------|---------|---------------|-------------------------------------------------------------------|\n| FLAN          | FLAN-T5<br>FLAN-PALM        | T5, PALM  | 10M to 540B | P   | Zero, Few, COT       | 1836    | 15M           | Covers General NLP Tasks                                          |\n| OPT-IML       | OPT-IML                     | OPT         | 30 to 175B  | P   | Zero,Few, COT       | 2067    | 18M           | Covers General NLP Tasks and Meta Learning Instructions           |\n| InstructDial  | DIAL-T0, DIAL-BART0       | T0, BART0 | 400M to 3B  | P   | Zero, Few               | 48      | 250k          | Covers Dialog Specific Tasks                                      |\n| CESAR         | CESAR                       | FLAN-T5     | 3B to 11B   | P   | Zero                      | 154     | 450k          | Covers Dialog Specific and Compositional Tasks                    |\n\n\n\n> The generalization experiments of CESAR-FLAN-XXL initialized from FLAN-XXL, are somewhat undermined. Given that FLAN-XXL already integrates around 400 datasets and over 1.5k tasks, InstructDial also introduces several tasks for each dataset. The paper's claim of \"unseen generalization\" isn't wholly convincing unless both datasets and tasks from FLAN-XXL and InstructDial are explicitly excluded from evaluation. Besides, it may also leak data information for seen domains if different tasks come from the same datasets.\n\n- Generalization on Compositions: Our focus on generalization pertains to the composition level rather than individual tasks. Thus, we are interested in assessing the model's ability to generalize to task compositions it hasn't been specifically trained on, irrespective of whether it was trained on the constituent tasks. \n- The reviewers' concern might be particularly applicable to the final column, where we assert that both tasks are novel (end-with generation and emotion-grounded generation). However, for Table 5, our finetuning doesn't encompass the entire InstrucdDial dataset; instead, it involves a subset of tasks.\n- Therefore, the primary consideration here should be whether FLAN includes a task resembling end-with generation or emotion-grounded response generation. While - - Flan models might have seen individual tasks and their corresponding prompts, the compositions are not present in Flan -- particularly for dialog tasks. This is indeed our main contribution.\n- We will edit the relevant parts of the paper again to clearly specify our generalization claims.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a unique framework that merges various 1-D atomic tasks, I think the motivation is clear and sound.\n\n2. The established InstructDial benchmark is expanded with an addition of 68 novel downstream tasks.", "reasons_to_reject": "1. While the paper introduces 68 new downstream tasks based on seven 2-D Cesar tasks, it mainly offers overall performance. This obscures the individual quality and contribution of each task. There's a possibility that only a few high-quality tasks drive the improvements, while the majority add little value.\n\n2. The methodological clarity regarding CESAR's task combination into coherent instruction prompts remains ambiguous. It's essential to discern the reliability of these instructions and ensure task diversity, preventing overlapping or overly similar tasks.\n\n3. The paper doesn't sufficiently differentiate itself from FLAN-T5, which already encompasses several dialogue tasks and also detailed task design. It would great if the paper could discuss the differences.\n\n4. The generalization experiments of CESAR-FLAN-xxl, initialized from FLAN-xxl, are somewhat undermined. Given that FLAN-xxl already integrates around 400 datasets and over 1.5k tasks, and InstructDial also introduces several tasks for each dataset. The paper's claim of \"unseen generalization\" isn't wholly convincing unless both datasets and tasks from FLAN-xxl and InstructDial are explicitly excluded from evaluation. Besides, it may also leak data information for seen domains if different tasks come from the same datasets.", "questions_to_authors": "See 3. and 4.", "paper_topic_and_main_contributions": "The paper presents CESAR, an innovative instruction-based framework designed for dialogue tasks. Unique in its capability, CESAR autonomously generates compositional task prompts, eliminating the need for human intervention. Building on this, the study expands the existing InstructDial benchmark by integrating 15 more datasets and introducing 38 novel atomic tasks. This results in the creation of 68 new downstream tasks by merging single-dimensional atomic tasks. Experimental results show the advantages of these compositional tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TWz8GkCHPD", "reviewer": "Reviewer_8BXP", "summary": "", "questions": "Could you please show the assumption \"open source models are really less capable of handling compositional tasks\" is correct?\n\nCould you please explain why \"ChatGPT does not perform well on author's generated compositional task compared to the finetuned model\".\n\nCould you please further include evaluation on more open source models like Alpaca, Vicuna, and WizardLM, Llama2-Chat. Please compare their performance in a single table. \n\nCould you please explain the correctness of CESAR, is it different from natural queries? Could you please filter some data collected from ShareGPT or somewhere else which has compositional task instructions as another benchmark? I doubt that your dataset is \"aritificial\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback from Reviewer \"8BXP\" and thank them for their encouraging views on **the novelty of our work** and the value they see in **InstructDial++ benchmark**. \n\nCESAR centers on the generation of prompts and instructions within dialogues. Our principal innovation lies in its capacity to craft compositional task prompts, an aspect previously overlooked in prior studies in this domain. \n\nBelow, we address each of the Reviewer's concerns:\n\n> Firstly, I suspect that the assumption of this paper is wrong. Are open-source models really less capable of handling compositional tasks? I did not find any evaluation on current state-of-the-art Llama models in the paper.\n\n> Could you please show the assumption \"open source models are really less capable of handling compositional tasks\" is correct?\n\n> Could you please further include evaluation on more open source models like Alpaca, Vicuna, and WizardLM, Llama2-Chat. Please compare their performance in a single table.\n\n\n| Model        | Release Date | Contemporaneous according to ACL policy? |\n|--------------|--------------|------------------------------------------|\n| [Llama](https://ai.meta.com/blog/large-language-model-llama-meta-ai/)        | 24.02.2023   | YES                                      |\n| [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html)       | 13.03.2023   | YES                                      |\n| [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/)       | 30.03.2023   | YES                                      |\n| [WizardLM](https://github.com/nlpxucan/WizardLM)    | 09.08.2023   | YES                                      |\n| [Llama2-Chat](https://huggingface.co/blog/llama2)  | 18.07.2023   | YES                                      |\n\n\n- As presented all SOTA models at the time were released approximately in the last three months before our submission date which can be tagged as \"contemporaneous\" as per [ACL policy](https://www.aclweb.org/adminwiki/index.php?title=ACL_Policies_for_Submission,_Review_and_Citation)\n- Our experimentation involved utilizing the contemporary state-of-the-art model at that time, namely Flan-T5. Flan-T5 is a good foundation model and competes with many of these latest Llama derivatives (https://arxiv.org/abs/2306.04757).\n- However, we also acknowledge the reviewer's suggestion that incorporating results on these models is pertinent due to the dynamically changing nature of the field. Thus, we shall incorporate the latest instruction-tuned models in our revised experiments.\n\n> Secondly, I saw in this paper (Table 7) that ChatGPT does not perform well on the author's generated compositional task compared to the finetuned model (Table 5). It is likely that the author created an \"artful\" dataset.”\n\n> Could you please explain why \"ChatGPT does not perform well on author's generated compositional task compared to the finetuned model\".\n\n> Could you please explain the correctness of CESAR, is it different from natural queries? Could you please filter some data collected from ShareGPT or somewhere else that has compositional task instructions as another benchmark? I doubt that your dataset is \"artificial\".\n\n- We would like to highlight that CESAR is not composed of natural queries but of structured queries that enable the automatic combination of tasks (which is the core contribution of this work).\n- We discuss the advantages and disadvantages of this trait in Appendix G and also discuss ways to translate this structure into natural form using an LLM such as ChatGPT. We also show such an example in Table 11.\n- Moreover, we would like to bring another recent paper to the reviewer's attention which induces compositional prompts structurally (via CFG grammar) and later translates them to natural prompts → https://arxiv.org/abs/2307.08689\n- The results in Table 7 are poor as ChatGPT was prompted in zero-shot. In order to explore this further we experimented with a small set with 3 compositional tasks incorporating both GPT3.5 and GPT4 and both zero-shot and three-shot prompting. The results are as follows:\n\n| Model  | # SHOTS | BW+EW  | BW+KC | EW+KC |\n|--------|---------|--------|-------|-------|\n| GPT-3.5| 0       | 24.13  | 67.85 | 37.93 |\n| GPT-3.5| 3       | 31.03  | 78.57 | 44.82 |\n| GPT-4  | 0       | 31.03  | 92.85 | 65.51 |\n| GPT-4  | 3       | 62.06  | 85.71 | 75.86 |\n\n- In general, we find that with few examples in the context and more recently available GPT-4 API, the results improve significantly. In the final version of the paper, we shall update Table 7 with the above setting to make the distinction between open and closed models clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a new form to represent compositional tasks. \n\nThe paper created a new benchmark InstructDial++ to evaluate model's capability on compositional tasks.", "reasons_to_reject": "Firstly, I suspect that the assumption of this paper is wrong. Are open source models really less capable of handling compositional tasks? I did not find any evaluation on current state-of-the-art Llama models in the paper. \n\nSecondly, I saw in this paper (Table 7) that ChatGPT does not perform well on author's generated compositional task compared to the finetuned model (Table 5). It is likely that the author created an \"artful\" dataset.", "questions_to_authors": "Could you please show the assumption \"open source models are really less capable of handling compositional tasks\" is correct?\n\nCould you please explain why \"ChatGPT does not perform well on author's generated compositional task compared to the finetuned model\".\n\nCould you please further include evaluation on more open source models like Alpaca, Vicuna, and WizardLM, Llama2-Chat. Please compare their performance in a single table. \n\nCould you please explain the correctness of CESAR, is it different from natural queries? Could you please filter some data collected from ShareGPT or somewhere else which has compositional task instructions as another benchmark? I doubt that your dataset is \"aritificial\".", "paper_topic_and_main_contributions": "The key contribution of this paper is proposing CESAR and InstructDial++, a framework to benchmark model's capability to handle compositional tasks.\n\nCESAR is able to unify multiple compositional task dataset.\n\nCESAR enables automatic generation of compositional dialog tasks.\n\nThe paper also make evaluation pipeline for compositional tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Currently, the evaluation for ChatGPT is placed in Appendix, while the results on author's finetuned model is in main section. They should be placed together to show the author's assumtion is correct. And results from more open source models should be included. Otherwise I could not verify the soundness of this paper."}}, {"review_id": "9GtGslfs8T", "reviewer": "Reviewer_5Uzf", "summary": "", "questions": "1) Could you provide more detailed explanations of the CESAR framework in the paper? It would be helpful for readers to understand the methodology and implementation.\n2) Can you provide more thorough evaluation and comparisons with existing methods or baselines in the field? This would strengthen the paper's contributions and provide more insights for readers.\n3) Is it possible to provide implementation details or make the code available for reproducibility? This would greatly benefit the research community.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback from Reviewer \"5Uzf\". It is motivating to hear that they think **CESAR is a novel framework** and we are happy that they see value in its **capability to unify a large number of dialog tasks**, **generate complex instructions** and its **enhancements on the InstructDial benchmark**.\n\nCESAR centers on the generation of prompts and instructions within dialogues. Our principal innovation lies in its capacity to craft compositional task prompts, an aspect previously overlooked in prior studies in this domain. \n\nBelow, we address each of Reviewer's concerns:\n\n> The paper lacks a detailed explanation of the CESAR framework, making it difficult for readers to fully understand the methodology.\n\n> Could you provide more detailed explanations of the CESAR framework in the paper? It would be helpful for readers to understand the methodology and implementation.\n\nThank you for your feedback. We aimed to provide a comprehensive explanation in the paper. To elaborate:\n- Section 4.1 delves into the overarching framework and the structure of the prompts within CESAR.\n- In Section 4.2, we elucidate how 0-D, 1-D, and 2-D CESAR tasks are constructed, highlighting the nuances between atomic and compositional tasks.\n- Furthermore, Section 5.2 describes the mapping of each task in InstructDial++ to a CESAR task.\n- We genuinely value your perspective and would be grateful for any specific suggestions or points where you feel more clarity could be added.\n\n> The evaluation section could benefit from more detailed analysis and comparisons with existing methods or baselines.\n\n> Can you provide more thorough evaluation and comparisons with existing methods or baselines in the field? This would strengthen the paper's contributions and provide more insights for readers.\n\n- We have endeavored to provide a detailed explanation of both atomic and compositional results in the evaluation section.\n- Our evaluation is broadened through generalization experiments that span three dimensions. This includes evaluations of CESAR's efficacy with both seen and unseen compositions, as well as those involving new tasks.\n- We believe that these multifaceted discussions together offer solid backing for our hypothesis, further strengthened by the insights from our preliminary experiments.\n- We would appreciate any specific insights or suggestions the reviewer might have to enhance our paper.\n\n> The paper does not provide implementation details or code availability, which could hinder reproducibility.\n\n> Is it possible to provide implementation details or make the code available for reproducibility? This would greatly benefit the research community.\n\n- As mentioned in the abstract we will publish the implementation publicly upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper introduces a novel framework, CESAR, which addresses the challenge of generating complex instructions for dialog tasks.\n2) CESAR unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without any manual effort.\n3) The paper enhances the InstructDial benchmark with new datasets and tasks and demonstrates the scalability of CESAR in providing rich instructions.", "reasons_to_reject": "1) The paper lacks a detailed explanation of the CESAR framework, making it difficult for readers to fully understand the methodology.\n2) The evaluation section could benefit from more detailed analysis and comparisons with existing methods or baselines.\n3) The paper does not provide implementation details or code availability, which could hinder reproducibility.", "questions_to_authors": "1) Could you provide more detailed explanations of the CESAR framework in the paper? It would be helpful for readers to understand the methodology and implementation.\n2) Can you provide more thorough evaluation and comparisons with existing methods or baselines in the field? This would strengthen the paper's contributions and provide more insights for readers.\n3) Is it possible to provide implementation details or make the code available for reproducibility? This would greatly benefit the research community.\n", "paper_topic_and_main_contributions": "This paper proposes a novel framework called CESAR for automatic induction of compositional instructions for multi-turn dialogs. The authors apply CESAR on the InstructDial benchmark and demonstrate its scalability in providing rich instructions. They also conduct experiments to evaluate the performance of CESAR on atomic and compositional tasks. The paper provides valuable contributions to the field of dialog systems and instruction-based multitasking.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the provided reviews, the paper has several strengths. It presents a novel framework called CESAR that addresses the challenge of handling complex instructions in dialog applications. The motivation and details of the approach are described in great detail, and the paper provides abundant experiments that validate the authors' method. The enhanced InstructDial benchmark, InstructDial++, is considered a meaningful contribution. The results show that CESAR-FLAN performs better than existing approaches on both atomic and compositional tasks.

However, there are also some concerns raised by the reviewers. One reviewer questions the lack of comparison and evaluation between CESAR and GPT-3.5-turbo, and suggests providing qualitative examples to analyze the performance gap. Another reviewer suggests providing more transparency in the generation of compositional tasks and addressing the ambiguity in task combination within CESAR. It is also suggested to differentiate the paper from FLAN-T5 and provide more thorough evaluation and comparisons with existing methods or baselines. Furthermore, one reviewer challenges the assumption that open-source models are less capable of handling compositional tasks and raises concerns about the dataset used for evaluations.

Overall, the paper is well-written and well-structured, and provides sufficient support for its main claims. It deepens the understanding of instruction-based multitasking in dialog applications and lowers the barriers to this research direction. However, there are some areas that need improvement, such as providing more detailed comparisons, addressing concerns about task generation and combining, and clarifying the assumptions and dataset used for evaluation.
Decision: Accept-Main