On the Compositional Generalization in Versatile Open-domain Dialogue
Year: 2023

Abstract:
Previous research has demonstrated the potential of multi-task learning to foster a conversational agent’s ability to acquire a variety of skills. However, these approaches either suffer from interference among different datasets (also known as negative transfer), or fail to effectively reuse knowledge and skills learned from other datasets. In contrast to previous works, we develop a sparsely activated modular network: (1) We propose a wellrounded set of operators and instantiate each operator with an independent module; (2) We formulate dialogue generation as the execution of a generated programme which recursively composes and assembles modules. Extensive experiments on 9 datasets verify the efficacy of our methods through automatic evaluation and human evaluation. Notably, our model outperforms state-of-the-art supervised approaches on 4 datasets with only 10% training data thanks to the modular architecture and multi-task learning. 1

heading: 1 Introduction
text: Building an open-domain dialogue system is an intriguing and challenging task. A good open-domain chatbot should be equipped with a well-rounded set of skills (Roller et al., 2021) including but not limited to providing an informative response, showing different emotions, keeping a consistent persona and conducting commonsense inference. Up to now, with more and more datasets proposed to train multiple conversation skills (e.g., Wizard of Wikipedia (Dinan et al., 2019), Personachat (Zhang et al., 2018)), multi-task learning is an efficient way to grasp all the versatile skills and quickly transfer to newly emerging datasets (Roller et al., 2021).
†Tingchen Fu and Xueliang Zhao contribute equally to this work. This work was done during their internship at Tencent AI Lab.
*Corresponding author: Rui Yan(ruiyan@ruc.edu.cn) 1The code is available at https://github.com/
TingchenFu/ACL23-ModularDialogue
However, as a core problem in multi-task learning, it is not easy to strike a balance between transfer and interference (negative transfer) among multiple datasets (Rosenbaum et al., 2019). For this, recent researchers mainly follow two lines of research. On one line, Roller et al. (2021) and Shuster et al. (2022a) simply mix all the datasets together to embody the blended skill required in dialogue. They update all the model parameters to minimize the loss of all of the data, which is also dubbed dense training (Gururangan et al., 2022). In spite of its simplicity, it easily incurs interference among different datasets (Aribandi et al., 2022). On the other line, Li and Liang (2021) learn multiple skills and store the knowledge from different datasets with different sets of parameter-efficient architectures. This approach eliminates underlying negative transfer among different corpora, but hinders positive transfer at cost. The model has to learn from scratch rather than reuse past knowledge every time a new corpus comes.
Inspired by recent advancements in neuroscience (Dehaene et al., 2021) suggesting that the human brain represents knowledge in a modular way, we incorporate this as an inductive bias and present a compositional modular architecture to balance transfer and interference (Rosenbaum et al., 2019). By decomposing the knowledge for dialogue into relative independent modules (Mittal et al., 2022), a neural model thus decides which module to invoke for different tasks or different samples. However, there are two challenges in applying modular architecture to building a versatile open-domain chatbot. First, the generation task is different from question answering, where the neural module network accomplishes impressive performance (Andreas et al., 2016; Hu et al., 2017; Gupta et al., 2020). It is untouched how to apply the ideology of modularity to the auto-regressive generation process. Second, the modules used in neural module network (Andreas et al., 2016) are typically
13585
trained with end-task supervision. Without intermediate supervision or specialized training data for each module (Ponti et al., 2022), modules might perform homogeneous functions rather than perform their predefined functions as intended (Gupta et al., 2020, 2021).
To deal with the above problems, in this paper, we present a neural modular framework for blended-skill dialogue generation. The principle of our approach is to decompose the generation process into the recursive execution of basic operators by various modules. Specifically, (1) as an attempt to conduct generation tasks in a modular way, we introduce content modules for basic content synthesis and linguistics modules for linguisticallyrelated surface realization. In addition, a programmer is trained to produce a reverse-polish-style code (Burks et al., 1954) which schedules the modules to produce the final response. (2) To overcome the homogeneity of modules, we construct pseudo labels and provide weak supervision signals to facilitate the training of each module. Since the output of the programmer and the modules are discrete and thus not differentiable, we employ GumbelSoftmax trick to produce “soft” sentences as the output of modules at training time, and employ reinforcement learning to bridge the gap between the programmer and the modules.
Extensive experiments are conducted on 9 opendomain datasets. Our approach surpasses other models on a similar parameter scale and achieves a new state-of-the-art by multi-task training on all the 9 corpora. Notably, our model outperforms state-of-the-art supervised approaches on DailyDialog, EmpatheticDialog, LIGHT and Cornell Movie with only 10% training data, demonstrating that our modular framework could compose existing skills more efficiently to attain superior performance on out-of-distribution data.

heading: 2 Related Work

heading: 2.1 Open Domain Dialogue
text: Most early attempts on dialogue generation construct dialogue systems using manually created rules or templates (Weizenbaum, 1966; Wallace, 2009). The advancements in the field of machine translation (Ritter et al., 2011; Gehring et al., 2017; Vaswani et al., 2017) have served as inspiration for a number of explorations to construct end-to-end open-domain dialogue generation models (Shang et al., 2015; Vinyals and Le, 2015). Following
that, the vanilla encoder-decoder architecture is widely employed to improve response quality, and it has undergone several revisions to enhance response diversity (Xing et al., 2017; Zhao et al., 2017; Tao et al., 2018), model conversation context structure (Xing et al., 2018; Zhang et al., 2019), and regulate response characteristics (Wang et al., 2018; See et al., 2019; Wang et al., 2020a). Smith et al. (2020) and Shuster et al. (2020) initiate the study of equipping the open-domain conversation agent with a well-rounded set of skills, whose key idea is to conduct simultaneous multi-task training on the blended data. These models have demonstrated encouraging results in skill blending and skill selection thanks to the careful design of the training scheme. BlenderBot (Roller et al., 2021) demonstrates how large-scale models can further promote the concurrent acquisition of several skills. BlenderBot 2.0 is created as a result of the additions made by Komeili et al. (2022) and Xu et al. (2022), who offer BlenderBot the capacity to access the Internet and memorize lengthy history respectively.

heading: 2.2 Multi-task Learning with Pre-trained Language Models
text: Multi-task learning is a common paradigm to transfer knowledge from multiple related tasks to enhance generalization capacity and has shown promising results in a variety of NLP tasks (Zhang and Yang, 2021; Crawshaw, 2020). Large-scale pre-trained language models (PLMs) have presented brand-new difficulties for multi-task learning. Aghajanyan et al. (2021) propose prefinetuning which refines the pre-trained representations through massively multi-task learning. In spite of its efficiency, pre-finetuning may result in catastrophic forgetting of the pre-training task. To alleviate this issue, Aribandi et al. (2021) propose multi-task pre-training which bridges the gap between pre-training and finetuning data distributions. T0 (Sanh et al., 2021) is an early attempt to induce the zero-shot generalization capability of PLMs through explicit multi-task learning, which converts NLP tasks into a manually-collected prompted form. Another prevalent paradigm in multi-task learning using PLMs is instruction tuning, in which the PLMs encode task-specific instructions together with input and produce task output (Wei et al., 2021; Mishra et al., 2022; Wang et al., 2022). Despite promising results, these methods may suffer from the negative transfer problem
due to the practice of activating all parameters for different tasks. To mitigate this issue, researchers have resorted to parameter-efficient methods which allocate separate adapters for each task (Mahabadi et al., 2021), and compositional modules which only activate relevant parts of the models (Ponti et al., 2022). Our method is orthogonal to earlier efforts in that it attempts to mitigate the unexplored negative transfer problem in auto-regressive decoding.

heading: 2.3 Neural Modular Network
text: The concept of neural module networks has drawn a lot of interest in a variety of computer vision and natural language processing tasks. Andreas et al. (2016) initially propose neural module network, which parses questions into linguistic substructures and builds question-specific deep networks from compositional modules, to conduct visual question answering. Following this work, several attempts have been made to eliminate the need for mediate supervision on semantic parsers (Hu et al., 2018; Mao et al., 2019), directly forecast the instancespecific network architectures in an end-to-end way (Hu et al., 2017), infer the answer with a purely symbolic executor (Yi et al., 2018), and perform visual co-reference resolution (Kottur et al., 2018). Gupta et al. (2020) and Chen et al. (2020) propose employing neural module networks in response to questions in machine reading comprehension. Another line of closely related works to ours are gen-
erative neural module networks, which activate a module when generating the next token (Yang et al., 2019; Tian and Oh, 2020) or only utilize modular architecture for encoder (Le et al., 2022). Our research differs significantly from theirs in that we break down dialogue response generation into independent operations in order to reduce catastrophic forgetting in each module.

heading: 3 Preliminary
text: For open-domain dialogue generation, each datum can be thought of a pair (x,y), where y is the response and x is the dialogue context composed of history utterances and other external resources such as background knowledge (CMU_DoG (Zhou et al., 2018b)), persona of speakers (ConvAI2 (Zhang et al., 2018)) or conversation setting (LIGHT (Urbanek et al., 2019)). The goal of an open-domain dialogue generation model is to generate y given x and exhibit the necessary skills to be more humanlike.
In the proposed modular generation framework, a programmer pθ(c|x) takes the dialogue context as input and produces a code sequence c = [c1, c2, · · · , cn], where n is the length of the code. Based on the generated code, different modules are activated to perform different functions. The execution of the code produces a response in the end. The workflow of our framework is shown in Figure 1.
The rest of our paper is structured as follows.
We illustrate the modular architecture in §4, including the implementation of the programmer and execution of the code with modules. In §5, we elaborate the training algorithm to cope with the paucity of human annotation and discrete optimization. The experiment results and further analysis are displayed in §6 and §7 respectively.

heading: 4 Modular Framework
text: In this section, we elaborate on how the modular generation framework works. Briefly, a programmer first generates a code in a special reversepolish-style programme language. Then we execute the code with a stack to store the intermediate result. When encountering some specific operators in code, we just activate corresponding modules to fulfill the function of the operator.

heading: 4.1 Module Definition
text: Our modules are devised to perform basic atomic tasks and realize the function of some operators. From another perspective, operators are high-level abstractions of modules. According to their functions and the format of input and output, there are 3 types of modules, namely span operator, content operator and linguistics operator. The Span operator is responsible for selecting a span from the dialogue context given the start index and the end index, whose role is similar to QUESTION_SPAN and PASSAGE_SPAN in Chen et al. (2020). Content operators (COPY, PARAPHRASE and INFER) generate diverse new content based on the input text. The linguistics operators (COMPOUND, VERB_MODIFY and NOUN_MODIFY) combine two texts together to form complex sentence or compound sentence. The computation result of the linguistics operators could also serve as the operands to other linguistics operators and content operators. We list all the operators used in our framework in Appendix A.
In the implementation, each content operator and linguistics operator are corresponding to an autoregressive generation module M(ymi |xm, ym<i), where xm, ym are the input and output of the module M. They are parameterized as standard transformers. We initialize the parameters of these models using pre-trained T5-small (Raffel et al., 2019). Customizing different modules according to their intended purpose might lead to better performance, but we focus on the overall framework in this paper and leave the sophisticated design of modules for
future work. The key insight behind the module instantiation is to decompose the response generation process into relatively independent and composable pieces. Although our framework bears similarities with previous works in visual QA (Andreas et al., 2016; Hu et al., 2017) and image captioning (Tian and Oh, 2019; Yang et al., 2019), the crucial difference of our framework lies in the sparsity of dependency between these highly abstract operators, which allows the respective learning of each module possible and thus eliminates the intra-operator interference.

heading: 4.2 Programme Generation
text: The programmer maps the natural language dialogue context to an executable programme in a reverse polish notation style. The code tokens (the vocabulary of the programmer) consist of two parts, namely the operator defined in Table 8 and the position index of the dialogue context. Following the design of Gupta et al. (2019) and Chen et al. (2020), the core architecture for programme generation is a MiniLM (Wang et al., 2020b) reader and a 1-layer GRU. At the t-th timestep, assume the embedding of past generated code tokens are [hc1,h c 2, · · · ,hct−1] and the dialogue context representation encoded by BERT is Hx = [hx1 ,h x 2 , · · · ,hxl ], where l is the length of the context. We first calculate ht, the hidden state of GRU at the current step:
ht = GRU(ht−1,hct−1). (1)
Then we apply the attention mechanism to compute the context vector sx:
sx = l∑
i=1
wih x i ,
w = Softmax(hTt H x),
(2)
The history code vector sc is computed in the same way. Afterwards, we concatenate the sc, sx and GRU hidden state ht together and computes:
s = W[sx; sc;ht], (3)
where [·; ·] denotes the vector concatenation operation. Finally, the probability distribution of the next code token ct is:
Pr(ct) = Softmax(s T[Hx;Eo]), (4)
where Hx plays the role of position index embedding and Eo is a trainable operator embedding. We refer our readers to Chen et al. (2020) for more details about the programmer.

heading: 4.3 Programme Execution
text: As mentioned before, the generated programme is essentially a reverse polish expression (Burks et al., 1954). Therefore, we maintain a stack to assist the execution of the programme. To be more specific, given a generated code c = [c1, c2, · · · , cn] and an empty stack, we scan every code token in c one by one and take actions according to the current code token ci:
• If ci is a position index, push it into the stack;
• If ci is SPAN operator, pop the top two items in the stack. Take the two items as the start index and end index to select a span. Push the span into the stack.
• If ci is one of the content operators, pop the top one items in the stack and send it into the corresponding content module. Push the generated text into the stack.
• If ci is one of the linguistics operators, pop the top two items in the stack, which should be two sentences. concatenate them together and send them into the corresponding linguistics module. Push the generated sentence into the stack.
Generally, the execution of programme bears similarity to that of push-down automata. The motivation behind this is to isolate different procedures in dialogue generation and use a stack to temporarily store the intermediate result, which is the only medium for message passing between modules.
At the end of the code execution, the item(s) left in the stack is popped out. To improve fluency, we attempt to polish the stack output with another neural network, but it seems that directly concatenating the outputted sentences together is enough.

heading: 5 Learning Details

heading: 5.1 Weak Supervision
text: Training the programmer and the modules jointly with the response as the only supervision signal is challenging (Gupta et al., 2020). More importantly, without the supervision of intermediate output, we have no idea whether the modules differentiate into
Algorithm 1 A high-level algorithm for producing pseudo labels. 1: Input: A pair of (x,y), a similarity function sim(·, ·), a
syntactic relation classifier dis(·, ·), threshold ψ1, ψ2, 2: Initialize an empty code sequence c and pseudo-labeled
datasets Dop for all modules 3: Use parsing tools to parse y into a tree T . 4: for Segment s among the in-order traverse sequence do 5: Search a span s′ from x that is most similar to s and 6: Locate the start and end position of the span and append them into c. 7: Append SPAN into c 8: if sim(s, s′) > ψ2 then 9: Append COPY into c. 10: Add the pair (s, s′) into Dcopy 11: else if sim(s, s′) < ψ1 then 12: Append INFER into c. 13: Add the pair (s, s′) into Dinfer 14: else 15: Append PARAPHRASE into c. 16: Add the pair (s, s′) into Dparaphrase 17: end if 18: if One child of s has been visited (denoted as schi) and the parent of s has not been visited yet then 19: Append OP=dis(s, schi) into c. 20: Add the pair (s, schi) into Dop 21: else if All children of s have been visited and the parent of s has been visited too (denoted as spar) then 22: Append OP=dis(s, spar) into c. 23: Add the pair (s, spar) into Dop 24: end if 25: end for 26: Return The pseudo-code label c and Dop
intended functions. Annotating training data with human labor for every module is costly and we instead use heuristically obtained pseudo labels for substitution.
Algorithm 1 is a high-level illustration of how we make pseudo labels. More details could be found in Appendix C.

heading: 5.2 Reinforcement Learning.
text: When trained respectively, the programmer and the modules may not adapt to each other well when directly assembled together. Therefore, we propose to further optimize the programmer with policy gradient (Sutton et al., 1999),
J(θ) = Ec∼pθ(c|x)[r(c)] (5)
and design the reward r(c) as the similarity between the generated hypothesis and the ground truth response. Combining the ratio-likelihood trick, we have
∇θJ(θ) = Ec∼pθ(c|x)[∇θ log pθ(c|x)r(c)], (6)
where g(y|x, c) represents the execution of the code to generate response.
In addition, to facilitate end-to-end training, we apply Gumbel-Softmax trick (Jang et al., 2017) to overcome the differentiable obstacle owing to the discrete nature of natural language when optimizing the modules. Formally, instead of selecting one token from module-predicted vocabulary distribution M(ymi |xm, ym<i), the content modules and linguistics modules sample a “soft word”:
y∗i = Gumbel(M(ymi |xm, ym<i), τ), (7)
where τ is the temperature of sampling.

heading: 6 Experiment

heading: 6.1 Experimental Setup
text: Setting. To comprehensively evaluate the multitask learning ability and the generalization ability, suppose we have N datasets, we evaluate our proposed framework in three settings: (1) All Task MTL. In this setting, we train our model on the mixed union of N datasets and evaluate it on each individual dataset. (2) Leave-one-out. In this setting, we train our model on N − 1 datasets and test on the left one dataset to evaluate a model’s
zero-shot generalization ability. (3) Low-resource. To further evaluate the generalization capability of our method, after training on other N − 1 datasets in the leave-one-out setting, we fine-tune the model on the left dataset with only 10% data available, and test the model on the left one dataset.
Datasets. We use N = 9 datasets to evaluate our framework: Cornell Movies (Danescu-NiculescuMizil and Lee, 2011), DailyDialog (Li et al., 2017), CMU_DoG (Zhou et al., 2018b), LIGHT (Urbanek et al., 2019), EmpatheticDialog (Rashkin et al., 2019), ConvAI2 (Dinan et al., 2020), Wizard of Wikipedia (Dinan et al., 2019), Mutual (Cui et al., 2020) and CommonsenseDialog (Zhou et al., 2021). Each dataset embodies one or more specific skills. More details about the datasets could be found in Appendix B.
Baselines. We use BART (Lewis et al., 2020) as one of our baselines, which is a standard sequenceto-sequence transformer pre-trained on the same corpus as Liu et al. (2019); We also compare against R2C2, a BlenderBot-like open-domain dialogue model trained in a multi-task way by Shuster et al. (2022a) and hold the current stateof-the-art on many datasets (Zhang et al., 2022). For parameter-efficient technique in multi-task learning, we compare our method with prefixtuning (Li and Liang, 2021). We also draw a comparison with the recent proposed Modular Skill (MS) (Ponti et al., 2022), a modular network that allows each task to choose its skill toolkit and optimize the global skill inventory together with the choice of each task jointly. For a fair comparison, we use BART-large ( 406M) and R2C2-base ( 400M) in our experiments. The parameter scale of Prefix-tuning ( 415M) and MS ( 448M) are both comparable with ours.

heading: 6.2 Main Result
text: All Task MTL. The experiment results are shown in Table 1. We could observe that (1) our proposed approach outperforms BART and R2C2 on most datasets. The advantage of our modular framework over prefix-tuning is also obvious, possibly because prefix-tuning hinders positive transfer among corpora. To have a more comprehensive understanding of our approach, we investigate the schedule frequency of modules on different datasets and it reveals that our modular design captures some distinctive patterns in different corpora. More information could be found in Appendix F.
(2) Meanwhile, we also provide the performance of the current state-of-the-art for each individual dataset2. We could observe that when trained in a multi-task way, our framework is superior or comparable to the SOTA without a sophisticated design of model architecture and learning algorithm for each individual dataset, which further verifies the capacity of our model to transfer knowledge from other corpus and manipulate multiple skills.
Leave-one-out. The results are shown in Table 2. There is a gap in performance between the baseline and ours, especially on Wizard of Wikipedia. It can be understood that Wizard of Wikipedia is less similar to other datasets since it contains some formal sentences from Wikipedia. Thus, zero-shot generalization on the dataset is more difficult. We can conclude that our model generalizes better than BART and R2C2, possibly because the modular framework could recursively compose the computations by modules to cope with new situations with existing knowledge. Besides, the comparison with MS further verdict the necessity of intermediate supervision for each module.
Low-resource The results are shown in Table 3. The proposed method attains a better performance than BART. Notably, our modular generation framework surpasses the fully supervised approach on DailyDialog, EmpatheticDialog, LIGHT and CommonsenseDialog, validating the potential of the compositional modular paradigm as a general method in the low-resource setting.
2Some numbers are directly cited from the papers

heading: 7 Further Analysis

heading: 7.1 Single Transfer Relation
text: To explore whether our framework enhances transfer in a multi-task learning scenario, we further draw a comparison in a single-task scenario where we train and test our model and all the baselines on each individual dataset. The experiment results are shown on Table 4. When comparing with Table 1, we could see that our approach achieves a positive transfer on most datasets while negative transfer is more common for baseline methods. It demonstrates that our modular design effectively alleviates the intra-operator transfer.

heading: 7.2 Pair-wise Transfer Relations
text: To have a closer look at the transfer relation among the datasets, we evaluate the transfer among datasets in a pair-wise multi-task learning setup. We use CommonsenseDialog, LIGHT, CMU_DoG and EmpatheticDialogie since they are diverse enough to be representative. The experiment results are shown in Table 5. Our approach attains positive transfer or at least avoids drop on most dataset pairs, while for BART the opposite is true. Besides that, an interesting trend manifests in individual relationships. For example, CMU_DoG and EmpatheticDialog seem to promote each other whilst LIGHT and CommonsenseDialog tend to hurt each other.

heading: 7.3 Ablation Study
text: An ablation study is conducted to explore how different mechanisms and components contribute to the performance. We compare our approach with the following variants: (1) -span: The SPAN operator is removed and we always select the entire dialogue context as a “span”. (2) -linguistic: The linguistics operator is replaced with a direct concatenation of two input segments. (3) -warm: The warm-up procedure is removed. (4) -reward: The reinforcement learning of programmer is removed. The results are shown in Table 6. The result reveals that warm-up is indispensable to the proposed method, and the conclusion is in coincidence with Gupta et al. (2020, 2021). The span operator and the linguistic operators are also helpful to the performance. The decline in appropriateness of -span and -linguistic validates the necessity of them.

heading: 7.4 Qualitative Evaluation
text: Automatic metrics are not perfect for evaluating an open-domain task (Dinan et al., 2019) and human evaluation is necessary. Concretely, in the all-task MTL setting, we randomly sample 300 responses from each dataset generated by ours and baseline methods and recruit well-educated native speakers to rate them. Each annotator is required to give a score ranging from 1 to 3. 1 means the response is correct in grammar and fluent; 2 means the response is coherent to the context and satisfies the requirements of 1. 3 means the response exhibits versatile skills if necessary including showing empathy, grounding on knowledge, commonsense inference, etc. Besides, the response should also meet the requirements of 2. Agreement of the annotators is measured via Fleiss’ kappa (Fleiss, 1971). As is shown in Table 7, the responses generated by our approach enjoy a higher quality, demonstrating the superiority of the modular generation framework. The evaluation results are also consistent with automatic evaluation.
A case study could be found in Appendix F.

heading: 8 Conclusions
text: In this work, we utilize the ideology of modular networks to address the transfer-interference prob-
lem in multi-task learning. We implement a model architecture that allows the composition of different modules to fulfill complicated functions and eliminate interference among modules. We apply our method to dialogue generation and conduct extensive experiments to verdict its efficacy. We hope our work would inspire relevant research in the community.
Ethic Considerations
The use of our approach could result in improved dialogue systems that enhance the quality of life for many individuals, especially in light of the widespread use of AI in everyday life. For instance, a more effective chatbot integrated with electronic gadgets will boost both productivity and user experience. On the other hand, the implementation of conversation systems could result in employment losses in some domains such as call centers.
Limitations
This work focuses on mitigating the negative transfer and catastrophic forgetting issue in multi-task dialogue generation. All technologies built upon the large-scale PLM more or less inherit their potential harms (Bender et al., 2021). Besides, we acknowledge some specific limitations within our methods:
1. The construction of pseudo labels requires dependency parsing with spaCy, which is timeconsuming. But we only construct pseudo labels offline in the training processing and it causes no latency at inference.
2. We instantiate our modular framework using MiniLM (Wang et al., 2020b) as the backbone of the reader within the programmer, and T5 (Raffel et al., 2019) as the backbone for the content operators and linguistic operators. We did not try other instantiations although the modular framework does not depend on the specific initialization choice of modules. Theoretically, any generative PLM could be the backbone of these linguistic and content modules.
3. We aim at decomposing the response generation into relatively independent and composable operators. Currently, the division of dialogue skills and module functions is in a heuristic way inspired by linguistics. Thus
it remains a future research question about how to design modular architecture in a more data-driven way.

heading: Acknowledgement
text: We thank all the reviewers and chairs for their suggestions and recommendation. This work was supported by National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Inter-disciplinary Platform for the ”Double-First Class” Initiative, Renmin University of China. We wish to acknowledge the support provided by Public Policy and Decision-making Research Lab, Renmin University of China and the Public Computing Cloud, Renmin University of China.

heading: A Details about Operators
text: The operators used in our framework are listed in Table 8.

heading: B More Details about Datasets
text: Cornell Movies (Danescu-Niculescu-Mizil and Lee, 2011) contains large-scale fictional conversations extracted from raw movie script. thus covering abundant topics and emotional change. The dataset is used for training and evaluating a chatbot to quickly capture the emotional change in dialogue and respond accordingly.
LIGHT (Urbanek et al., 2019) is about situated interaction between characters in a text adventure game. The dialogue context includes not only the historical utterance but also the persona and action of the speakers together with the background setting. The skill in this dataset is grounding discussion on the dynamic environment.
EmpatheticDialogue (Rashkin et al., 2019) is a crowd-sourced dataset in which a speaker describes his or her situation and a listener responds with empathy. The dataset provides the emotion labels for interlocutors at each turn, but we do not include that in the dialogue context of our experiment.
ConvAI2 (Dinan et al., 2020) is the dataset used for NeurIPS 2018 competition and is adapted from PERSONACHAT (Zhang et al., 2018). In conversation, the interlocutors are required to exhibit a given persona and try to know the persona of the partner at the same time. The dataset mainly focuses on the skill of getting to know each other and engaging in friendly greeting conversation.
Wizard of Wikipedia (Dinan et al., 2019) is a knowledge-grounded dataset in which an interlocutor plays the apprentice and asks questions while the other interlocutor plays the wizard and gives informative responses. The wizard has access to background knowledge from Wikipedia. We only include the golden knowledge in the dialogue context. The dataset is for training and evaluating the skill of grounding conversation on knowledge.
CMU_DoG (Zhou et al., 2018b) is also a dataset for knowledge-grounded dialogue. In each conversation, two interlocutors discuss a given movie. The basic information of the movie including rating, release year, review and main plots are provided as background knowledge. Similarly, we use only the golden knowledge. The dataset focuses on grounding knowledge.
Mutual (Cui et al., 2020) is collected from Chinese students’ English listening comprehension exams. The model needs to generate a logically correct continuation of the conversation based on historical utterances. The dataset facilitates reasoning ability on social etiquette and relationships.
CommonsenseDialog (Zhou et al., 2018a) consists of two parts. The first part is extracted from the existing dialogue dataset using ConceptNet while the second part is crowd-sourced asking the crowd workers to exhibit social commonsense in an interacting environment. We only use the crowdsourced part to avoid overlap with other datasets used in our experiments. The dataset requires the skill of performing latent or explicit commonsense inference in communication.
DailyDialog (Li et al., 2017) is a dataset intended to reflect conversations occurring in daily life, covering a wide range of domains and topics. The dataset is also annotated with the topic, emotion and utterance act, but we only use the history utterance as the dialogue context.
Since the test set of ConvAI2 and Mutual is not publicly released, we conduct validation on a separate subset (10%) of the training set and test on the original validation set.
The statistics of our datasets are listed in Table 9.

heading: C More Details about Weak Supervision
text: Since algorithm 1 is only a high-level description, we provide more details here about how to produce our pseudo training data. In implementation, we use spaCy 3 as our parsing tool. It outputs a parsing tree and every token in the sentence is a node. We process the token-level parsing tree into a segment-level parsing tree by merging the nodes into verb phrases. Specifically, we merge all the nodes and within the subtree of a verb node unless it is another verb node or its nearest verb ancestor is another verb node. The edge between the verb nodes is kept unchanged. As a result, we parse the golden response into a segment tree T . To traverse all the segments in the tree, we use a pseudo inorder traverse because in the parsing tree a node may have more than two children and a traditional in-order traverse does not work here. Precisely, we for every node to visit, we first visit its first child, then the node itself, and finally all the other children. In algorithm 1, the similarity function
3 https://spacy.io/
sim(·, ·) is unigram F14 (Dinan et al., 2019). We set ϕ1 to be 0.35 and ϕ2 to be 0.75. The syntactic relation classifier cls(·, ·) is based on the dependency relation r between the two verb nodes in two segments:
cls(s1, s2) =    COMPOUND, r = conj, VERB_MODIFY, r = advcl NOUN_MODIFY, r ∈ {relcl, acl} (8)

heading: D More Implement Details
text: All the content modules and linguistics modules are sequence-to-sequence transformers initialized with T5-small (Raffel et al., 2019). The reader within the programmer is a bidirectional 6-layer transformer with an embedding size of 512. Its parameters are initialized from MiniLM (Wang et al., 2020b). The GRU in the programmer is 1-layer with the dimension of hidden state 512. All the models are learned with Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9 and β2 = 0.999. We sweep learning rate from [5e−6, 1e−5, 2e−5, 4e−5, 6e−6, 8e−5] and sweep batch size from [16, 32, 64, 128, 256].
4 https://github.com/facebookresearch/ParlAI/blob/
master/parlai/core/metrics.py
We set the weight decay as 1e− 2 and sweep the warmup steps from [1000, 2000, 4000]. The gradient clip is set to 2.0 to avoid the explosion of the gradient. The reward for reinforcement is implemented as the unigram-F1. We keep the temperature τ to be 1.0 through our experiment. A cosine learning schedule is applied to adjust the learning rate during training. An early stop on the validation set is adopted. We truncate the input dialogue context to a maximum length of 480. We conduct experiments on two RTX 3090. We use greedy search for decoding and report the performance averaged in three repetitive experiments.
E Inference Speed
We further compare the decoding speed at inference time with BART to see whether the modular
generation framework suffers from high latency. We conduct experiments on CMU_DoG, LIGHT, EmpatheticDialog and CommonsenseDialog with an RTX 3090. The experiment results are shown in Table 10. From the table, we could observe that our model has a lower inference latency than BART. We gauge the reason that the modules in our framework are much smaller in scale. Meanwhile, those modules only attend to the partial text selected by SPAN rather than the entire dialogue context.

heading: F Case Study
text: To have an intuitive understanding of how our modular framework takes effect, we show three cases in all-task MTL setting in Table 12, Table 11 and Table 13. Besides, we are also interested in whether the schedule frequency of each module varies in different datasets. We believe the difference of schedule frequency exhibits some intrinsic feature of the dataset. From Figure 2 to Figure 19, we conjecture that the schedule frequency gives out the linguistic style of a dataset. For example, in Wizard of Wikipedia (Dinan et al., 2019), a portion of sentences are directly copied from the knowledge; In ConvAI2 (Dinan et al., 2020), some sentences are paraphrased from the given persona of speakers.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 6,7

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6, 7
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 6,7
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 6,7

heading: Future Work
text: Thus
it remains a future research question about how to design modular architecture in a more data-driven way