Instruct Once, Chat Consistently in Multiple Rounds: An Efficient Tuning Framework for Dialogue (2024)

Abstract: Tuning language models for dialogue generation has been a prevalent paradigm for building capable dialogue agents. Yet, traditional tuning narrowly views dialogue generation as resembling other language generation tasks, ignoring the role disparities between two speakers and the multi-round interactive process that dialogues ought to be. Such a manner often leads to unsatisfactory chat consistency for the built agent. In this work, we emphasize the interactive, communicative nature of dialogue and argue that it is more feasible to model the speaker roles of agent and user separately, enabling the agent to adhere to its role consistently. With this in mind, we propose an efficient Multi-round Interactive Dialogue Tuning (MIDI-Tuning) framework1. It models the agent and user individually with two adapters built upon large language models. The adapters make use of respective utterances round by round in alternating order and they are tuned via a round-level memory caching mechanism. Extensive experiments demonstrate that, our framework performs superior to traditional finetuning and harbors the tremendous potential for improving dialogue consistency.

heading: 1 Introduction
text: Building human-like intelligent dialogue agents is a long-standing ambition for the research community of dialogue systems. Recently, we have witnessed a substantial revolution in advanced conversational agents such as ChatGPT (OpenAI, 2022) and GPT4 (OpenAI, 2023), which are fundamentally built upon large language models (LLMs) (Brown et al., 2020; Bommasani et al., 2021). Similar efforts have also been made by academia and open-source communities, leading to a variety of notable chat language models, such as Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), and LLAMA 2- Chat (Touvron et al., 2023b). These chat language
1Our code and data are available at https://github. com/iwangjian/Midi-Tuning.
Sample 1
Sample 2
Sample 𝑛
⋯
𝑢!
Conditional Context Response
(instruction)
𝑢! 𝑠!
Conditional Context Response
𝑢" 𝑠"(instruction)
𝑢! 𝑠!
Conditional Context Response
⋯ 𝑠#(instruction) 𝑢#
𝑠!
(a) One-dialogue-n-sample: split a multi-round dialogue into multiple single-round yet non-independent samples.
𝑢! 𝑠! 𝑢" 𝑠" ⋯ 𝑠#(instruction) 𝑢#
Predict
Predict
Predict
Causal Mask
(b) One-dialogue-one-sample: utilize a multi-round dialogue sample at once based on causal masks.
models can be attained by instruction fine-tuning on downstream dialogue data, demonstrating promising performance in generating natural and comprehensive responses. Tuning LLMs for dialogue generation has been the de-facto mainstream practice towards creating capable dialogue agents.
Traditional dialogue tuning narrowly views dialogue generation as resembling other language generation tasks without distinction. It performs in either one-dialogue-n-sample (see Figure 1(a)) or one-dialogue-one-sample (see Figure 1(b)) man-
3993
ner. The former transforms dialogue model training into general language generation via splitting each multi-round dialogue into multiple single-round samples, yet results in non-independent distributions among those samples. The latter enhances training efficiency by utilizing each multi-round dialogue at once, which computes the prediction loss for the agent’s responses through causal masks, such as Vicuna (Chiang et al., 2023) and UltraLLaMA (Ding et al., 2023). However, these methods simply concatenate utterances from two speakers (e.g., user and agent) together (and instructions for the agent, if any) and mix their content in the same language model space, ignoring the role disparities between two speakers and the multi-round interactive process that dialogues ought to be. Such tuning methods inevitably hinder a built dialogue agent from maintaining the chat consistency (Touvron et al., 2023b; Lu et al., 2023), requiring that the agent always adhere to its role even with the dialogue rounds moving forward. It remains urgent to solve for many consistency-demanding scenarios.
One of the primary challenges for improving dialogue consistency lies in the disparity modeling of the two speaker roles. It is because the inconsistency issue in real-world human communication (Wu et al., 2021; Bao et al., 2022; Takmaz et al., 2023) is often caused by various types of speaker disparities, such as background knowledge, cognitive level, personalities, and goals. We emphasize that it is more feasible to model the roles of agent and user separately (see Figure 1(c)), such that the agent and user models can consistently adhere to their respective roles and interact with each other round by round, similar to humans.
When tuning LLMs for conversation, we have a similar motivation towards consistent dialogue generation. We propose a general, simple, and effective framework, namely Multi-round Interactive Dialogue Tuning (MIDI-Tuning). It employs two language model adapters (e.g., LoRA (Hu et al., 2022)) built upon LLMs, to represent the agent and user, respectively. The two adapters are tuned by utilizing respective utterances round by round in alternating order, with each adapter learning to distinguish language model distribution about its role. However, such separate modeling is non-trivial in tracking the complete dialogue context. Considering that the foundation architecture of mainstream LLMs is Transformer (Vaswani et al., 2017), we propose a round-level memory caching mechanism
to address it efficiently, which reuses previousround cached keys and values as ongoing context when processing present-round utterance.
In summary, our main contributions are as follows: (1) To the best of our knowledge, this is the first work investigating how a new way of tuning could affect dialogue consistency in the era of LLMs. (2) We propose MIDI-Tuning, a general, simple, and efficient framework to tune LLMs for dialogue generation, which can be applied in broad downstream dialogue scenarios. (3) Extensive experiments demonstrate that MIDI-Tuning outperforms traditional fine-tuning over various LLMs, especially in maintaining consistency for multi-round dialogues.

heading: 2 Related Work
text: Language Models for Dialogue Many language models have been developed as dialogue agents for chatting with humans. As an early trial in industries, DialoGPT (Zhang et al., 2020) and BlenderBot (Roller et al., 2021) employed crawled conversational data to fine-tune pretrained language models (e.g., GPT-2 (Radford et al., 2019)) for open-domain dialogue. Built upon an LLM, i.e., GPT-3 (Brown et al., 2020), ChatGPT (OpenAI, 2022) has astounded the community with its powerful chat ability, which is optimized with instruction tuning and alignment tuning. In academia and open-source communities, there have emerged a variety of notable chat language models, such as Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), Baize (Xu et al., 2023), and UltraLLaMA (Ding et al., 2023). They are fine-tuned from an LLM named LLaMA (Touvron et al., 2023a) using different collected dialogue datasets. Similar efforts are observed in ChatGLM series (Du et al., 2022; THUDM, 2023a,b) and LLAMA 2-Chat (Touvron et al., 2023b). Tuning language models has become a prevalent paradigm for building capable dialogue agents, and this work mainly focuses on open-source LLMs for dialogue.
Consistency in Dialogue Dialogue consistency measures whether an agent’s generated utterances are consistent with the agent’s role and dialogue context, especially from several distinguishable aspects such as topics, styles (Wang et al., 2017), personas (Zhang et al., 2019; Song et al., 2020; Ju et al., 2022), and characters or roles (Urbanek et al., 2019; Shuster et al., 2022; Chen et al., 2023a). For checking dialogue consistency, most prior works
leveraged natural language inference (NLI) techniques (Song et al., 2020; Nie et al., 2021) or dataset benchmarking (Qin et al., 2021).
Existing works have attempted to build personaconsistent dialogues (Liu et al., 2020; Kim et al., 2020; Chen et al., 2023b). For example, Kim et al. (2020) adopted the Rational Speech Acts framework to improve persona consistency. Another line of research exhibits that interlocutor modeling is of high necessity for pragmatic communications (Bao et al., 2022) and multi-party conversations (Gu et al., 2023). These studies move a step towards improving dialogue consistency. Nevertheless, the challenge of improving consistency is far from being conquered, even for LLMs (Touvron et al., 2023b). More recently, MemoChat (Lu et al., 2023) enhanced LLMs’ chat consistency by carefully designing tailored tuning instructions. In comparison, our work is the first to explore how a new way of tuning brings consistency improvement.
Parameter-efficient Tuning Conventional finetuning is inefficient as the parameter size grows since it requires training all parameters of LLMs. Parameter-efficient tuning (Houlsby et al., 2019; Lester et al., 2021) adds a small number of tunable parameter layers, namely adapters, for fine-tuning while freezing the original parameters. Prefix Tun-
ing (Li and Liang, 2021) fine-tunes a sequence of task-specific vectors inserted before the input. LoRA (Hu et al., 2022) adopts trainable low-rank decomposition matrices into LLMs’ layers, making it adaptive to new data while retaining the previous knowledge. As LoRA has been widely verified as effective in fine-tuning LLMs and achieving superior performance, this work follows this affordable and reproducible way to develop an efficient tuning framework for multi-round dialogues.

heading: 3 Our Method
text: We first provide the necessary background about general dialogue generation and LoRA-based tuning (see §3.1). Then, we dive into the details of the proposed Multi-round Interactive Dialogue Tuning (MIDI-Tuning) framework (see §3.2).

heading: 3.1 Preliminaries
text: Dialogue Generation We consider a dialogue dataset as D = {(Ii, Ci)}Ni=1 for downstream tasks, where N is the total number of dialogues. Ii denotes task-specific dialogue instruction and necessary additional information, such as domain knowledge facts, specified character descriptions, etc. Ci = {< ui,t, si,t >}Tt=1 denotes utterances between the user ui and agent si in the i-th dialogue, T denotes the total number of dialogue rounds.
Given a task-specific dialogue instruction I that provides necessary information and a dialogue context C = {< u1, s1 >,< u2, s2 >, · · · , < ut, >}, the objective is to generate a proper agent utterance st. Essentially, the probability distributions over the agent’s utterances are estimated as follows:
p =
T∏
t=1
p(st|s<t;u≤t; I) (1)
More generally, if let X = [s<t;u≤t; I] denote the input context and Y denote the output utterance, the language models-based tuning is to minimize the negative log-likelihood as follows:
L(θ) = − N∑
i=1
p(Y (i)) log pθ(Ŷ (i)|X(i)) (2)
where θ denotes all trainable parameters.
Low-Rank Adaptation Low-Rank Adaptation (LoRA) (Hu et al., 2022) hypothesizes that the weight updates in pretrained language models possess a low “intrinsic rank” during adaptation. For a pretrained weight matrix W ∈ Rd×k, it is updated with a low-rank decomposition W +∆W = W + BA, where B ∈ Rd×r, A ∈ Rr×k, and the rank r ≪ min(d, k). During fine-tuning, W is frozen with no gradient updates, while A and B are trainable, making LoRA tuning is much more efficient than full fine-tuning. In practice, LoRA can be specified to adapt the attention weights Wq, Wk, Wv, and Wo corresponding to LLMs’ query, key, value, and output projections.

heading: 3.2 MIDI-Tuning
text: We propose the MIDI-Tuning framework, which enables the agent and user to achieve round-level interactions. Figure 2 shows the overview of our framework. Below, we introduce how it works, from intuitive ideas to technical details.
User-Agent Separate Modeling As we emphasized before, it is more feasible to model the roles of the agent and user separately, such that the agent and user models can consistently adhere to their respective roles. As shown in Figure 2, we employ an agent adapter like LoRA (Hu et al., 2022) built upon an LLM, e.g., LLaMA (Touvron et al., 2023a), to model the dialogue instruction for the agent (denoted as sinst) and the agent’s utterances st (t ≥ 1). Here, t denotes the dialogue round. We employ another LoRA adapter built upon the same LLM to model the user’s utterances ut. The backbone LLM is shared since it is frozen during tuning, while the weight parameters of the two LoRA adapters are trainable to distinguish role disparities between the agent and user.
Mathematically, we decompose the probability distribution over all the utterances in dialogue into two distributions for the user model and agent
model, respectively, shown as follows:
p(u) = T∏
t=1
p(ut|u<t; s<t) (3)
p(s) = T∏
t=1
p(st|s<t;u≤t; I) (4)
where p(u) and p(s) are language models whose task is to predict the next token given the preceding context. Finally, the objective of our tuning is to optimize the joint losses of the agent model (denoted as Lst) and user model (denoted as Lut):
L = Lst + βLut (5)
where β is a hyperparameter controlling the weight.
Round-level Memory Caching One of the key challenges of the introduced separate modeling is that tracking the complete dialogue context is nontrivial. To address it efficiently, we propose a roundlevel memory caching mechanism in this section. Since Transformer (Vaswani et al., 2017) lays the foundation architecture of existing LLMs, it is capable of applying memory recurrence (Dai et al., 2019; Wang et al., 2020; Wu et al., 2021) to cache Transformer’s self-attention (Q, K, V for queries, keys and values) computations to maintain context information during user-agent interactions.
Let us define ht as the hidden states for an utterance at t-th round, Mt as the memory at t-th round containing all key-value pairs from the past. As shown in Figure 2 and Figure 3, we reuse history keys (i.e., K≤t−1) and values (i.e., V≤t−1) as the cached memory Mt−1, to perform self-attention computation to obtain ht, and then store ht back to the memory as Mt. The keys and values computed from previous rounds are fixed and cached to be reused as ongoing context when the agent/user model processes the present-round utterance (see Figure 3), allowing the model to exploit information in history. The entire process is formulated as follows:
Mt = [(K (1) ≤t , V (1) ≤t ), · · · , (K (l) ≤t, V (l) ≤t )] (6)
K (i) ≤t = [K (i) ≤t−1;K (i) t ] (7) V (i) ≤t = [V (i) ≤t−1;V (i) t ] (8)
h (i) t = Attention(Q (i) t ,K (i) ≤t−1, V (i) ≤t−1) (9)
where [·; ·] denotes concatenation, h(i)t is the hidden states at the i-th layer. The last layer’s hidden states
ht is used to calculate loss during tuning and to obtain generation probability during inference.
Since there are two LoRA adapters, two individual Wq weight matrices for query projections will be trainable. Recall that obtaining a good agent model is the ultimate goal, we adopt a context value protection strategy to train the agent model’s value projection, i.e., the weight matrix Wv of LoRA, without training the user model’s value projection. This operation enables the agent model to exploit context value in a consistent space.
Tuning and Inference Although the idea presented before is appealing, some technical challenges still need to be solved in practice. First, the rounds of different dialogues and sequence lengths of different utterances within one dialogue might be unequal, how can we achieve batched tuning on downstream data? We pad batched utterances to the maximum utterance length within a batch, and similarly, pad instructions to the maximum batched instruction length. We sort batched dialogues by their rounds in descending order, similar to inverted triangular causal masks, making it easier to compute losses for valid utterances. We set a maximum number of rounds according to downstream tasks, truncating early-round utterances if longer.
Second, the paddings among different rounds result in the positions of utterance tokens not continuous since most LLMs adopt the Rotary Position Embedding (Su et al., 2021). When reusing the cached memory, how can we keep the positional information consecutive? To this end, we set valid positional ids at each round by counting valid tokens and masking out positions that should not be seen. Then, we explicitly pass the necessary positional ids as part of the model input during both training and inference.
Our inference process differs from that of traditional methods. We feed the past ground-truth utterances < u≤t, s<t > round by round to obtain the cached memory, which is finally used to generate the agent’s corresponding utterance st at t-th round. In realistic interactions, we use the memory yielded from previously generated utterances since we do not have ground-truth dialogue history.

heading: 4 Experiments

heading: 4.1 Experimental Setting
text: Tasks We consider validating our framework on two challenging dialogue tasks: character-based
dialogue (Urbanek et al., 2019; Han et al., 2022; Chen et al., 2023a) and target-oriented proactive dialogue (Wang et al., 2023a,b; Deng et al., 2023). For character-based dialogue, the challenge for an agent lies in maintaining character identity consistent with the assigned role throughout the conversation, where the agent may incorrectly take on the roles or activities of its faced users (Shuster et al., 2022) instead of its assigned role. For targetoriented proactive dialogue, an agent should proactively direct the conversation towards its assigned target (a specific goal) step by step. This long-term goal-directed behavior makes it non-trivial to maintain the consistency that adheres to its goal with the dialogue rounds moving forward.
Datasets Our experiments are conducted on the LIGHT (Urbanek et al., 2019) and TOPDIAL (Wang et al., 2023a) datasets. LIGHT is a characterbased dialogue dataset collected from crowdworker interactions with a set of game location settings (e.g., countryside, forest, castle). It contains various game characters, from animals to humans (e.g., dragon, wizard, servant). Each dialogue has a background description of the setting, while each character has a persona with several sentences describing its traits (see detailed examples in Appendix A). Table 1 shows statistics of LIGHT, where the seen test set consists of dialogues with their locations and characters that can appear in the training set. In contrast, the unseen test set comprises dialogues collected on the unseen set of locations and characters, providing a more challenging test.
TOPDIAL is a target-oriented dialogue dataset for proactive agents with personalized users. The agent is assigned a target consisting of a <dialogue act, topic> pair, where these target dialogue acts mainly lie in recommendations on the domains of movies, music, and food. The agent must proac-
tively lead the discussed topic towards the target topic based on domain knowledge, and meanwhile, adapt its faced user’s personalized aspects (e.g., profiles and personalities) to maintain engagement instead of obtrusively driving to the target. The agent’s ultimate task is to achieve the target act on the target topic (see detailed examples in Appendix A). Table 2 shows statistics of TOPDIAL. Appendix A describes more details for preprocessing the data into the general format with instructions.

heading: 4.2 Evaluation
text: Consistency Evaluation Consistency evaluation in dialogue has been a long-standing yet challenging problem (Nie et al., 2021; Shuster et al., 2022; Han et al., 2022). Inspired by these prior studies, we utilize a binary classifier trained on the downstream datasets to measure the consistency probability (Consist. Prob.) of the agent’s generated responses. We concatenate a given context input and a response as the complete input to yield the classification label y ∈ {1(consistent), 0(inconsistent)}. For each ground-truth (consistent) response in the LIGHT dataset, we construct the inconsistent set by sampling from (1) the user’s utterances under the current dialogue setting due to the user’s character being obviously different from the agent’s, and (2) the agent’s utterances with the same character but under different dialogue settings. Then, we finetune a pretrained BERT (Devlin et al., 2019) model followed by a linear layer for binary classification, producing an automatic consistency estimator. We also employ this method to train a consistency estimator on the TOPDIAL dataset accordingly. Appendix B.1 provides complete details.
Figure 4 shows the performance of our trained consistency estimator on the LIGHT validation set. The convex ROC curve (see Figure 4(a)) with an AUC (Area Under the Curve) value of 0.95 shows
that our estimator is highly discriminative in recognizing whether an agent’s response is consistent with the dialogue context. Meanwhile, the calibration curve (see Figure 4(b)) indicates that our estimator has high confidence in its predicted probabilities for positive (i.e., consistent) responses. Therefore, it is reliable to use our trained estimator to automatically measure the consistency of an agent’s generated response during evaluation.
In addition, we adopt the state-of-the-art LLM, GPT-4 (OpenAI, 2023), to automatically evaluate the score of dialogue consistency (GPT-4 Score), similar to existing works (Zheng et al., 2023; Lu et al., 2023). We take the necessary checking information (e.g., specified character descriptions), dialogue context, and the agent’s generated response as a whole, then ask GPT-4 to rate the consistency with an integer scale of 1 ∼ 10. We provide the details of the prompt setting in Appendix B.2.
Dialogue Evaluation Metrics In addition to consistency, we also adopt commonly used automatic evaluation metrics for dialogue generation. Our evaluation metrics include word-level F1 (Word F1), BLEU-n (Papineni et al., 2002), and distinct (DIST) (Li et al., 2016) for the LIGHT dataset. For the TOPDIAL dataset, we adopt the Word F1, BLEU-n, and target success rate (Succ.) (Wang et al., 2023a), following prior studies (Wang et al., 2023a; Dao et al., 2023) for target-oriented proactive dialogue. Appendix B.3 provides the details of the above metrics.

heading: 4.3 Implementation
text: Baseline Models We adopt popular open-source LLMs as baseline models for experiments, including LLaMA (Touvron et al., 2023a), Mistral7B (Jiang et al., 2023), Vicuna (Chiang et al., 2023), and LLAMA 2-Chat (Touvron et al., 2023b). As our primary focus is the way of tuning, we mainly consider using 7B-size models since they are widely compute-affordable. Our framwork can be easily adapted to much larger models, e.g., with a size of 13B or 70B.
Baseline Settings We consider the following two settings for all baseline models: (i) No Tuning, which indicates that each model directly takes the concatenated text of the task instruction and a dialogue context as input prompt, then generates an utterance as the agent’s response. Since this setting performs without any tuning, it can be used to measure the fundamental chat ability of an LLM and
the difficulty level of a downstream dialogue task. We also include the GPT-3.5-Turbo version of ChatGPT as an additional baseline for this setting. (ii) Fine-tuning, which tunes an LLM based on LoRA (Hu et al., 2022) using the downstream training set, following the conventional one-dialogue-onesample tuning manner since the adopted baseline models are all causal LLMs.
Note that we ensure the input task instructions of the No Tuning, fine-tuning, and ours are identical for a test sample, following the format as described in Appendix A. This will mitigate the influence of generation caused by different instructions. We have two additional special tokens, e.g., [USER] and [ASSISTANT], inserted ahead of each utterance from the user and agent, respectively.
Implementation Details We implement the baseline settings and our MIDI-Tuning using the Huggingface PEFT (Mangrulkar et al., 2022) library, and we incorporate DeepSpeed (Rasley et al., 2020) to improve the training efficiency. For all experi-
ments that involve tuning, the LoRA’s target modules are Wq and Wv, the rank r is set to 8, and the scaling parameter α is set to 16. We adopt 4-bit quantization (Dettmers et al., 2023) for efficient finetuning of LLMs. The optimizer we used
is AdamW (Loshchilov and Hutter, 2018), with a warmup ratio of 0.03. The learning rate is set to 2e-5 with a cosine scheduler. Due to the memory constraint, the maximum number of dialogue rounds is set to 10. The maximum text window for all models is set to 2k, sufficient to cover the context length for the two datasets. The hyperparameter β is set in the range (0, 1]. The other hyperparameters are set as the default, following Vicuna (Chiang et al., 2023). Appendix C provides more details on tuning and inference.

heading: 5 Results and Discussions

heading: 5.1 Automatic Evaluation Results
text: Table 3 and Table 4 report the automatic evaluation results on the LIGHT test-seen and test-unseen datasets, respectively. Though GPT-3.5-Turbo performs very well, we find that the majority of baseline models with no tuning perform inferior, indicating that merely relying on prompting may not be effective enough for the LIGHT-like dialogue tasks. With downstream training data, vanilla fine-tuning enables these baseline models to deeply understand a specific dialogue task and achieve much better generation performances. Nonetheless, our MIDITuning outperforms vanilla fine-tuning remarkably in terms of the consistency probability and GPT-4 score, and meanwhile, achieves higher or on par with scores in other dialogue generation metrics (e.g., word F1, BLEU). We observe a similar trend between the consistency probability predicted by our created estimator and the GPT-4 score rated by GPT-4, widely demonstrating the effectiveness of our framework in consistency improvement. Similarly, as shown in Table 5, our MIDI-Tuning performs better than vanilla fine-tuning on the TOP-
DIAL dataset. Overall, our MIDI-Tuning is superior in generating more consistent responses without compromising much performance in other aspects.

heading: 5.2 Per-round Consistency Analysis
text: To look at how our MIDI-Tuning performs as the dialogue rounds moving forward, we visualized perround consistency comparison between the finetuning (FT) and ours on the LIGHT test-unseen set. Figure 5 shows the comparison results, where the curve for gold response is obtained by feeding ground-truth response at each round into the created consistency estimator, serving as an approximal upper bound for per-round consistency.
As shown in Figure 5, LLaMA-7B with our MIDI-Tuning achieves a large margin of consistency improvement compared to that with FT. We highlight the importance of our framework for promoting consistent dialogue generation since LLaMA is a foundation LLM with neither instruction tuning nor human alignment, which can be a fair pedestal for comparing different tuning methods. We observe that Vicuna-7B with FT performs closely compared to Vicuna-7B with ours, which might be attributed to Vicuna’s intrinsic powerful chat ability since it is an instruction-tuned LLM based on a variety of dialogue data. More importantly, the estimated consistency of both LLaMA7B and Vicuna-7B with our MIDI-Tuning drops slowly and even maintains stable (e.g., 5 ∼ 7 rounds), while the estimated consistency is continuously declining with the dialogue rounds going on for the two models with vanilla fine-tuning. It suggests that the proposed method is able to maintain multi-round dialogue consistency.

heading: 5.3 Human Evaluation
text: To further assess the performance of the proposed framework in realistic multi-round dialogue scenarios, we conducted an interactive evaluation following existing studies (Li et al., 2023; Cheng et al., 2024). We used ChatGPT to simulate the roles of different users based on the TOPDIAL test set and converse with the evaluated models round by round, where we adopted LLaMA and Vicuna, tuned by vanilla fine-tuning (FT) and our MIDITuning. Given a pair of dialogues produced by two variants of tuning for the same backbone model, we recruited three well-educated graduate students as the annotators to evaluate which one is better (or tied) from the dimensions of consistency (Consist.), proactivity (Proact.), coherence (Coh.), and target success rate (Succ.). We provide the metrics explanations and detailed setting in Appendix D.
Figure 6 shows the comparison results between FT and ours. We obtain an average Fleiss’s kappa of κ = 0.486, indicating a moderate (0.41 < κ < 0.60) agreement among annotators. The results shown in Figure 6 suggest that our MIDI-Tuning significantly outperforms FT in maintaining consistency (56.2% win rate for LLaMA) in the interactive setting. It is also superior to or on par with FT in other dimensions. To give a better sense of generation quality, we provide a case study in Appendix E. In summary, our MIDI-Tuning is more effective in generating consistent, coherent, and appropriate utterances.

heading: 6 Conclusion
text: This work explores how the way of tuning can improve the consistency of dialogue generation over multiple rounds. We highlight the importance of separately modeling agents and users due to their role disparities. We propose a general, efficient tuning framework called MIDI-Tuning, which rep-
resents the agent and user using two adapters and tunes them via round-level memory caching. Empirical experiments show that our framework outperforms traditional dialogue tuning significantly.
Limitations
We recognize the limitations of this work in the following aspects. First, our MIDI-Tuning framework requires padding among dialogue rounds to achieve batched tuning. It might result in redundant GPU memory consumption (see Appendix Table 6) as the rounds become longer and longer. Second, the current framework needs to be more compute-efficient since it cannot compute losses in parallel for different rounds of utterances. We will consider improving the compute efficiency of our framework by employing advanced acceleration techniques, such as FLASHATTENTION (Dao et al., 2022). Third, the MIDI-Tuning relies on the architecture of causal language models for encoding, decoding, and round-level memory caching. It cannot directly tune encoder-decoder language models for dialogue generation. We will leave addressing these challenges as our future work.
Ethics Statement
This work mainly focuses on developing a general, efficient framework to tune LLMs for multi-round dialogue generation. The adopted LLMs for tuning are all open-sourced. We strictly follow the protocols for the academic use of these LLMs. Our experimental datasets are publicly available and do not involve sensitive or private information. It is also known that response generation from these LLMs may have concerns about toxicity and bias. Thus, we emphasize that ensuring safe deployment and interaction is a necessity. In addition, we partially use AI assistants, such as Copilot and ChatGPT, to assist us with coding and writing.

heading: Acknowledgments
text: This work was supported by the National Natural Science Foundation of China (62076212), the Research Grants Council of Hong Kong (15207122, 15207920, 15207821), and PolyU internal grants (ZVQ0, ZVVX). The authors would like to thank the anonymous reviewers for their valuable feedback and constructive suggestions.

heading: A Dataset Preprocessing
text: LIGHT Dataset Figure 7 shows an example dialogue from the LIGHT (Urbanek et al., 2019) dataset. For each dialogue in the dataset, we transform the given character-related information and the setting description into natural languages, following the instruction template presented in Figure 8. As such, we obtain dialogue data with the instruction-based format (a dialogue instruction for the agent and multi-round user-agent utterances), as we introduced in the preliminaries.
TOPDIAL Dataset Figure 9 shows an example dialogue from the TOPDIAL (Wang et al., 2023a) dataset. For each dialogue in the dataset, we transform the assigned target, domain knowledge facts, and user information into natural languages, following the instruction template (Wang et al., 2023a) presented in Figure 10. Similarly, we obtain dialogue data with the instruction-based format, as we introduced in the preliminaries.

heading: B Evaluation Settings
text: B.1 Buildup of Consistency Estimator
We first report data preprocessing for building the consistency estimator. For the LIGHT dataset, the
input is a concatenation of (1) dialogue setting description, (2) the agent’s character-related information, (3) the dialogue history within the latest 4 utterances (to alleviate potential training bias caused by the dialogue history with different rounds, following Shuster et al. (2022)), and (4) a candidate agent response Xb. The output label will be 1 (i.e., consistent) if Xb is the ground-truth positive response Xb+ in the dataset, while it will be 0 (i.e., inconsistent) when Xb is a sampled negative response Xb− . For each positive response, we construct the candidate set containing negative (inconsistent) responses by sampling from (1) the user’s utterances under the current dialogue setting due to the user’s character/role being obviously different from the agent’s character/role, and (2) the agent’s utterances with the same character information but under different dialogue settings. Our obtained training and evaluation data have the proportion of positives to negatives = 1:10. The original validation set is used to evaluate the created estimator, while the original training, test-seen, and test-unseen sets are utilized for training. The original test sets can be included here for training to enhance test confidence since the created estimator is to judge other dialogue models on the test sets.
Similarly, the input for the TOPDIAL dataset is a concatenation of (1) the agent’s target (a <dialogue act, topic> pair), (2) the user’s profile information, (3) the dialogue history within the latest 4 utterances, and (4) a candidate agent response Xb. We construct the candidate set containing negative (inconsistent) responses by sampling from the agent’s utterances that satisfy the two requirements: (1) They are from other dialogues assigned with the same target act but with different target topics, and (2) their progression step is similar (i.e., at the same round or the neighboring round) to that of the groud-truth response. The reason is that the target acts in the dataset are limited, while the target topics are diverse. The agent needs to proactively direct the dialogue towards its assigned target <dialogue act, topic> (or a specific goal) step by step. This long-term goal-directed behavior leads to maintaining consistency that adheres to its goal, with the dialogue rounds moving forward to be the main challenge. The proportion of positives to negatives is also set to 1:10.
We build the consistency estimator per each dataset individually. We fine-tune a pretrained
BERT (Devlin et al., 2019) base model2 followed by a linear layer for binary classification. The maximum sequence length is limited to 512. The initial learning rate is 2e-5, with a warmup of the first 500 training steps. The model is trained for a maximum
2https://huggingface.co/google-bert/ bert-base-uncased
of 3 epochs, with a batch size of 32.
B.2 Consistency Evaluation by GPT-4 Inspired by LLM-as-a-judge strategy (Zheng et al., 2023; Lu et al., 2023), we adopt the state-of-the-art LLM, GPT-4 (OpenAI, 2023), to evaluate the score of dialogue consistency automatically. To this end, we take the necessary prompt as input and execute
an API call of GPT-4-turbo3, asking it to rate the consistency with an integer scale of 1 ∼ 10. Due to the discrepancy between character-based dialogue and target-oriented proactive dialogue tasks, Figure 11 and Figure 12 show the evaluation prompts for the LIGHT and TOPDIAL datasets, respectively.
B.3 Dialogue Evaluation Metrics
The word-level F1 (Word F1) is a commonly used metric to evaluate dialogue generation, which estimates the precision and recall at the word level by comparing the generated and ground-truth re-
3https://platform.openai.com/docs/models/ gpt-4-and-gpt-4-turbo
sponses. By considering word order, the BLEU-n (Papineni et al., 2002) calculates n-gram overlaps between the generated and ground-truth responses. The distinct (DIST) (Li et al., 2016) score measures the diversity of the generated responses, where DIST-1 and DIST-2 are the number of distinct unigrams and bigrams divided by the total number of generated words. The target success rate (Succ.) (Wang et al., 2023b) counts the proportion of correct target topic generation within the ground-truth round and the adjacent rounds in the test set. It measures how successfully a model can achieve the target exactly.

heading: C Additional Implementation Details
text: The open-source LLMs we adopted are listed as follows: LLaMA-7B4 (Touvron et al., 2023a), Mistral7B5 (Jiang et al., 2023), Vicuna-7B6 (Chiang et al., 2023), and LLAMA 2 Chat-7B7 (Touvron et al., 2023b).
For fair batched tuning, we use gradient accumulation and set the accumulation steps according to different tuning methods, ultimately achieving the same global batch size of 16 and tuning for 3 epochs. During inference, we adopt Nucleus Sampling (Holtzman et al., 2020) decoding with top-p 0.75 and top-k 40 to generate a response token by token, with a maximum decoding length of 100.
We experiment on one server equipped with 4 NVIDIA V100 GPUs. Table 6 shows the GPU memory usage between fine-tuning (FT) and our MIDI-Tuning on the LIGHT dataset. We observe that the memory usage of our framework is larger due to the need for padding batched utterances and round-level memory caching.

heading: D Human Evaluation Details
text: Our human evaluation was conducted based on the TOPDIAL test set, where we randomly selected 100 dialogue examples with various targets for experiments. We used ChatGPT (GPT-3.5-turbo version) to simulate the roles of different users according to the user profiles and personality information provided in the dataset, following the prompt template for the user as per Wang et al. (2023a). Then, we asked ChatGPT to converse with each evaluated model round by round, resulting in dialogues between the user simulator and an agent model. We set a maximum limit of 8 interactive rounds based on our observation of target accomplishment.
Given a pair of dialogues produced by two variants of tuning for the same backbone model, we
4https://huggingface.co/yahma/llama-7b-hf 5https://huggingface.co/mistralai/
Mistral-7B-v0.1 6https://huggingface.co/lmsys/vicuna-7b-v1.3 7https://huggingface.co/meta-llama/ Llama-2-7b-chat-hf
asked our human annotators to compare each pair of dialogues in terms of the following metrics: consistency (Consist.), proactivity (Proact.), coherence (Coh.), and target success rate (Succ.). We used a question form to describe these metrics, with the wording of questions presented as follows:
• Consistency (Consist.): Which dialogue shows that the agent is more consistent with its assigned target and the dialogue context under the specified setting?
• Proactivity (Proact.): Which dialogue shows that the agent takes the initiative during the conversation and proactively leads the topic threads toward the target topic?
• Coherence (Coh.): Which dialogue is more natural and coherent, like humans? Whose dialogue context flows more smoothly?
• Target Success Rate (Succ.): Which dialogue successfully achieves the target dialogue act on the target topic?
The agreement among the annotators is measured by Fleiss’s kappa (Fleiss, 1971). We reported each model’s average score from different annotators as the ultimate evaluation result. In addition, we transparently informed all human annotators of our research intent and provided enough time for them to complete the evaluation.

heading: E Case Study
text: Table 7 presents some generated cases from the interactive evaluation on the TOPDIAL test set.

heading: Future Work
text: We will leave addressing these challenges as our future work