Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models (2024)

Abstract: Humans often interact with large language models (LLMs) in multi-turn interaction to obtain desired answers or more information. However, most existing studies overlook the multi-turn instruction following ability of LLMs, in terms of training dataset, training method, and evaluation benchmark. In this paper, we introduce Parrot, a solution aiming to enhance multi-turn instruction following for LLMs. First, we introduce an efficient but effective method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis. Second, we propose a context-aware preference optimization strategy to further enhance LLMs for complex queries in multi-turn interaction. Moreover, to quantitatively evaluate LLMs in multi-turn instruction following, we manually build a multi-turn benchmark derived from existing ones. Extensive experiments show that Parrot improves current LLMs by up to 7.2% in multi-turn instruction following. Our dataset and codes will be open-sourced to facilitate future research 1.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729–9750 August 11-16, 2024 ©2024 Association for Computational Linguistics
Humans often interact with large language models (LLMs) in multi-turn interaction to obtain desired answers or more information. However, most existing studies overlook the multi-turn instruction following ability of LLMs, in terms of training dataset, training method, and evaluation benchmark. In this paper, we introduce Parrot, a solution aiming to enhance multi-turn instruction following for LLMs. First, we introduce an efficient but effective method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis. Second, we propose a context-aware preference optimization strategy to further enhance LLMs for complex queries in multi-turn interaction. Moreover, to quantitatively evaluate LLMs in multi-turn instruction following, we manually build a multi-turn benchmark derived from existing ones. Extensive experiments show that Parrot improves current LLMs by up to 7.2% in multi-turn instruction following. Our dataset and codes will be open-sourced to facilitate future research 1.

heading: 1 Introduction
text: Large language models (LLMs) (OpenAI, 2022, 2023; Taori et al., 2023; Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023; Li et al., 2023a; Zhou et al., 2023a; Zhao et al., 2023) have demonstrated their strong capability in understanding a range of human instructions. By leveraging synthetic or human-created instructions to fine-tune the LLaMA model (Touvron et al., 2023a,b), a series of studies (Taori et al., 2023; Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023; Li et al., 2023a; Zhou et al., 2023a) achieve promising results, even in some benchmarks (Chiang et al., 2023; Li et al., 2023b) performing close to ChatGPT and GPT-4.
*Equal contributon. This work was performed when Yuchong Sun was visiting Kuaishou as an intern.
†Corresponding author. 1https://github.com/kwai/KwaiYii/Parrot
tion, e.g., anaphora and ellipsis. Directly using ChatGPT to simulate users can not fully mimic the above real-world occasions, while our Parrot-Ask trained on real-world conversations can better human-like queries.
However, most of these studies and benchmarks overlook the multi-turn instruction following ability of LLMs, which is a more common demand in real-world scenarios.
Developing LLMs capable of multi-turn interaction presents significantly greater challenges compared to single-turn interaction. In terms of collecting instruction tuning data, as the number of turns increases, the cost of manual annotation for data collection rises considerably, making it economically unaffordable. Existing user logs with ChatGPT, such as ShareGPT, are limited in quantity, and approximately 60% of the data does not exceed three turns, which is still not enough to achieve satisfactory performance (Chiang et al., 2023). Some LLM-based multi-turn instruction generation methods, even after employing specific prompts to let
9729
ChatGPT act as a user (Ding et al., 2023), usually produce information-complete queries that lack common features found in human queries, such as anaphoras and ellipses, as shown in Fig. 1. In terms of training strategies, most of the current models depend on straightforward supervised finetuning (SFT) methods (Chiang et al., 2023; Ding et al., 2023; Xu et al., 2023b), without specific optimization design for complex queries involving anaphora and ellipsis in multi-turn interactions. It may mislead the LLM to neglect or hallucinate the context when generating responses.
To overcome the above challenges, we propose Parrot, which aims to facilitate the development of LLMs with stronger multi-turn instruction following capabilities. First, we introduce an efficient but effective approach for automatically collecting multi-turn instruction tuning data. Instead of designing complicated prompts to make powerful LLMs (such as ChatGPT or GPT-4) act as users and generate queries, we propose training a ParrotAsk model based on a smaller LLM to learn features found in human queries from a small number of real user dialogues, and then it can be used for generating human-like queries. Second, to enhance the capability of LLMs in handling complex queries during multi-turn interactions, we propose a Context-aware Preference Optimization (CaPO) strategy. We first select queries that heavily rely on contextual information to obtain accurate responses, and then construct negative responses by simulating common error scenarios such as ignoring context or misunderstanding anaphora. Subsequently, we leverage these pairs to optimize the LLM’s preferences so that they can better exploit contextual information when generating responses.
Moreover, for a quantitative assessment of LLMs’ multi-turn abilities, we ask annotators to expand the MT-Bench benchmark (Zheng et al., 2023) that originally consists of two-turn queries, to an eight-turn MT-Bench++ benchmark. MT-Bench++ includes complex queries like anaphoras, ellipses, and topic transitions, better reflecting real multiturn interaction. We conduct extensive experiments on MT-Bench and MT-Bench++. Compared with previous approaches, our proposed method can generate higher-quality multi-turn instruction tuning data with more human-like queries. Our multi-turn instruction dataset combined with our proposed CaPO strategy improves the current LLMs by 7.2% in multi-turn instruction following evaluation.
We summarize our contributions as follows:
• We propose Parrot, a novel method with a new instruction dataset for enhancing the multiturn instruction following capability of LLMs.
• We design CaPO, a training strategy that simulates common errors in multi-turn conversation, and learns to avoid them in generation.
• Our model trained on Parrot dataset with CaPO achieves superior performance among 13B open-source LLMs, especially for the multi-turn instruction following capabilities.

heading: 2 Related Work

heading: 2.1 Instruction Tuning for LLMs
text: Instruction tuning plays an important role in inspiring the instruction following ability of LLMs and aligning with humans (Wang et al., 2022b; Wei et al., 2021; Ouyang et al., 2022; OpenAI, 2022, 2023). Due to the expensive costs to collect human-annotated instruction tuning data (Conover et al., 2023; Ouyang et al., 2022), recent works explore leveraging the powerful LLMs to generate instruction-response pairs in an automatic manner (Taori et al., 2023; Ding et al., 2023; Xu et al., 2023b; Wang et al., 2022a; Xu et al., 2023a; Peng et al., 2023; Anand et al., 2023). SelfInstruct (Wang et al., 2022a) designs seed prompts as examples to prompt GPT-3 (Brown et al., 2020) to generate instructions. Alpaca (Taori et al., 2023) adopts the same pipeline to collect instructionresponse pairs using ChatGPT and then fine-tune a LLaMA model (Touvron et al., 2023a). Humpback (Li et al., 2023a) proposes instruction backtranslation that trains an LLM to generate instructions for web corpus. However, they mainly focus on single-turn instructions. Baize (Xu et al., 2023b) collects multi-turn instructions by leveraging ChatGPT to generate dialogues in a self-chat manner. UltraChat (Ding et al., 2023) utilizes two ChatGPT APIs to play the roles of user and assistant respectively. Vicuna (Chiang et al., 2023) adopts user-ChatGPT logs from the ShareGPT platform for instruction tuning. A concurrent work also attempts to train a user simulator to collect instruction tuning data (Kong et al., 2023). However, these multi-turn instruction data still have several drawbacks, such as less detailed responses, not human-like instruction, or a limited number of turns.

heading: 2.2 Evaluation of Instruction Following
text: The current benchmarks for LLMs mainly focus on single-turn evaluation (Hendrycks et al., 2020; Zhong et al., 2023; Srivastava et al., 2022; Li et al., 2023b; He et al., 2023; Zhou et al., 2023b). MMLU (Hendrycks et al., 2020) and Big-bench (Srivastava et al., 2022) are designed as multiple-choice questions to measure the knowledge and reasoning ability of LLMs. AGIEval (Zhong et al., 2023) constructs humancentric evaluation for LLMs from standardized exams. However, the above kinds of evaluation obey the nature of the open-ended generation of LLMs and cannot reflect the ability to follow user instructions (Zheng et al., 2023). Alpaca-Eval (Li et al., 2023b) builds a single-turn instruction following benchmark with 805 open-ended questions and adopts GPT-4 to give evaluation. Chatbot Arena (Zheng et al., 2023) is a platform where users can vote to compare diverse LLMs. MTBench (Zheng et al., 2023) builds the first evaluation benchmark for multi-turn instruction following. It adopts GPT-4 to judge the quality of model responses and shows there is a high agreement to human evaluation. However, MT-Bench only contains two queries for each session, thus cannot reflect the ability of LLMs to handle multi-turn instruction following. Thus we build an eight-turn MT-Bench++ benchmark based on MT-Bench in this work.

heading: 3 Approach
text: We describe our proposed Parrot framework in this section. As illustrated in Fig. 2 (a), we first train a Parrot-Ask model to mimic the asking style of
humans in generating multi-turn queries and then use it for collecting a multi-turn instruction tuning dataset. Then we design three strategies to construct negative responses, involving context neglect, context hallucination, or context misunderstanding as shown in Fig. 2 (b), to compose context awareness preferences. Finally, we use the collected data to train the Parrot-Chat model by instruction tuning (See Fig. 2 (c)) and context-aware preference optimization (See Fig. 2 (d)) to enhance its multi-turn instruction following capability.

heading: 3.1 Preliminary
text: Multi-turn instruction following refers to a process of successive interactions between a user and a model, where the user poses queries and the model responds. This cycle of query and response continues until a desired answer is reached or all necessary information has been gathered. Instruction tuning (Wang et al., 2022b; Wei et al., 2021; Ouyang et al., 2022; OpenAI, 2022) is capable of greatly improving the capability of LLMs to follow human instructions and generate helpful responses. Existing work mostly collects single-turn or multi-turn query-response pairs to compose the instruction dataset for tuning LLMs (Taori et al., 2023; Xu et al., 2023b; Ding et al., 2023; Chiang et al., 2023; Touvron et al., 2023b). For multi-turn instruction tuning, a training sample typically consists of T query-response pairs:
X = (X1q , X 1 r , X 2 q , X 2 r , ..., X T q , X T r , ), (1)
where q denotes query and r denotes response. All the tokens from these query-response pairs are concatenated to a sequence and then processed by an LLM. The loss for multi-turn instruction tuning (Chiang et al., 2023) is similar to language modeling loss but only computed on the response tokens as:
L = − L∑
i=1
log p(xi|Xq,<i, Xr,<i), xi ∈ Xr, (2)
where L is the token length of sequence X , xi is the current predicted response tokens, Xq,<i and Xr,<i are the context query and response tokens before xi.

heading: 3.2 Dataset Collection
text: Human queries exhibit diverse styles and features, such as anaphora and ellipsis, which lead to significant divergence between multi-turn interactions
and single-turn ones. In application, such kinds of natural queries are common and also important in user experience, but have not been fully considered in existing work due to the difficulty in collecting them via existing instruction synthesis methods (e.g., prompting ChatGPT) (Xu et al., 2023b; Ding et al., 2023). Therefore, in this part, we propose training a specialized Parrot-Ask to generate queries using the available real user-ChatGPT logs based on LLaMA (Touvron et al., 2023a). Then we employ Parrot-Ask to interact with ChatGPT and thus collect multi-turn instruction tuning data.
Training Parrot-Ask Model. The process of training the Parrot-Ask model is essentially the inverse of standard instruction tuning. Specifically, instruction tuning is trained by predicting response tokens, conditioned on the user query and the conversation history; whereas, the Parrot-Ask model is trained to predict query tokens, conditioned on the assistant query and the conversation history. Accordingly, we modify the training loss from Eq. 2 to focus exclusively on the query tokens as:
Lask = − L∑
i=1
log p(xi|X<iq , X<ir ), xi ∈ Xq, (3)
this adjustment enables the model to learn to generate queries conditioned on conversation history.
Collecting Data with Parrot-Ask. We utilize ChatGPT to produce responses corresponding to the queries. To ensure that the first-turn queries are meaningful and topic-rich, and can be fairly compared with existing multi-turn instruction datasets during ablation experiments, we first sample 20K first-turn queries from the two most popular multiturn instruction datasets, ShareGPT 2 dataset and UltraChat dataset respectively (Ding et al., 2023). First-turn queries in the ShareGPT dataset come from real users and have greater authenticity, while first-turn queries in UltraChat dataset have a diverse range of topics. When given an initial query X1q , we first use ChatGPT to generate an appropriate response X1r . We then employ Parrot-Ask to generate a new query, denoted as X2r . This conversation is then continued by iteratively generating subsequent responses and queries until we reach the target number of turns. Similar to UltraChat, we have filtered out some repetitive questions, short questions, or sensitive information. A total of 1.91% of
2https://huggingface.co/datasets/ anon8231489123/ShareGPT_Vicuna_unfiltered
the queries in our dataset are identified as having issues. Among these, 1.28% are removed due to repetitive content, 0.40% are excluded for being too short (less than three words), and 0.23% are flagged by the OpenAI ChatGPT service for including sensitive information. Note that within a session, if a current query has an issue, we retain only the preceding utterances, and the current and subsequent rounds are discarded.

heading: 3.3 Context-Aware Preference Optimization
text: In contrast to single-turn instruction following, multi-turn instruction following presents a challenge for LLMs to handle complex contexts that may contain anaphora and ellipsis, which need to leverage context to infer missing information. To enhance the multi-turn instruction following ability, we propose a training strategy called Contextaware Preference Optimization (CaPO), which constructs three types of preferences between positive and negative responses in context and uses them to train the LLM with Direct Preference Optimization (DPO) (Rafailov et al., 2023).
As not all queries need context, we choose those relying on context to compose the preference data. Specifically, we use pronoun recognition and leverage the judgment capabilities of GPT-4 to identify queries that require contextual information for generating accurate responses (more details are in the supplemental materials). In our work, we choose 10K queries of this kind. Then, we design three strategies for constructing negative responses, corresponding to three situations where LLMs are not aware of context in a correct way:
• Context Neglect. By using ChatGPT to generate responses without considering the dialogue history, we simulate the incorrect response an LLM might produce when it does
not reference the dialogue history.
• Context Hallucination. We first prompt ChatGPT to guess what anaphora or ellipsis refers to or what is omitted without seeing the context, and then generate responses based on its guesses. This strategy mimics the situation where the LLM lacks sufficient referential reasoning capabilities, hence has to hallucinate the irrelevant context.
• Context Misunderstanding. We intentionally instruct ChatGPT to select irrelevant information from the conversation history and misinterpret it as ellipsis or anaphora information in the current query to generate a response. This method simulates the deficiencies of LLMs in context comprehension ability or the recognition of referential ellipsis thus leading to mistakes.
Based on these three strategies, we construct corresponding negative responses and use them together with positive responses to train the LLM with preference optimization using DPO algorithm. Through this approach, we train the LLM to avoid generating the above errors in application, which improves the ability of LLMs to understand complex queries in multi-turn instruction following, especially for the situations requiring the background information from the context.

heading: 3.4 Comparison to Previous Works
text: Our work focuses on enhancing the multi-turn instruction following abilities of LLMs, to improve the user experience in real world. As there are also several related works that focus on improving instruction tuning of LLMs, we discuss our major difference with them in this part.
As the statistics shown in Table 1, we can see that although several instruction tuning works (i.e., Baize (Xu et al., 2023b), UltraChat (Ding et al., 2023) and ShareGPT) have utilized the multi-turn instruction dataset, their average turns are generally fewer than our proposed Parrot-40k dataset. It indicates that our dataset is more useful for capturing the multi-turn characteristic in real-world conversations. Furthermore, as measured by the Self-Rouge metric 3, Parrot-40K demonstrates similar or better query diversity than other datasets. It also indicates the quality of our dataset. Besides, we ask GPT-4 to evaluate how many queries on average in a session are context-dependent, which contain anaphoras, ellipses, and other elements. Compared to Baize and UltraChat, Parrot-40K contains much more context-dependent queries, nearly to ShareGPT. To verify the quality of our data, we randomly selected 100 sessions from the Parrot-40K dataset, totaling 953 utterances, for human evaluation. The results indicate that 95.1% of the utterances are contextually relevant, demonstrating the utterance-level fluency of our synthesized conversations.
More importantly, our dataset also provides the negative responses (30k) for context-dependent queries, to better depict the possible errors in the multi-turn conversation. By training on the negatives with RLHF methods, we can further enhance the LLMs to avoid making similar mistakes as the provided ones, which can well guide the training of the LLM towards better human alignment. Concretely, our approach employs ChatGPT under three strategies to generate corresponding negative responses, enabling the LLM to learn from the contrast between positive and negative instances to better utilize contextual information when ellipses or anaphoras occur.

heading: 4 Experiment

heading: 4.1 Experimental Setup

heading: 4.1.1 Evaluation Settings
text: MT-Bench Benchmark. MT-Bench (Zheng et al., 2023) has well-designed questions spanning eight categories, including writing, coding, math, among others. However, since its instructions are limited to 2 turns, it is hard to comprehensively assess the capability of LLMs to follow multi-turn
3We utilize the "Self-Rouge" metric to measure the diversity of queries within a session. The specific calculation involves pairwise computation of the ROUGE scores between the queries and then averaging those scores. A lower value indicates a greater diversity of queries in a session.
instructions. MT-Bench employs GPT-4 to evaluate the responses, assigning a score from 1 to 10 as the final result.
MT-Bench++ Benchmark. To quantitatively evaluate the ability of LLMs to follow long-turn instructions, we expand MT-Bench by manually annotating six additional follow-up questions, creating an eight-turn evaluation benchmark called MT-Bench++. There are 80 sessions and 640 utterances in MT-Bench++.
During the annotation process, we instruct the annotators to pose queries that are not only clear and fluent but also rich in ellipsis and anaphora, thereby introducing a higher level of challenge to better assess multi-turn capabilities. For the queries submitted by our annotators, we have conducted multiple rounds of manual quality checks, revising any queries that do not meet our standards to ensure that all queries adhere to the standards above. We show an example from our MT-Bench++ benchmark in Tab. 2. Following MT-Bench, we employ GPT-4 to evaluate the quality of responses at each turn, the scoring range for GPT-4 evaluation is from 1 to 10, and we report the average GPT-4 score as the final result. We provide GPT-4 evaluation prompts, comprehensive instructions for annotators, and more cases in Appendix A.

heading: 4.1.2 Baselines
text: We compare Parrot-Chat with SOTA LLMs including both closed-source and open-source models.
• Baize (Xu et al., 2023b) is a model trained on 200K multi-turn dialogues generated by ChatGPT in a self-chatting manner.
• UltraLM (Ding et al., 2023) is trained with 1.5M conversations from the UltraChat dataset constructed through iterative chatting leveraging two ChatGPT APIs.
• Vicuna (Chiang et al., 2023) is trained with user-ChatGPT logs from ShareGPT. It is one of the most advanced multi-turn instructionfollowing models available.
• ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) are developed by OpenAI. They are the most advanced LLMs today, but only APIs are available to use them.
• LLaMA-2-13B-chat (Touvron et al., 2023b) is trained with 27K human-annotated instruc-
1st Provide insights into the correlation between economic indicators such as GDP, inflation, and unemployment rates. Explain how fiscal and monetary policies affect those indicators.
such as anaphora and ellipsis.
tion tuning data and optimized with Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al., 2020).

heading: 4.1.3 Implementation Details
text: Parrot-Ask Training Details. We build our Parrot-Ask model on the open-source LLaMA13B model. We train it using 70K ShareGPT sessions. We adopt a max length of 4096 tokens for the model. We train the model for 3 epochs with AdamW optimizer in an initial learning rate of 3e5, followed by a cosine learning rate schedule, we warm up the learning rate for 0.1 epoch. We train the model on 8 A100-80G GPUs with a total batch size of 32 and accumulate the gradients for 8 steps.
Parrot-Chat Training Details. The training setting of Parrot-Chat is similar to Parrot-Ask, except for the training data and loss computation. We train the final Parrot-Chat model on our Parrot-40K dataset. Parrot-Chat is built on the LLaMA-2-13B model, consistent with the baseline models used for comparison. For ablation, we also train the model on other datasets, the details are in Sec. 4.3. After instruction tuning, we conduct CaPO using DPO with 30K constructed positive-negative response pairs, utilizing 8 A100-80G GPUs with an effective batch size of 32 and a learning rate of 1e-5.

heading: 4.2 Main Results
text: We show the results of Parrot-Chat and baseline models in Tab. 3. Benefiting from the powerful strong foundation LLMs, high-quality humanannotated instruction tuning data, and further optimization through RLHF, OpenAI’s ChatGPT and
GPT-4 lead in performance compared to other models, but they are only accessible via API. Baize is trained on low-quality data with the issues of less detailed responses, leading to the poorest performance. Vicuna demonstrates good performance with 200K training samples, demonstrating the importance of using high-quality training data. UltraLM, which is trained with 1.5 million data from UltraChat, still falls short of Vicuna on MTBench++ benchmarks, especially in 6rd to 8th turns. We attribute this to the presence of non-humanlike instructions in UltraChat. Among the publicly available models, LLaMA-2-13B-Chat performs the best, which could be due to the highquality human-annotated data and RLHF optimization; however, its training data is not open-sourced, making it impossible to replicate its results.
Among open-sourced models, our Parrot-Chat w/o CaPO model achieves the best performance with only 40K training examples, showing the effectiveness of the multi-turn instruction following dataset collected using our methodology. Our final Parrot-Chat model trained with Parrot-40K and equipped with CaPO achieves the best performance. On MT-Bench, CaPO significantly improves the scores of second-turn queries, proving that the proposed strategy specifically for multi-turn optimization is effective. On MT-Bench++, CaPO also consistently improves performance, in particular over the 3rd to 5th turns.

heading: 4.3 Analysis
text: Analysis of Training Data. Based on the source of the first-turn query, we can split Parrot-40K into two parts: Parrot-20K(S) corresponded to
ShareGPT and Parrot-20K(U) corresponded to UltraChat. We then investigate how LLMs are affected by the instruction-tuning datasets. As Tab. 4 shows, the data collected with Parrot-Ask improves both their counterparts’ performance across two benchmarks. Parrot-20K(S) outperforms ShareGPT-20K by 2.3 scores on MT-Bench, and 0.8 on MT-Bench++, while Parrot-20k(U) outperforms UltraChat-20k more significantly, by 2.4 scores on MT-Bench, and 1.9 on MT-Bench++. Finally, our model trained on Parrot-40K shows a further performance improvement, demonstrating the importance of both human-like instructions and long-turn instruction following data.
Analysis of Session Turns. We further study the influence of session turns in the training dataset. We train models in four settings by truncating the data in Parrot-40K to 1, 3, and 5 turns, as well as using all turns. As shown in Tab. 5 While the model
trained with 1-turn data performed well on MTBench, it performs much worse on MT-Bench++. Increasing the number of turns to 3 and 5 significantly improves performance on MT-Bench++. The model trained with all turns of data performs best, especially on MT-Bench++.
Analysis of Negative Responses. We analyze the effects of our three proposed negative response construction strategies on the final performance, and the results are shown in Table 6. Overall, the negative responses constructed using all three
strategies lead to improved performance. On MTBench, using Context Hallucination gives a significant improvement, while using Context Misunderstanding leads to a slight decrease in performance. However, on the 8-turn MT-Bench++, all three strategies brought significant improvements. This demonstrates that all three strategies we propose to construct negative responses are helpful for LLMs to make better use of context in multiturn interactions. Finally, when the three strategies are combined for CaPO, the performance is further improved, which proves that the diversity of negative responses is also important for enhancing performance.
Analysis of Evaluation Metric. Evaluating instruction-following ability is challenging. Existing work indicates that traditional automatic metrics like BLEU can not align well with the true ability (Wang et al., 2023), and human evaluation is costly and hard to reproduce (Zheng et al., 2023). MT-Bench proposes using GPT-4 for scoring the quality of model-generated outputs, a practice that has gained wide acceptance within the community. We use GPT-4 to evaluate Parrot-Chat three more times to verify the stability of the evaluation. The results are shown in Tab. 7, where the low std. proves that GPT-4 evaluation is generally stable and replicable.
We also conduct a human evaluation to validate the reliability of using GPT-4 for evaluation. We ask three annotators to score the quality of 500 utterances generated by Parrot-Ask and 7 baseline methods. We find a high Spearman correlation coefficient of 0.89 between the human and GPT4 evaluations. Our findings are consistent with those from studies like MT-Bench, confirming the reliability of using GPT-4 for evaluation.
Analysis of Parrot-Ask. Our Parrot-Ask is designed to mimic human style by learning from real user-ChatGPT dialogue data. To verify that the queries generated by Parrot-Ask have a human style, we conducted a human evaluation where annotators evaluated 800 utterances generated by Parrot-Ask and ChatGPT, respectively. The results
Backbone MT-Bench MT-Bench++
LLaMA3-8B 6.54 6.49 LLaMA-13B 6.81 6.56

heading: Acknowledgements

heading: 5 Conclusion
text: Limitations and Ethics Statements
Our curated dataset and the proposed CaPO method have successfully enhanced the multi-turn instruction-following ability of LLMs. However, it is imperative to acknowledge that our work still has its limitations and may raise ethical concerns.
Limitations. In terms of evaluation, although we built an 8-turn MT-Bench++ evaluation set based on MT-Bench, it contains only 80 sessions with a total of 640 queries, which limits the diversity of the samples. We hope that in future work and with the support of the community, more comprehensive multi-turn benchmarks will be developed to more thoroughly evaluate the multi-turn instructionfollowing ability of LLMs.
Regarding data collection, due to cost constraints, we have relied on ChatGPT for our data collection. In the future, it may be possible to use more powerful models, such as GPT-4, to further enhance performance. We can also focus on the selection of high-quality multi-turn training samples, and use larger models to train the Parrot-Ask model with more data to further improve results. These are aspects we plan to explore in our future work.
Ethics Statements. Our work aims to enhance the instruction-following capabilities of LLMs in multi-turn scenarios, but the models we train could have negative impacts. For example, they could be used inappropriately, although we have performed data cleansing to avoid offensive content. However, this is a common issue currently faced in the LLM field, and it is not amplified by this work. In the future, we will consider more work on the safety of LLMs to optimize their security in multi-turn scenarios.

heading: A Details of MT-Bench++
text: In this section, we provide the annotation guidelines given to annotators in Fig. 3, the prompts for GPT-4 used in the evaluation in Fig. 4, and more examples from MT-Bench++ in Tab. 9. For the queries submitted by our annotators, we have conducted multiple rounds of manual quality checks, revising any queries that do not meet our standards to ensure that all queries adhere to the standards.
Example 1
1st Imagine yourself as a doctor tasked with devising innovative remedies for various ailments and maladies. Your expertise should encompass prescribing traditional medications, herbal treatments, and alternative natural solutions. Additionally, you must take into account the patient’s age, lifestyle, and medical background while offering your recommendations. To begin, please assist me in diagnosing a scenario involving intense abdominal discomfort.

heading: B Details of Data Collection
text: B.1 System Prompt of Parrot-Ask Similar to LLMs like Vicuna, we also introduce a system prompt, which serves to remind the model to adopt the role of a user and generate questions: The following is a conversation between a user and an AI assistant. User statements start with [USER] and AI assistant statements start with [ASSISTANT]. You need to tell me what the user is likely to ask in each round of the conversation.
B.2 Prompt for ChatGPT to Act as a User We adopt the prompt from UltraChat, and we have also enhanced it to generate queries that feature ellipses, anaphoras, and other such linguistic characteristics: {###conversation history} Above is a conversation between a user and an intelligent assistant. Now suppose you are the user, say something to continue the conversation based on the given context. Make the response short and the language casual, and incorporate pronouns, ellipses, and other natural language phenomena in your response to make it more akin to real humans.
B.3 Prompt for GPT-4 to Identify Queries that Require Contextual Information
B.4 Examples of Parrot-Ask Generated Queries
B.5 Examples of Parrot-40K Dataset

heading: C Examples of Model Generated Responses
text: In this section, we provide some examples of model-generated responses and the GPT-4 judgments.

heading: Future Work
text: We hope that in future work and with the support of the community, more comprehensive multi-turn benchmarks will be developed to more thoroughly evaluate the multi-turn instructionfollowing ability of LLMs. Regarding data collection, due to cost constraints, we have relied on ChatGPT for our data collection. In the future, it may be possible to use more powerful models, such as GPT-4, to further enhance performance. We can also focus on the selection of high-quality multi-turn training samples, and use larger models to train the Parrot-Ask model with more data to further improve results. These are aspects we plan to explore in our future work