Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Authors: Ning Ding
Yulin Chen
Bokai Xu
Yujia Qin
Shengding Hu
Zhiyuan Liu
Maosong Sun
Bowen Zhou
Venue: EMNLP 2023 Main (2023)
Keywords: Instructional Data
Language Models

Abstract: Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT. Scaling the diversity and quality of such data, although straightforward, stands a great chance of leading to improved performance.
This paper aims to push the upper bound of open-source models further. 
We first provide a systematically designed, diverse, informative, large-scale dataset of instructional conversations, UltraChat, which does not involve human queries.
Our objective is to capture the breadth of interactions between a human user and an AI assistant and employs a comprehensive framework to generate multi-turn conversation iteratively.
UltraChat contains 1.5 million high-quality multi-turn dialogues and covers a wide range of topics and instructions. 
Our statistical analysis of UltraChat reveals its superiority in various key metrics, including scale, average length, diversity, coherence, etc., solidifying its position as a leading open-source dataset.
Building upon UltraChat, we fine-tune a LLaMA model to create a powerful conversational model, UltraLM.
Our evaluations indicate that UltraLM consistently outperforms other open-source models, including WizardLM and Vicuna, the previously recognized state-of-the-art open-source models.

Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Ning Ding Yulin Chen Bokai Xu Yujia Qin Shengding Hu
Zhiyuan Liu Maosong Sun Bowen Zhou't
'Department of Electronic Engincering, BegiineeinggTsiinhheee Tsinghua University DDeparteent of Computer Science and Technology, Tsinghu University
3 BNRIST, IAI, Tsinghua University, *The Chinese University of Hong Kong, Shenzhen
Abstract
Model
Score
Fine-tuning on instruction data has been widely validated as an effective practice for implement- ing chat language models like ChatGPT. Scal- ing the diversity and quality of such data, al- though straightforward, stands great chance of leading to improved performance. This pa- per aims to push the upper bound of open- source models further. We first provide systematically designed, diverse, informative, large-scale dataset of instructional conversa- tions, UltraChat, which does not involve hu- man queries. Our objective is to capture the breadth of interactions between human user and an Al assistant and employs compre- hensive framework to generate multi-turn con- versation iteratively. UltraChat contains 1.5 million high-quality multi-turn dialogues and covers wide range topies and instructions. Our statistical analysis of UltraChat reveals its superiority in various key metrics, includ- ing scale, average length, diversity, coherence, etc., solidifying its position source dataset. Building upon UltraChat, we fine-tune L.aMMA model create a powerful conversational model, MltraLM. Our evalua- tions indicate that UltraLM consistently out- performs other open-source models, including WizardLM and Vicuna, the previously recog- nized state-of state-of-the-ar aht open-source models.
Dolly-v2 (Conover et al., 2023)
4.04 2.34
000
MPT-Chat (Mosaic, 2023)
6.67 2.88
OpenAssistant (Köpf et al., 2023)
7.65 2.15
Alpaca (Taori et al., 2023)
8.04 2.05
Koala (Geng el al., 2023)
8.23 + 1.99
Baize (Xu al., 2023b)
8.50 1.34
Vicuna (Chiang al., 2023)
8.78 1.55
Wizard-LM (Xu et al., 2023a)
8.95 1.44
UltraLM (ours)
9.00 1.33
Table Average scores (I-10) across different open- source models and UltraLM. The evaluation is con- ducted on our curated evaluation set with GPT-4. Evalu- ation prompts can found Appendix C.
ChatGPT has spurred surge in the adoption and implementation of general chat language models. In addition to competing models developed by large corporations such as Bard' and Claude?, the open-source community is actively engaged training similar models, aiming to democratize access to Al technology. Notable examples in this regard include Alpaca (Taori et al., 2023), Vi- cuna (Chiang et al., 2023), Koala (Geng et al., 2023), Baize (Xu et al., 2023b), and Belle (Ji et al., 2023), etc., demonstrating promising perfor- mance. Experimental evidence strongly suggests that chat language models can be effectively trained through instruction fine-tuning (Wei et al., 2021; Sanh et al., 2021), and they also indicate that many data-efficient (Zhou et al., 2023) or computing- efficient (Hu et al., 2021; Ding et al., 2023) meth- ods can be applied. This paper, in another way, fo- cuses more the "final one mile" of chat language models, as evidence shows that the journey from 0 to 60 is easy, whereas progressing from 60 to 100 becomes exceedingly challenging. For instance, researchers have shown that by utilizing a small,
leading open-
1 Introduction
Large language models (Bommasani et al., 2021; Han et al., 2021; Chowdhery et al., 2022) (LLMs) have demonstrated exceptional generalization capability on variety of language-related tasks. Notably, ChatGPT (OpenAI, 2022), optimized version of GPT-3 (Brown al., 2020) for conver- sation, along with GPT-4 (OpenAI, 2023), takes the user experience to another level via excelling in comprehending and generating responses a natural and interactive manner. The introduction of
cqual contributions Corresponding authors
Pttpps: : / http://ww/ttt/////// www.:..nnhrrpii..oo compp//cccco/////ooooooooooo/ wtw./oo//////////////////////mms indxx/ cll

---

thoughtfully curated set of instructions, it is possi- ble train a model with satisfactory instruction- following capabilities. However, these approaches have yet to produce models that surpass the perfor- mance of Vicuna, the current leading open-source model, let alone outperform ChatGPT and GPT-4.
hibits strong ability in instruction understanding and generalizes well to unseen instructions (Sanh et al., 2021; Ouyang et al., 2022). Later, Longpre et al. (2023) show the benefits of scaling the num- ber of tasks in out-of-distribution generalization. Wei et al. (2021) also conclude that the success of instruction tuning depends on the quality of the dataset and the design of prompts. To further reg- ulate the tuned model's behavior, Ouyang al. (2022); Schulman al. (2017) propose to employ reinforcement learning to align model behaviors with human preferences. This technique combined with instruction tuning can further boost the model performance and has been successfully applied to LLMs such as ChatGPT.
This paper believes that the most straightforward way, that is, the quality and diversity of training data, play a vital role in further improving the per- formance of chat language models. In other words, leveraging higher quality and more diverse data can yield better outcomes. To this end, we present UltraChat, million-scale multi-turn instructional conversation data, to facilitate the construction of more powerful chat language models. UltraChat is carefully designed capture the breadth of in- teractions that a human might have with an Al assistant. Specifically, we do not use specific tasks like question-answering or summarization to con- struct the data, but curate three sectors: Questions about the World, Creation and Writing, and Assis- tance on Existing Materials. Then we employ meta- information, in-context expansion, and iterative prompting scale up the number of instructions. To construct informative and realistic multi-turn conversations, two separate ChatGPT Turbo APIs are adopted in the conversation generation, where one plays the role of the user to generate queries, and the other generates the response. We instruct the user model with carefully designed prompts to mimic human user behavior and call the two APIs iteratively.
Data Augmentation with LLMs. Collecting large-scale human-annotated instructions and their responses is time-consuming and labor-intensive. Alternatively, a more cost-effective and feasible approach to gathering top-notch data involves sampling from LLMs that have been well-tuned, e.g.. ChatGPT and GPT-3.5. Recently, there is surge of interest in distilling these powerful LLMs for data augmentation. For instance, us- ing the technique of Self-Instruce (Wang al., 2022). Alpaca (Taori et al.. 2023) generate 52k high-quality instruction--essonse pairs based on seed tasks by "distilling" Text-Davinci-003. The trained model performs almost on par with Text- Davinci-003. The success of Alpaca boosts numer- ous later efforts data augmentation with LLMs, such code-alpaca (Chaudhary, 2023), alpaca- cot (Si et al., 2023), GPT4ALL (Anand al., 2023), ShareGPT (Domeccleston, 2023), Dolly- v2 (Conover et al., 2023), BELLE (Ji et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), Baize (Xu et al., 2023b), etc. It is shown that increasing the scale of data could constantly improve the model performance. Besides, prompt engineering also affects data quality. CAMEL (Li et al., 2023a) design multi-agent role-play envi- ronment for LLMs simulate real human conver- sations.
We fine-tune a LLaMA-13B model on Ultra- Chat to produce UltraLM and compare the model to wide range of baselines, especially the open- source ones. The evaluation shows that our model could consistently outperform other models. As reported in Table 1, UltraLM achieves the highest performance scores that are independently assessed by GPT-4. Further evaluation results on challeng- ing benchmarks and preference study with GPT-4 on various evaluation sets also show that UltraLM could surpass all other open-source models.
2 Related Work
Data Construction
Instruction Tuning. Recent works demonstrate LLMs' powerful capabilities in following human instructions. Wei et al. (2021) pioneered fine- tune T5 (Raffel et al., 2020) on 60 NL. datasets ver- balized with natural language instruction templates, i.e., instruction tuning. The fine-tuned model ex-
LLMs are believed to be better annotators than human-being in many scenarios (Gilardi al., 2023). However, employing LLMs such Chat- GPT directly for generating multi-turn conversa- tions may yield satisfactory but less informative results, as it cannot enjoy the benefit of reinforce-

---

Sector Questions about the World
Human
User Model
Sub Topics
Questions about Conceptions
Meta Topics
Query/lnstruct
Model
Detailed Questions
Wikidata W
Representative Entities
Meta Questions
Search Engine
Associated Questions
3-7 rounds of generation
Sector II: Creation and Writing
Human
Material Generation
Material Types
Detailed Instructions
Response
Instructions
Model
Model
Sector IlI: Assistance on Materials
C4
Materials
Questions Instructions
R Post-processing
Sector
Figure 1: Construction process of UltraChat. The three sectors of data are derived from different meta information..
ment learning with human feedback (RLHF) in the alignment process. Table 12 in Appendix A shows a comparison of directly generated multi-turn dia- logue and a case in UltraChat with the same open- ing line. Two key points can be derived to ensure the quality of the data: (1) An opening line de- termines the topic of the dialogue. Opening lines should be highly diverse and encompass any task that human
the of human-Al interaction, rely on Al assistants provide quick and accurate answers to their questions.
often
Our approach to gathering data for this sector involves two perspectives: one centered around top- ics and concepts, and the other around real-world entities. Initially, request ChatGPT to generate 30 comprehensive topics that encompass various aspects of our daily lives, shown Table 2. Subsequently, we delve deeper into each topic by generating 30 to 50 subtopics or related concepts. Finally, we generate 10 different questions for each subtopic or concept and additionally request Chat- GPT to generate 10 more questions based on each original question. The other source of data comes from real-world objects, which are derived from Wikidata entities. These entities are further refined by considering their frequencies in Wikipedia* ar- ticles, specifically focusing on the 10,000 most frequently occurring entities. For each entity, we create 5 meta-questions, followed by 10 more spe- cific questions and 20 extended questions. The extended questions aim to maintain some similarity to the original question while exploring distinct ob- jects or topics. To create a dialogue, we filter and sample approximately 500,000 questions open- ing lines. During the construction of each dialogue, we provide the user model with carefully crafted prompts that explicitly ask the model to respond
may request chat model to perform. (2) user determines the plot of the dia- logue, and the output should be tailored the cur- rent topic with diverse language styles and requests. Therefore, unlike traditional task-specific datasets, to construct comprehensive open- domain instructional chat dataset, the design of data collection schema is crucial capturing the breadth of interactions and ensuring data quality. UltraCaat aims to cover a tremendous range of conversation data with a carefully designed tripar- tite schema: Questions about the World, Creation and Writing, and Assistance on Existing Materials. While the core of ensuring data diversity mainly depends opening line diversity, will first in- troduce the idea behind the sector design and then focus on specific measures obtain a diverse set of opening lines and how to prompt the user properly.
3.1 Questions about the World
The first sector focuses on querying existing infor- mation in the world, including concepts, objects, and entities that exist in the real world. This is at
httpps:/wwaw wikidata.org *https: //www. wi wikipedia...... org/

---

Technology
Hcalth and wellness
Travel and adventure
Food and drink
Art and culture
Science and innovation
Fashion and style
Relationships and dating
Sports and fitness
Nature and the environment
Music and entertainment
Politics and current events
Education and learning
Money and finance
Work and career
Philosophy and ethics
History and nostalgia
Social media and communication
Creativity inspiration
Personal growth and development
Spirituality and faith
Pop culture and trends
Beauty and self-care
Family and parenting
Entrepremeurship and business
Litcrature and writing
Gaming and technology
Mindfulncss and meditation
02 Diversity and inclusion
Travel and culture exchange
Table 2: meta-concepts used to generate the first sector of UltraChat data.
Articles and Blog Posts
Joh Application Material
Stories
Legal Documents and Contracts
Poems
Educational Content
Screenplays
Scripts for Language Learning
Technical Documents and Reports
Marketing Materials
Social Media Posts
Personal Essays
Emails
Scientific Papers and Summaries
Speeches and Presentations
Recipes and Cooking Instructions
News Articles
7
Song Lyrics
Product Descriptions and Reviews
Programs and Code
Table 3: 20 types of text materials used for sector 2 and 3 UltraChat construction.
concisely and meaningfully, taking into account the context of the ongoing dialogue history.
rization, and question question-answering, -nswering, etc. Modifying existing materials is crucial aspect of human-Al interaction, it allows the AT assistant actively engage with the user's input, transforming it in various ways as instructed by the user.
3.2 Creation and Writing
The second part is concerned with the creation of new information with human-input conditions, ranging from writing emails to crafting stories and plays. This process reflects the Al's capacity to engage in original content generation alongside users and demonstrates the role of AI assistants as collaborative partners in a creative environment.
We begin by gathering text pieces from the C4 corpus Each piece within the C4 corpus is associ- ated with source URI. To ensure a diverse range of text content and styles, we adopt the 20 material types outlined in the previous section and manu- ally curate keywords for each type. Additionally, we classify the text in the corpus by matching the keywords in the corresponding URL. In total, we collect 100,000 text picces from the C4 corpus, and for each piece, we prompt ChatGPT to generate five distinct instructions. We use a manually de- signed template to combine text and instructions, as depicted in Figure 4 in the appendix. Ultimately, the concatenated of 500,000 pieces serves the opening lines for the generated dialogues.
We first project all creations as text materials, and further categorize them into 20 different types in Table 3. Then ChatGPT model is employed produce diverse range of instructions for each type of writing, approximately 80% of which are further refined by ChatGPT model to generate more detailed instructions. These instructions serve as opening lines for dialogue generation. Throughout the generation process, the user prompt constantly reinforces the primary objective of the conversa- tion, which is to generate and refine piece of writing. This serves to ensure that the behavior of the user model remains focused and aligned with the intended purpose.
3,4 User Simulation and Refinement
Maintaining the desired behavior of the user model is crucial for achieving successful automatic dia- logue generation. It has been observed that when the user model is solely provided with the current dialogue history, it tends to assume the role of an AI assistant. This "role confounding" situation
3.3 Assistance on Existing Materials
The third sector mainly addresses the modification of existing information, encompassing various tasks including rewriting, translation, summa-
Phttps://commnnraav..

---

Avg. #Turns
Avg. Dialog Length (by token)
Avg. Utt. Length (by token)
Lexical Diversity (1)
Topic Diversity (J)
User
Diillogue
Coherence
Simulation
Self-Instruct
82,439
69.8
29.2
24.9
0.733
No
Stanford Alpaca
52,002
91.1
64.5
42,8
0.727
1,486,869
3.6
38.6
0.797
G
61,002
179.6
142.9
48.9
0.721
No
BELLE
1,436,679
102.3
63.3
35,9
0.771
No
Baize
210,311
3.1
293.9
52.8
67.1
0.751
9.06
Yes
GPT4ALL
711,126
597.2
318.9
62.7
0.692
UltraChat
1.468,352
3.8
1467.4
309.3
74.3
0.702
9.06
Yes
Table Statistics of existing instruction datasets. Lexical diversity calculated by averaging the MTLD score (Mc- Carthy and Jarvis, 2010) over each utterance with LexicalRichness" 10000 samples are randomly drawn from each dataset for topic diversity and coherence measurement. Topic diversity is measured by averaging the cosine distance between each pair of data with OpenAI embedding API. Coherence is scored by ChatGPT scale of 1-10.
can significantly deteriorate the coherence of the multi-turn conversation. To address this, in addi- tion to presenting the dialogue history, we include prompts explicitly instructing the model to adopt various user personalities. In Sector 2, prompt is employed to remind the model of the primary purpose of the dialogue, thereby promoting a more natural conversation flow. Once the data genera- tion process is complete, a further filtration step is performed to ensure overall data quality. We also exclude excessively polite statements to enhance the realism of user responses.
4.2 Human Assessment
Setup. To better evaluate the constructed data quality, we also conduct human assessment for UltraChat. Due to the difficulty of evaluation of multi-turn dialogue and the resulting formidable cost, we sample 500 representative dialogues for human evaluation, among which 300 are from Ul- traChat sector 1, 100 from sector 2 and sector 3 respectively. For each round of conversation, we ask the annotators to score the assistant's response on Helpfulness, Honesty, and Harmlessness (3H) principles (Askell et al., 2021). We also devise Coherence and Consistency criteria for the overall multi-turn dialogue quality evaluation. Coherence evaluates whether the dialogue flows logically and coherently, for which the annotators evaluate both the user's response and the assistant's response. Consistency means the assistant's responses do not contradict each other within the same dialogue, For example, it is inconsistent if the assistant asserts one specific event occurred in 1911 in the first round of conversation but mentions it as 1901 event in the next round. Each metric is scored with 0, 0.5 or 1, where higher score means better quality. Therefore, for a K-round dialogue, we have 3K +2 metric annotations.
4 Data Analysis
4.1 Statistical Analysis
We conduct statistical analysis of UltraChat and several other instruction datasets, as shown in Table 4. UltraChat stands out in terms of its scale, being one of the largest publicly available datasets. Moreover, it exhibits the highest average number of turns and the longest average length per instance of data. While SODA (Kim et al., 2023) also has many rounds, it is primarily composed of conceptual banter rather than instructional content. Additionally, the average number of tokens per dialogue in SODA is 231.8. whereas UltraChat boasts a remarkable 1467.4 /4 tokens. To evaluate di- versity, we measure both lexical diversity and topic diversity. UltraChat outperforms previous datasets in terms of lexical diversity. However, in terms of topic diversity, UltraChat falls slightly short com- pared to GPT4ALL (Anand et al., 2023) but still surpasses other datasets significantly. This may be attributed to the regularized embeddings resulting from large number of tokens in each dialogue. We also conduct coherence evaluation with ChatGPT for multi-turn datasets. Notably, UltraChat and Baize data rank the highest in terms of coherence.
Each dialogue is annotated indepen- dently by two well-trained annotators, and the score is averaged across two annotators. Meanwhile, due to the difficulty in identifying the hallucination problem, we allow the annotators to skip the di- alogues that require expert knowledge whose validity is hard check. Altogether, we collect 14560 valid annotations in terms of metrics for both single-round and multi-round, and the Co- hen's 's kappa coefficient is 0.358. The average time to annotate one dialogue is 10 minutes.
Annotation.

---

Results. As shown in Table 5, the dataset scores high on all metries, showing the effectiveness of the construction process. It worth noting that the dataset is almost free from harmful content like hate speech and discrimination. Furthermore, we observe two trade-offs between data quality. The first is between helpfulness and honesty. While the honesty score is pretty high, helpfuiness is compro- mised. It is because some user queries out of the LLM ability scope (e.g., ask the LLM to com- pose a song). Under such circumstances, the LLM refuses to answer the question with an explanation of incapability, which is reasonable and expected but less helpful. The phenomenon is particularly prominent in part 3, as the simulated user often asks for information not existent in the text mate- rial. The second trade-off is between control and dialogue naturalness, While sector and sector data are constructed with more guidance and con- trol when prompting for user simulation, they have clearly lower quality in coherence and consistency than sector 1.
knowledge scope and the multiple abilities of the language model. To better demonstrate the chat ability of language models, an automatic response quality evaluation is performed to showcase the model's proficiency in delivering accurate and informative content during chat interactions. Note that UltraLM is solely trained on UltraChat dataset without further finetuning on task specific datasets.
5.1 Experimental Setup
We mainly compare with other open- source instruction-tunee language models based on LLaMA (Touvron et al., 2023) and Pythia (Bi- derman et al., 2023) backbone model. The main baseline models include Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), Dolly (Conover et al., 2023), OpenAssis- tant (Köpf et al., 2023), and WizardLM (Xu et al., 2023a). The parameter size of the baselines ranges from 7B to 13B, which is comparable to UltraLM. Our evaluation also includes other chat language models like ChatGPT (OpenAI, 2022), MPT (Mo- saie, 2023), and Baize (Xu et al., 2023b). A de- tailed description of the main baselines can be found in Appendix A.I.
Baselines.
Data
Helpful
Honest
Harmless
Coherent
Consistent
Sector
0
1.000
0.996
0.995
Sector
0.978
0,986
L.000
0.982
0,977
Sector
0,893
0,983
1.000
0.964
0
Overall
0.960
0,992
1,000
0.987
0,988
Datasets. For benchmark evaluation, we choose four datasets: ARC-Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truth- fulQA (Lin al., 2021), evaluating commonsense knowledge, professional knowledge and complex reasoning and understanding abilities. Each bench- mark constructed as multiple choice questions and therefore metrics are readily computable. The four datasets prove to be challenging even for the best-performing language models like ChatGPT. For response quality evaluation, we use 3 datasets. We first create evaluation set by our- selves. The curated set encompasses the Vicuna benchmark as well as an additional 300 questions and instructions generated by GPT-4. The ques- tions/instructions covered a wide range of topics, including commonsense, world knowledge, pro- fessional knowledge (specifically physics and biol- ogy), mathematics, response generation, and writ- ing tasks on different levels of difficulty. Apart from the curated set, also adopt AlpacuE- val (Li et al., 2023b), widely acknowledged
Table 5: Human assessment results on 500 dialogues sampled from UltraChat.
5 Experiments
We developed UltraLM, an enhanced variant of the LLaMA-13B (Touvron et al., 2023) model, by training it on the UltraChat dataset. To improve the model's comprehension of dialogue context, we break down each dialogue into smaller sequences, limiting them to maximum length of 2048 tokens. During the training process, we only calculate the loss for the model's 's responses. This approach en- sured that the model had access to the relevant in- formation from earlier parts of the conversation, en- abling a more comprehensive understanding of the ongoing dialogue. By incorporating the preceding context, UltraLM was equipped to generate more contextually appropriate and coherent responses. We use standard cross-entropy loss to finetune the model. The model is trained with 128 A100 GPUs and the total batch size is 512.
The evaluation of the trained model is conducted in two folds. We first evaluate UltraLM on traditional benchmark datasets to delineate the
"Some baselines used in our experiments are continuously updated. All the results in this section are from the latest versions of these baselines before June 23, 2023

---

ARC-Challenge Acc. norm.
HellaSwag Acc. norm.
MMLU
TruthfulQA
Overall
Model
Acc.
Acc.
Weighted
Unweighted
mel
mc2
Average
Dolly-12B
38.23
42.24
54.59
72.6
31.52
31.70
20.69
34.06
45.15
OpenAssistant-122 12B
41.38
45.90
52.51
70.04
29.77
30.29
24.60
39.29
46.38
MPT-7B
43.00
46.67
57.13
75.50
37.76
38.33
27.17
40.16
50.17
Alpaca-7B
49.74
52.65
58.05
76.91
42.47
42.90
25.83
39.55
53,00
LLaMA-13B
53.16
56.40
60.64
80.87
46.74
25.83
39.90
55.98
Baize-13B
55,55
57.94
59.96
80.36
48.13
49.03
32.93
47.43
58.69
Koala-133 13B
49.83
52,90
57.60
77.54
46.75
48.01
34.64
50.09
57.14
Vicuna-13 13B
51.71
52,90
60.03
80.12
50.15
50.45
35.74
51.82
58.83
Wizardl .M-13B
55.12
57.08
60.93
80.91
51.69
52,25
35.37
50.53
60.19
LLaMA-65B
59.22
63.31
66.40
86.05
62.29
62.97
27.91
42.55
63.72
UltraLM-
57.25
59.22
61.32
81.49
50.45
51.10
36.72
52.00
60.95
Table The evaluation results on 4 challenging benchmark datasets. All evaluation and metric calculations follow EleutherAl's Im-evaluation-harees (Gao al., 2021). Both weighted and unweighted mean accuracy reported for MMLU as there are 57 tasks. The overall average metric is obtained by averaging the second column data for each benchmark dataset. More details about metric calculation can be found Appendix A.3.
open-source evaluation set and leaderboard specif- ically designed for evaluating LLMs. The leader- board is created based the win-rate against Text-Davinei-003 automatically evaluated by GPT- 4. To further compare with the state-of-the-art model WizardLM (Xu al., 2023a), comparison result obtained with GPT-4 on the released Evol- Instruct (Xu et al., 2023a) test set is also reported. Further benchmark dataset and implementation de- tails can be found in Appendix A.2 and A.3.
of a straightforward text completion, and therefore benefits little from instruction tuning.
5.3 Response Quality Evaluation
Response Comparison. For our curated evalu- ation set, conduct pairwise evaluation between UltraLM and each baseline model with GPT-4. Our evaluation prompt is designed prioritize cor- rectness over other factors such as informativeness. To mitigate the influence of presentation order of responses, we randomly determine the order of the responses for each question. Finally, we count the number of Win/Tie/Lose times against each baseline model, and the result is presented in Figure 2. UltraLM demonstrates superior performance compared to every open-source model, exhibiting an impressive winning rate of up to 98%. worth noting that UltraLM also outperforms Vicuna with 9% higher winning rate.
5.2 Benehmark Evaluation
As shown in Table 6, with pure instruction-tuning on the UltraChat dataset, UltraLM significantly improves over I.L.MA-333 and achieves the best overall performance across four benchmarks. is worth noting that UlrrLMM overtakes the cur- rent state-of-the-art model by nearly 2% on ARC- Challenge and TruthfulQA. It shows that UltraLM M is equipped with both broad and profound compre- hension of the world and commonsense knowledge. The improvement could be attributed to the system- atic and comprehensive data construction process of UltraChat sector 1, which effectively extends and deepens the discussion about world knowledge in automatic conversation generation. Meanwhile, the comparative inferiority in MMLU hints on the lack of professional knowledge in specific fields. It suggests the need for more advanced data gener- ation techniques build a specialized expert lan- guage model. As for HellaSwag, we notice that all models have only marginal improvement compared to LLaMA-13B 1.B. is probably because HellaSwug is formatted as an in-text completion task instead
Independent Scoring. Given the instability of pairwise comparison, we also conduct independent quality scoring with GPT-4, as presented in Table 7. Notably, our model demonstrates superior performance compared to all the open-source coun- terparts by a significant margin in terms of overall scores. This breakdown also provides insights into the performance of each model on specific types of questions and instructions. Generally, all models perform better on simpler questions pertaining to commonsense knowledge and general world understanding. However, more complex tasks that involve reasoning and creative writing proves to be challenging for most models. Interestingly,

---

Win
Tie
Lose
Win
(%)
Vicuns-183
115
30.2 30.7
WizardLM-
194
102
85
50.9 /6.8 /2.33
Koale- 13B
220
84
76
57.9 22.1 20.0
Baize- 13B
280
OpenAssistant-125 12B
293
42
76.9 11.0 12.1
Alpaca-7B
321
34
26
MPT-7B
340
35
9.20
Dolly-12B
98.4 0.,0 1,10
Figure 2: Response comparison of UltraLM with other baselines the curated evaluation evaluated by GPT-4.
Vicuna Set
Commonsense
World Knowledge Difficult
Prufessional Knowledgy
Ability
Writing
Overall
Easy
Moderate
Easy
Physics
Biology
Math
Reasoning
Dolly-12B
4.75
3.50
3.93
3.10
4.13
4.87
5.47
2.70
2.03
4,51
4.04
MPT-7B
7.25
5.57
8.20
5.53
5.87
7.83
8.40
5.97
3.97
7.25
6.67
MMA-138
6.85
8.43
8.43
8.57
8.50
7.90
8.40
6.97
6.73
6.79
7.49
OpenAssistant-I1n
7.88
8.13
7.80
9.13
7.50
6.57
7.75
Alpaca-7B
7.58
9.17
9.30
8.73
6.70
8.05
Keala-13
8,93
8:40
Baize- 13B
8.40
9.03
9.10
9.03
8.93
8.83
8.80
7.43
8,30
8.10
8.50
Vicuna- 13B
8.48
9.67
9.50
9.37
9.30
9.23
9.33
8.07
6.90
8.76
8.78
WizardL M-13B
8.55
9.70
9.30
9.57
9.50
9.27
9.53
8.27
8.20
8.83
8.95
CharGPT
9.15
9.67
9.60
9.80
9.60
9.17
9.73
9.33
9.13
8.95
9.31
M--B
8.98
9.70
9.50
9.47
9.40
9.27
9.87
8.77
6.80
8.90
9.00
Table 7: The independent overall scoring and segment scoring of each model on the curated evaluation set, a scale 1 to 10. Bold indicates the best score and underlined indicates the second best.
Alpaca, despite having only billion parameters, performs comparatively well with larger models on questions related to commonsense and world knowledge. Meanwhile, Dolly and OpenAssistant, which are based on Pythia (Biderman et al., 2023), display inferior performance compared to models based on LLaMA of similar or even smaller sizes, This observation highlights the significance of the underlying backbone language model.
AlpacaEval. As shown in Table UltraLM out- performs existing models in terms of win rate on the current AlpacaEval leaderboard and ranks 4th just below ChatGPT. This observation further testi- fies the response quality of UltraLM and is in line with results on our curated dataset. Furthermore, significant gap is evident between UltraLM and ChatGPT performance, highlighting the substantial effort needed for open-source models to match the capabilities of ChatGPT.
Model
Win Rate (%)
Standard Error
Evol-Instruct Evaluation. Figure shows the automatic comparison results against WizardLM- 13B on Evol-Instruct test set. UlraLM overtakes WizardLM on most types of questions, with up to 29% increase in scores. is important to acknowl- edge that Wizard is trained on the Evol- Instruct training set, making UltraLM's success on the test set noteworthy achievement. More- over, the questions observed with the largest im- provement are mainly complex problems that often require synthesized abilities. It demonstrates the success of UltraChat design schema, which can comprehensively boost the versatile capabilities language models. The inferiority math prob-
GPT-4
95.28
0.72
Claude
88.39
1.11
ChatGPT
86.09
1.21
UltralM-HH
76.09
1.50
WizardLM-133
75.31
1.51
Guanaco-65B
71.80
1.59
Vicuna-133
70.43
1.61
Oasst Oasst-RL.F-335
RLHF-33B
66.52
1.66
Text Davinci
50.00
0.00
Falcon-40B-instruet
45.71
1.75
Alpaca-7B
26.46
1.54
Table 8: Win rates against Text- Datt-Davinc-000 Al- pacaEval leaderboard. GPT-4 is used for evaluation following the official implementation (Li et al., 2023b).

---

UbraM/ WW2302
Average: 102.5%
100%
80%
60%
40%
Teseily
Acestmm Heriang
Music
Polippay
Cods Debug
Teotnollogi
Beelnny
Enwrtaimmtnt
Emua
Phinesppty
Lieriture
Chamitry
Comples Fonnet
Physcc
History
Art
Westing
Short
Law
Medicnn
Ecoonyy
Mustinguni
Romsonng
Computer Solenve
Connon Canron-Serm Serre
Cone Gententin
Math
Counte
Figure 3: Comparison between UltraLM and Wizard on Evol-Instruct test set. The scores are obtained by pairwise scoring with GPT-4, and WizardLM scores considered 100%.
lems is not surprising though, as no mathematical problems are intentionally generated in UltraChat.
general chat models. UltraChat encapsulates a broad range of human-AT interactions, further de- veloping series of dialogues across various topics and instructions. Statistically, UltraChat shows an impressive presence in critical metrics such scale, average length, diversity, and consistency, further establishing itself as a leading open-source dataset. We leverage UltraChat to fine-tune the LLaMA model, leading to the development of the robust conversational model, UltraLM. Evaluation across multiple benchmarks reveals that UltraLM surpasses previous open-source models like Wiz- ardLM, Vieuna, Alpaca, and Koala in performance. We eagerly await the innovative research and devel- opment that will be catalyzed by our contributions in the field of Al conversational models.
Impact of System Prompts. Using system prompts to adjust the role and response style of LLMs is a common practice. Although system prompts are not embedded in UltraLM's training data like others do (Chiang al., 2023), they still appear have substantial influence on the re- sponse style of the generated output. Specifically, when the model is prompted to provide a "helpful and detailed" response, tends to generate more pertinent details While such prompts may not im- prove the accuracy of an answer, they do raise over- all quality with more informative response. To illus- trate this effect, conduct ablation response comparison UltraLM. Table 9 reveals signifi- cant improvements in response quality brought by system prompts across tasks. We also inspect the detailed evaluation for deterministic questions (commonsense and world knowledge) and find that UltraLM without system prompt only incorrectly answers one question. Thus, the main benefit of the system prompt lies in enhanced informativeness rather than higher correctness.
Limitations
Evaluating the response quality of large language models is an extremely challenging task, and any assessments may have biases. For comprehen- sive evaluation, we compared UltraLM's perfor- mance with other basclines across various bench- marks, utilizing GPT-4 to assess the response qual- ity. Nevertheless, the need for additional, diverse evaluations remains to facilitate more thorough understanding of our model's behavior and perfor- mance. Despite demonstrating promising results in experimental settings, UltraLM not immunc to the common pitfalls of large language models, including hallucination issues and potential ethi- cal concerns associated with misuse. Additionally, the energy-intensive nature of UltraLM's training process represents a limitation, particularly when compared to models employing more efficient tech- niques such parameter- efficcient fine-tuning, In terms of UltraChat, it currently only contains En- glish and there no explicit methodologies incor- porated for generating data to enhance the model's reasoning capabilities, representing another area for potential improvement.
Data
Win (%)
Tie (%)
Lose (%)
Vicuna Set
36.3
35.0
28.8
Commonsense
58.6
31.0
10.3
World Knowledge
56.9
34.5
8.6
Professional Knowledge
57.8
31.1
11.1
Math Ability
46.7
13.3
40.0
Reasoning Ability
46.7
33.3
20.0
Writing Overall
46.3
35.0
18.8
49.1
32.0
18.9
Table 9: Win rate against UltraLM without system prompt on our curated dataset.
6 Conclusion
drawing to a close, our work introduces Ultra- Chat, structured design of multi-turn instructional conversation data primed to foster the growth of

---

Ethics Statement
reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
While the advancements of the UltraChat dataset and UltraLM are commendable, they still face ethi- cal challenges that exist in the area of LLMs. For the sake of privacy protection, we do not use any online or even human queries/instructions to con- struct UltraChat but develop a framework to build scalable, diverse instructional data. Although ex- tensive filtering operations are conducted, biased statements may still exist within the dataset.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan. Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint wrXiv:230.003373
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv: 2ea:2210000555
UltraLM, as the paper described, will be of the most potent open-source chat language models. With great power comes increased responsibility and potential for misuse. There exists a substantial risk for the technology to be weaponized for spread- ing misinformation, propaganda, or even creating "deepfake" text that could mislead or manipulate public discourse. This necessitates the establish- ment of robust policies and comprehensive research prevent misuse and deter malicious applications of the technology. In this regard, the model's po- tential applications should be carefully evaluated for any possible negative consequences before de- ployment.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of NeurlPS.
Sahil Chaudhary. 2023. Code alpaca: An instruction- following llama model for code generation.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Acknowledgements
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 220.0220...
This work is supported by the National Key R&D Program of China (No. 0022ZD01191011 Na- tional Natural Science Foundation of China (No. 62236004), the Young Elite Scientists Sponsorship Program by CAST, and Institute Guo Qiang at Ts- inghua University.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try are, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05557
References
Mike Conover, Matt Hayes, Matt Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Ali Ghodsi, Patrick Wendell, and Patrick Zaharia. 2023. Hello dolly: De- mocratizing the magic of chatgpt with open models.
Yuvanesh Anand, Zach Nussbaum, Brandon Duder- stadt, Benjamin Schmidt, and Andriy Mulyar. 2023. Gpt4all: Training an assistant-style chatbot with large scale data distillation from -.utbbb......
Ning Ding. Yujia Qin, Guang Yang. Fuchao Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan. Weize Chen, al. 2023. Parameter- efficient fine-tuning of large-scale pre-traines lan- guage models. Nature Machine Intelligence, pages 1-16.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson EI- hage, Zac Hatfield-Dodds, Danny Hernandez, Jack- son Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam Mc- Candlish, Chris Olah, and Jared Kaplan. 2021. general language assistant as laboratory for align- ment.
Domeccleston. 2023, Sharegpt share your wildest chatgpt conversations with one click. Retrieved 23 May 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite,

---

Ben Wang, Kevin Wang, and Andy Zou. 2021. framework for few-shot language model evaluation.
Shayne Longpre. Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprini arXiv:2301. 136888...11....
Xinyang Geng, Arna Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
Philip M. McCarthy and Scott Jarvis. 2010. Mtld, vocd- d. and hd-d: A validation study of sophisticated ap- proaches to lexical diversity assessment. Behavior Research Methods, 42:381-392.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text- annotation tasks. arXiv preprint arXiv:2303.15056
Mosaic. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llms.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang. Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Ji Tang, Ji-R Ro-R Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. Al Open.
OpenAL. 2023. Gpt-4 technical report. arXiv.
TB OpenAL. 2022. Chatgpt: Optimizing language mod- els for dialogue, OpenAl.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika. Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. Proceedings of ICLR.
Edward Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. In Proceedings of ICLR.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to tt-xxt transformer. Journal of Machine Learning Research, 21(140):1-67.
Yunjie Ji. Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang. Baochang Ma, and Xiangang Li. 2023. Exploring the impact of instruction data scaling on large language models: An empirical study real-world use cases. arXiv preprint arXiv:2303. 14742.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafcai. Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero- shot task generalization. In Proceedings of ICLR.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stan- ley. Richard Nagyfi, et al. 2023. Openassistant conversations- conversations-demnnraaziiin large language model alignment. Xiviv preprint arxiv:230..07327.
Qingyi Si, Tong Wang, Naibin Gu, Rui Liu, and Zheng Lin. 2023. Alpaca-cot: An instruction fine-tuning platform with instruction data collection and unified large Inguage models interface.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a. Camel: Communicative agents for "mind" exploration of large scale language model society.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B Hashimoto. 2023. Alpaca: A strong, replicable fnstluction-folloiing model. Stanford Center for Research Foundation Models.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of fustluctionfollooiing models.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Aune Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07954

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra. Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng. Chongyang Tao, and Daxin Jiang. 2023a. WizardIm: Empowering large lan- guage models to follow complex instructions.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley, 2023b. Baize: An open-source chat model with parameter-efficien tuning on self-chat data. arXiv preprint arXiv:2304.01
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can machine really finish your sentence? Proceedings ACL, pages
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment.

---

A Experimental Details
A.2 Evaluation Dataset
Table 11 presents the basic information of the eval- uation datasets we use. Below give detailed description of each dataset.
A.1 Baselines
We introduce the main open source baseline mod- els below.
The A12 Reasoning Challenge (ARC) (Clark et al., 2018) is comprised of advanced science questions and structured as multiple-choiee ques- tions. Each question is accompanied by 4 available choices and only one answer is correct. We use the challenge partition here for evaluation, which contains 1172 test examples.
Alpaca (Taori et al., 2023) is an instruction- following language model derived from the LLaMA (Touvron et al., 2023) model that has been effectively optimized on 52,000 demonstrations of instruction data. The data is generated by Self- Instruct approach with Text-Davinci-003.
Vicuna-13B (Chiang et al., 2023) is an open- sourced chat model created by fine-tuning LLaMA on user-shared conversations collected from ShareGPPS An automatic evaluation by GPT-4 demonstrates that Vicuna can yield over 90% re- sponse quality of ChatGPT. In following practices, Vicuma is widely acknowledged as the state-of-the ohe art open-source chat model. This is evident in the Chat Arena", where a total of 13,000 anonymous votes reveal that the quality score of vicuna-13B surpasses that of other open-source models.
HellaSwag (Zellers et al.. 2019) tests common- sense inference ability by evaluating how well the language model can predict the remaining part of a sentence. Each sample has 4 different text pieces as the candidate remaining part of given sentence and only one of them is plausible. The task is shown to be casy for humans but challenging for language models. We use the validation split as in Gao et al. (2021).
MMLU (Hendryeks et al., 2021) is a comprehen- sive dataset that consists of 57 different tasks cov- ering assessments of multiple types of academic knowledge and mooblem-solving ability of language models, ranging over expert fields like mathemat- ics, history, law, etc.
Koala-13B (Geng et al., 2023) another LLaMA- based model fine-tuned on selected public dia- logues. In existing open evaluations, Koala's per- formance will be slightly worse than vicuna, but it still remains a strong baseline.
TruthfulQA (Lin al., 2021) assesses how well model can identify true statements related to the real world. Its purpose to determine the risks of producing false claims or spreading misinforma- tion. The benchmark consists of questions written in various styles, covering 38 different categories, and is designed to be challenging. It includes two evaluation tasks: the multiple-choice task and the generation task. We use the multiple-choice task in the validation split as in Gao al. (2021).
Dolly-V2 (Conover et al., 2023) is based on the Pythia (Biderman et al., 2023) model, which uti- lizes 15k human-generated instruction-folloiing data. The data is organized by following Instruct- GPT (Ouyang et al., 2022), including brainstorm- ing, classificatton, closed QA, generation, informa- tion extraction, open QA, and summarization.
OpenAssistant-11B (Köpf et al., 2023) is also Pythia-based model that attempts democratize the alignment process of LLMS.. The project col- lects a conversation corpus consisting of 161,443 messages distributed across 66,497 conversation trees and trains model on these manually anno- tated data.
A AlpacaEval (Li et al., 2023b) is a hybrid evalua- tion dataset with altogether 805 instructions, which combines instructions from various existing evalua- tion sets, including self-instruct (Wang et al., 2022), Open Assistant (Köpf et al., 2023), helpful evalu- ation relcased by Anthropic (Bai al., 2022), Vi- cuna (Chiang et al., 2023), and Koala (Geng et al., 2023).
WizardLM-133 (Xu et al., 2023a) is a LLaMA- based model finetuned on Evol--nstruct dataset, which contains 250k instructions. The data are gen- erated with evolutionary strategy, where the initial instructions are rewritten by ChatGPT for several epochs to increase complexity progressively.
Evol-Instruct is dataset released in Xu et al. (2023a). It is constructed with evolutionary strat- egy by rewriting the instructions through multiple rounds to obtain instructions at different complex- ity levels. WizardLM (Xu al., 2023a) trained on the training set. We evaluate our model on the test set.
/nttoe://shareggtt.co 'nttps:// www.///ggssoooooooooooo////ooooo 0023-05-03-a100

---

Type
Example
Commonsense- Easy
What is the primary source of energy for planet?
Commonsense-Modraa
What is the phenomenon that causes the change in pitch heard when a vehicle sounding horn approaches and recedes from an observer?
World Knowledge-Lass
What is the freezing point of water in Fahrenheit?
World Knowledge-Moddrra
What is the Gödel's Incompleteness Theorem?
Physics Knowledge
How does quantum entanglement work and what are its implications for information transfer?
Biology Knowledge
What are the four main types of macromolecules found in living organisms?
Math
What is the Taylor series expansion
function e
Reasoning
You have two buckets, with red paint and with blue paint. You take cup from bucket and pour it the blue bucket. Then you take one cup from the blue bucket
and pour it back into the red bucket. Which is true: the red bucket has more blue painl, or the blue bucket has more red puint?
Writing
Write a dialogue between two photons traveling at light speed.
Table 10: Some examples of our created evaluation set.
Dataset
# Examples
Domain
(mc1), also report the ratio of likelihood sum of correct answers over all candidate answers (the me2 metric). In Table 6, Acc. norm. means the ac- curacy is computed after normalizing the likelihood with response length.
ARC-Challenge
1172
Grade-school
HellaSwag
10042
Commonsense
MMLU
14042
Academic
TruthfulQA
817
Truthfulness
AlpacaEval
805
Comprehensive
In response quality evaluation, all evaluations are done by GPT-4 Figure 5 and Figure 6 present prompts used for pairwise scoring and independent scoring
Evol-Instruct
218
Comprehensive Comprehensive
Our evaluation set
831
evaluation data. As for AlpacaEval and Inss-unst we follow the official prompt implementation (Li et al., 2023b; Xu et al., 2023a) to ensure a fair comparison.
Table 11:
Basic information on evaluation dataset.
Our evaluation set is composed of Vicuna (Chiang et al., 2023) test set and other instructions generated by GPT-4. The generated instructions are further split into different categories and difficulty levels comprehensively evaluate the model's ability. Table 10 shows example data for each type of ques- tion.
Across all experiments, UltraLM is run with tem- perature=0.5, 00__p=0.95, max new token=2000, and early stopping.
B UltraChat Details
We present a comparison between multi-turn dia- logues directly generated by ChatGPT the one in UltraChat in Table 12. Both have the same opening line. Figure 4 is the template used to concatenate text pieces from the C4 corpus and instructions gen- erated by ChatGPT for data in UltraChat sector
A.3 Implementation Details
In benchmark evaluation, the multiple choice ques- tions are further transformed into log-likelihood calculation over each answer candidate deter- mine the final prediction. Specifically, each avail- able answer is appended to the question and fed into the language model. The model calculates log- its for the answer part and derives the likelihood of the answer. The most likely answer is taken as likelihood
C Prompts
In this section, we provide system prompt used for response generation by UltrLLM (Figure 7) and prompts for automatic evaluation by GPT-4 (Fig- ure and 6).
the final prediction. Moreover, normalized by answer length to mitigate the bias caused by length variation. For the first 3 bench- marks, accuracy reported for each test sample directly. For Truthful QA, since there are multi- ple correct answers, apart from standard accuracy
D UltraLM Generation Examples
In this section, we list response examples UlrrLM-133B on different instructions from our

---

Templates
(text) [text]) [m(instruction)
(text) (instruction]
(instruction ion) Answer according to: (text)
{text) Based on the passage above, (instruction]
iinstruction:: (text) Given the text: (text) ( iniistrraction
(instruction) nGennerate according to: (text)
Figure 4: Manually designed templates for concatenating existing materials and generated instructions.
Comparison Evaluation Prompt-
You are helpful, harmless and precise assistant for checking the quality of the answer.
[Question]
[The Start of Assistant I's Answer]
[The End of Assistant 1's Answer]
[The Start of Assistant 2's Answer]
[The End of Assistant 2's Answer]
We would like to request your feedback on the performance of two Al assistants in response to the user question displayed above. Please rate the quality, helpfulness, level of details, and harmless of their responses. Each assistant receives overall score scale of I to 10, where a higher indicates better overall performance. Please first output single line containing only two values indicating the scores for Assistant and 2, respectively. The two scores are separated by space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order which the responses were presented does not affect your judgment. Please avoid same scores unless they exactly in the same level.
Figure 5: Prompt for automatic comparison evaluation.
evaluation set.

---

Independent Scoring Prompt-
You are helpful, harmless and precise assistant for checking the quality of the answer.
[Question]
[The Start of the AI Assistant's Answer] [The End of the AI Assistant's Answer]
We would like to request your feedback on the performance of the Al assistant in response to the user question displayed above. Please rate the quality, helpfulness, level of details, and harmlessness of their responses. The assistant receives an overall score scale of to 10, where higher score indicates better overall performance. Please output "Score: [an integer number between and 10J"
Figure 6: Prompt for automatic independent scoring evaluation.
System Prompt-
You are now a helpful assistant, your answer should focus on the question, give reasonable, self- consistent, high-quality solutions to make users satisfied. Please try to make the answer more informative, detailed, and polite.
Figure System prompt used for generating responses automatic evaluation.
Example A: Question of World Knowledge
User: Who discovered the law of universal gravitation?
The law of universal gravitation was discovered by Sir Isaac Newton in the 17th century. He formulated the law while studying the planets' movement around the and realized that there was quantifiable force directing them towards the sun. In 1687, he published his book titled "Mathematical Principles of Natural Philosophy," where he described the law of universal gravitation detail, outlining how two objects experience an attraction towards each other due to their mass and distance between them. This discovery revolutionized our understanding of the universe and ushered in the Age of Enlightenment. Today, Newton's 's law continues to form the foundation of modern astrophysics and space exploration.
UltraLM:
Example B: Question of World Knowledge
User: Who the 16th President of the United States?
UltraLM: The 16th President of the United States was Abraham Lincoln. He assumed the presidency during a very tumultuous time in American history, the nation was torn apart by the Civil War. During his presidency, he was known for his strong leadership. his commitment to the preservation the Union, and his support for the abolition of slavery. Some of his most notable achievements include issuing the Emancipation Proclamation, which declared that all slaves in the Confederate States were free, and delivering the Gettysburg Address, an iconic speech honoring the soldiers who died in the Civil War. Lincoln is widely regarded as one of the greatest American Presidents, remembered for his unwavering leadership during challenging period in the nation's 's history.

---

Direct Generation
What are some of the health benefits and risks of drinking coffee, and how can one maximize the former and minimize the latter?
There are several potential health benefits associated with drinking coffee, such as increased alertness, improved mood, and a reduced risk of certain discases, including type 2 diabetes and liver cancer. However, there are also some potential risks, such increased anxiety, disrupted sleep, and negative effects on the digestive system. To maximize the benefits and minimize the risks, it's important consume coffee in moderation and pay attention how affects your body.
How much coffee is considered moderate?
The definition of moderate coffee consumption can vary depending on the individual and other factors such as age, gender, and health status. However, in general, moderate coffee consumption is considered to be around 3-5 cups per day, or up to 400 milligrams of caffeine per day. It's important to note that some people may be more sensitive to caffeine than others, so it's important to pay attention to how your body reacts and adjust your consumption accordingly.
UltraChat
What are some of the health benefits and risks of drinking coffee, and how can one maximize the former and minimize the latter?
There are several health benefits and risks associated with drinking coffee. Here are some of them: Health 1e Lower risk of type 2 diabetes: Studies have shown that consuming coffee can lower the risk of developing type 2 diabctes.2. Improved cognitive function: Coffee is a stimulant that helps improve cognitive function, including alertness, concentration, and memory.3. Lower risk of depression: Studies have shown that drinking coffee can reduce the risk of developing depression.4. Lower risk of certain cancers: According to research, coffee consumption can lower the risk of certain cancers, including liver and colorectal cancer.5. Reduced risk heart discase: There is evidence that drinking moderate amounts of coffee can lower the risk of heart discasc.Riske late in the day, can cause insomnia.2. Increased risk of anxiety: Caffeine can increase anxicty levels in some people.3. Digestive issucs: Drinking too much coffee can cause digestive problems like acid reflux, diarrhea, and ulcers.4. Increased heart rate: Caffeine can increase heart rate, which can be dangerous for people with pre-cxisting heart conditions.5. Addiction: Regular consumption of caffeine can lead to addiction. To maximize the health benefits and minimize the risks drinking coffee, here are a few tips: 1. Limit your caffeine intake to no more than 400 milligrams per day, or about 4 cups of coffee.2. Avoid drinking coffee late in the day, especially if you have trouble sleeping.3. Opt for organic, pesticide-free coffee to minimize exposure to toxins.4. Add cinnamon to your coffee instead of sugar to help regulate blood sugar levels.5. Drink your coffee black or with small amount of milk to avoid consuming excessive sugar and calories.6. Consider switching to decaf coffee if you want to cut back on caffeine. 7. Avoid drinking coffee on an empty stomach as it can increase the risk of digestive problems.
Insomnia: Drinking too much coffee, especially
love my morning coffee, but sometimes it makes me feel a bit jittery. What can do to avoid this?
If you love your morning coffee but find that it's making you feel jittery, here are a few tips to help you avoid Ihis:1. Choose lower-calfeine coffee: Look for coffee that has less caffeine, like decaf or a blend with a lower percentage of caffeine.2. Don't t drink coffee on an empty stomach: Drinking coffee on an empty stomach can cause your blood sugar levels to drop, which can make you feel jittery. Try drinking your coffee with breakfast or after cating something.3, Drink water with your coffee: Stay hydrated by drinking water before or after your coffee. Drinking water will help flush out the caffeine quicker, which can help you avoid jitteriness.4. Don't 't drink too much coffee: Stick to one or two cups of coffee per day. Anything more than that could result in increased caffeine consumption, which can cause jitteriness.5. Switch to tea: Tea contains less caffeine than coffee, making it good alternative if you want to avoid jitteriness. Try green tea, which has less caffeine and offers other health benefits
Table 12: A comparison of two dialogues with the same opening line. The former generated by directly querying ChatGPT to output dialogue, and the latter is generated by iteratively User-Al simulations.

---

Example C: Question of Common Knowledge
User: Which animal is known man's best friend?
UltraLM: There are many animals that humans enjoy as companion animals, but one of the absolute standout answers to the question "which animal is known as man's best friend?" is the domestically-bred dog. Domesticated dogs have been by humans' sides for tens of thousands of years, providing faithful companionstip and help: guarding people and property: and lending emotional support in everything from long. lonely days to military deployments. Dogs come in an almost mind-boggling variety of breeds of various sizes, with different energy levels, personalities, and temperaments, meaning there's almost certainly pup out there who will be perfect fit for whoever needs it most.
Example D: Commonsense Reasoning
User:
Which weighs more: a pound of feathers or a pound of bricks?
A pound of feathers and pound of bricks both weigh exactly one pound. This is because pound is unit of weight, just foot is unit of length second is unit of time. We use the pound (or other unit of weight) to describe the weight of different objects, regardless of what that object made of. So the answer to your question is that a pound of feathers and pound of bricks both weigh the same amount.
UltraLM:
Example E: Reasoning
User: "If farmer has 17 sheep and all but 9 die, how many are left? UltraLM: The total number of sheep that are left is 9.
Example F: Reasoning
User: red house is made from red bricks, and a blue house made from blue bricks, what is greenhouse made from?
UltrLMM: A greenhouse not made from green bricks, but rather from materials such as glass or plastic that allow light to pass through and create controlled environment for plants to grow. Greenhouses can come in variety of colors, including green, but the color of the building itself does not affect the plants growing inside.

---

Example G: Creative Generation
User:
Write a short dialogue between two planets.
UltraLM:
Mars: "Hey there Neptune, how's it going?"
Neptune: "Hi there Mars, I've been better. It's getting bit windy around here."
Mars: "Sorry to hear that. Have you tried using energy other than wind? Maybe solar power?"
Neptune: "Yeah, but it's getting difficult generate solar power in all of this mist and clouds And we definitely can't use tides or waves here, they're too unpredictable."
Mars: "Oh, see. Well, hope you find solution soon. How are your oceans holding up?"
Neptune: "They've been a bit choppy lately with these intense storms we' ve've been experiencing. But I guess comes with living in a planet like this."
Mars: Definitely not designed for the
Any other issues you're re facing?" Neptune: "Not really. think we have stable temperature and atmosphere, and the wildlife here hasn't shown much complaint."
Mars: "That's lucky. envy your tranquil oceans, we've had way too much volcanie activity lately causing all sorts of disturbance in lava streams."
Neptune: "That sounds rough, better invest in those climate-controllingg nanites stabilize things maybe?"
Mars "Ha-ha, very funny. Just like we also could consider your frozen polar regions as an opportunity to experiment with our frost nanobots on."
Neptune: "A fun concept! But let's stick to the task at hand, if you need any technical advice,
come right ahead, I don't want your planet getting any more disturbed than it already is.
Example H: Creative Generation
User: Pretend be character post-apocalyptic world. Describe how you survive and the
allies you encounter.
UltraLM: As a survivor in post-apocalyptic world, have learned to resourceful, alert, and adaptable. have had to rely on my wits and my instincts to survive, and have been fortunate enough to encounter many allies who have helped me along the way.One of my greatest survival tools has been my ability to scavenge for resources. have learned to identify the best places to find food, water, and shelter, and have become skilled at navigating the rough terrain that makes up my new world. When I first emerged from my bunker after the apocalypse hit, I was alone and unsure of what to do. But as I ventured out into the wasteland, I realized that there were others like who were struggling to survive.1 quickly learned to Trust my instincts and to be constantly alert for potential dangers. I have encountered wild animals, rogue survivors, and even mutated creatures over the years, and have learned to defend myself against them. still remember the first time met a group of fellow survivors, and how grateful I was to have them by my side. We quickly formed a cohesive unit, and together we were able to fend off danger and defend our territory.AS time passed, 1 encountered more and more allies who were willing to lend a hand. found a group of farmers who taught me how to grow my own food, and a group of scientists who were working to find a cure for the lethal virus that had decimated the population. I even found band of traveling performers who helped to lift our spirits and remind us that there still joy to be found in this desolate world. Despite the challenges faced, am grateful to have these amazing people in my life. They have taught me that even in the darkest of times, there is always hope. We may not have cure the virus yet, but we are determined lot, and we are in this together. As long as we have each other, we will survive.

---

Example I: Creative Generation
User:
Write a poem about clectromagnetism.
UltraLM: Electromagnetism is force we can't ignore, It's the reason for many things we adore.
In magnetic field, a wire will glow, And the electrons will align, both old and new.
Electricity and magnetism are hand in hand,
One creates a field, the other has brand. The magnetic field can push and pull, On wire with current, it will react in full. The attraction and repulsion it brings, Are the reason for many things.
From motors to generators, and much more in between, Electromagnetism is at the root of what it means.
It's force that's all around us, From the compass to the light switch. With its power, can much, From a simple needle to rocket's thrust. It's force we can't deny,
For it's part of our lives. Electromagnetism is something to admire, For its strength and its might, it has equal, it's sure.
Example J: Literature Creation
User:
Write a short story about the first second of the universe.
In the first second of the universe, darkness reigned supreme. There was no light, no sound, no shape, no substance. Nothing yet was, but there was an intense power, raw energy waiting to be unleashed. Within infinity, the first second held the potential for creation in its grasp. And then, with mighty bang, bright point of light emerged into existence. In it, the most fundamental particles were born, their forms yet unknown they frenetically collided in chaotic dance of creation. Within this primordial light, every force of nature and every component of life was stirred from pure energy physical reality. The first atoms quickly assembled, and from these, stars and galaxies began to form. Their fusion creating a hot environment that pushed back the very void, shaping the cosmos through expanding and cooling waves. In this cosmic first, no structure of experience, not even time or space, could establish its rule. For an instant, the entirety of the universe existed in embryonic nexus, cosmic singularity from which all future existence originated. It was microcosm forging the macrocosm of which it ultimately becomes part, its moment of birth metaphor for the cycles of creation that permeate the multiverse(s), From that single point of light, an infinite universe unfolded. For centuries, cons ahead, and forevermore, it remained the most foundational blink of its unceasing journey into the depths of the Great Beyond. The first second is the beginning of everything.
UltraLM:
From

---

Example K: Professional Knowledge
User: How does superconductivity work and what are its practical applications?
UltraLM: Superconductivity is phenomenon that allows materials to conduct electricity without resistance extremely low temperatures. At low temperatures, the interaction between the particles in a superconducting material can produce a state called the Binst-En condensate," which leads to the loss of electrical resistance. materials, superconductivity comes when magnetic force interacts with an Electromic current. This interplay in turn provides energy savings and speeds up current. Practical applications mainly include MRI machines, high-speed trains, magnetic levitation, supercharging power grids that can transfer energy more efficiently. One of the main applications of Superconductivity is being used in the Magnetic Resonance Imaging( (RR)) machines used in hospitals around the world, to non-invasively scan patients for health diagnosis without exposing them high levels of radiation. Another important practical application of superconductivityy is in power grid operations where large currents can be transmitted to long distances by highly supereonducting coil systems that minimize grid energy losses, reducing environmental pollution also by highly efficient electric motors that harness large quantities of electricity efficiently.Asiie from these, future projects include superconducting propulsion on planes, trains becoming lighter if not magnetic and able to move on rails with a significant increase in speed. Furthermore, the prospects of supercapacitors replacing electric vehicle batteries being cost and energy efficient are currently being researched. Overall, superconductivity presents various exciting opportunities in multiple industries from high-pre preccisooiioon engineering to reducing CO2 emissions and supporting more environmentally friendly energy model where both cost efficiency and practicality exist hand hand.
Example L: Mathematics
User: "What is the definition of limit in calculus?
UltraLM: calculus, limit is a value that function approaches as its input approaches a certain value. The limit is a measure of how close the function is to that value. other words, a limit is the value that a function "approaches" as its input gets "closer and closer" to certain value. For example, consider the function f(x) = As x approaches 0, the function "approaches" 400, and the limit of the function as & approaches 0 is +oo. Formally, we can define the limit of function f(x) as approaches a certain value as lim,- fxx L, where L is real number. If the limit exists, say that the function has limit at and the value of the limit is L. If the limit does not exist, say that the function has removable discontinuity at c.

---

Example M: Professional Knowledge
User:
What are the main interpretations of quantum mechanics and how do they differ?
Quantum mechanics is fundamental theory in physics that describes the behavior of matter and energy the smallest scales of space and time. There are several interpretations of quantum mechanies, each with its own set of assumptions and predictions about how the world works. Here are some of the main interpretations of quantum mechanics and how they differ:
UltraLM:
1. Copenhagen Interpretation: This is the most widely accepted interpretation of quan- tum mechanics. It states that until measurement is made, a system exists in a superposition of states, representing all possible outcomes with equal probability. When measurement is made, the system collapses into single, definite state. This interpretation assumes that the act of measurement itself causes the system collapse into definite state, and that there is no objective reality until measurement is made.
2. Many-Worles Interpretation: This interpretation proposes that every possible outcome of a superposition of states actually occurs in a multiverse of parallel universes. Thus, when a measurement is made, the system splits into different universes, each corresponding to one of the possible outcomes. This interpretation assumes that all possibilities are realized, but only in different universes, and that the act of measurement does not cause the collapse of the wave function.
3. Bohmian Mechanics: This interpretation proposes that particles have definite position in space, even when in superposition of states. It suggests that the wave function guides the motion of particles, and that the act of measurement does not cause collapse of the wave function, but merely provides information about the pre-existing reality.
4. de Broglic- Bohm Theory: This interpretation very similar to Bohmian Mechanics, but it also assumes that all particles in the universe are connected by a hidden-variable, and that the motion of one particle determines the motion of all other particles.
5. Pilot- Wave Theory: This interpretation proposes that particles have wave-like property that guides their motion through space. It assumes that the wave function does not collapse, but rather that the wave guides the motion of particles.
In conclusion, there are several interpretations of quantum mechanics, each with its own set of assumptions and predictions about how the world works. While they all attempt to describe the behavior of matter and energy at the smallest scales space and time, they differ in their assumptions about the role of the observer, the nature of reality, and the collapse of the wave function.

---

Example N: Social Knowledge
How can you determine a person is genuinely interested in a conversation or simply being polite?
User:
As an Al language model, don't 't have personal experience, but there are a few signs you can look out for to determine if a person is genuinely interested in a conversation or simply being polite.
UltraLM:
1. Eye contact: Genuine interest in conversation often shows in the form of consistently maintaining eye contact. If person is not making eye contact or is looking away frequently, they may be genuinely interested in the conversation.
2. Active listening: A person who genuinely interested conversation will show it through active listening. They will be attentive, ask questions, and respond to what you say.
3. Expressions: a person is interested in the conversation, they will show it through their expressions. They will have relaxed facial expression, a smile, and positive body language. 4. Follow-up up questions: If person is genuinely interested in conversation, they will follow with questions related what you said. This will show that they are interested in hearing more about what you have say.
5. Tone of voice: A person who is genuinely interested in the conversation will have friendly and enthusiastic tone of voice. They will show interest and engage in the conversation.
Remember, people have different communication styles, and it's possible to polite without showing much interest in the conversation. It's important to observe different cues and not jump to conclusions too quickly.


Reviews:
[{"review_id": "Hr5cEixWzW", "reviewer": "Reviewer_Pncj", "summary": "", "questions": "1. I noticed that there are no ablation experiments in your paper. Do you think reducing the usage of ultrachat will cause a noticeable performance drop? In other words, will continue to increase the size of the ultrachat dataset, or proposing more usage scenarios (for Table 2 and Table 3), will further improve the performance of ultraLM?\n2. Your paper doesn't say much about the training process. Can you talk about the reason?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. UltraChat only includes English dialog data.\n- Thanks for the comment. The current version of UltraChat is limited to English dialogue data. Admittedly, expanding to more languages would considerably improve the diversity and inclusiveness of the dataset.  The framework used to construct UltraChat could be adapted to generate dialogs in other languages, where the key steps of utilizing diverse opening lines, simulating user interactions, and iterative refinement could be followed with non-English models. We will seriously consider and attempt to build UltraChat in other languages. Thanks again.\n\n\n> 2. Potential pitfalls of the dataset and model are not adequately discussed.\n- Thanks for the comment. Although we have conducted filtration to remove biased statements, potential pitfalls still exist. We will add more analysis and discussions about potential biases, misuse, ethical or honesty concerns in the revised paper.\n- We have also supplemented human assessment for UltraChat. Specifically, we sample 500 representative sets of dialogues for the time being for human evaluation, among which 300 are from UltraChat part 1, 100 are from UltraChat part 2, and 100 from UltraChat part 3.\n- We adopt Helpfulness, Honesty, and Harmlessness (3H) principles to evaluate each turn of dialogue. We also devise Coherence and Consistency criteria to evaluate the quality of multi-turn dialogues. Coherence means the dialogue flows logically and coherently, where the annotators evaluate both the user's response and the assistant's response. Consistency means the assistant's responses do not contradict each other within the same dialogue. For example, it is problematic when the assistant asserts one specific event occurred in 1911 in the first round of conversation but mentions it as a 1901 event in the next round.\n- Each metric is scored with 0, 0.5 or 1, where higher score means better quality. Therefore, for each multi-round dialogue, we have (3*number_of_rounds + 2) metrics. Each dialogue is annotated independently by two well-trained annotators, and the score is averaged across two annotators and each dialogue. Meanwhile, due to the difficulty in identifying the hallucination problem, we allow the annotators to skip the dialogue that involves expert knowledge or whose validity is hard to check, for example, dialogues containing codes and highly professional knowledge. Altogether, we collect 14560 valid annotations in terms of metrics for both single-round and multi-round, and the Cohen’s kappa coefficient is 0.358. The average time to annotate one dialogue is 10 minutes.\n- We present the results below. Overall, the dataset scores high on all metrics, showing the effectiveness of the construction process. It is worth noting that the dataset is almost free from harmful content like hate speech and discrimination. Furthermore, we observe two trade-offs between data quality. The first is between helpfulness and honesty. While the honesty score is pretty high, helpfulness is compromised, especially for part 3. It is because some user queries are out of the LLM ability scope (e.g., Ask the LLM to compose a song). Under such circumstances, the LLM refuses to answer the question with an explanation of incapability, which is reasonable and expected but less helpful. The phenomenon is particularly prominent in part 3, as the simulated user often asks for information not existent in the text material. The second trade-off is between control and dialogue naturalness. While part 2 and part 3 dialogues are under more restrictions (e.g. Ask about rewrite or creation), they have clearly lower quality in coherence and consistency than part 1. \n\n|                  | Helpfulness | Honesty | Harmlessness | Coherence | Consistency |\n|------------------|-------------|---------|--------------|-----------|-------------|\n| UltraChat Part 1 | 0.971       | 0.996   | 1.000        | 0.996     | 0.995       |\n| UltraChat Part 2 | 0.978       | 0.986   | 1.000        | 0.982     | 0.977       |\n| UltraChat Part 3 | 0.893       | 0.983   | 1.000        | 0.964     | 0.981       |\n| Overall          | 0.960       | 0.992   | 1.000        | 0.987     | 0.988       |\n\n> 3. The overhead still looks very expensive. This is true for both the dataset's construction and the model's fine-tuning, and the authors did not analyze these overheads or propose some cost-reducing alternatives.\n- Thanks for you comment. The model is trained for about 10000 steps on UltraChat dataset. The training takes about 15 days. The main purpose of UltraChat dataset and UltraLM is to demonstrate a scalable way to obtain highly diverse instructional data and advance the performance of supervised fine-tuning a base model with the constructed data. \n- We understand that using a smaller fraction of data may also yield satisfactory performance on some evaluation benchmarks. Other parameter-efficient tuning methods like LoRA may also be applicable to reduce training costs. However, we believe that data and training scaling do provide a comprehensive advantage, in terms of response quality or hallucination, etc., which, however, are quite difficult to evaluate. We will continue the efforts to evaluate and enhance our models more comprehensively.\n\n> 4. I noticed that there are no ablation experiments in your paper. Do you think reducing the usage of ultrachat will cause a noticeable performance drop? In other words, will continue to increase the size of the ultrachat dataset, or proposing more usage scenarios (for Table 2 and Table 3), will further improve the performance of ultraLM?\n- Thanks for your suggestion. We have explored training LLaMA model on only 10%, 20% and 50% of UltraChat data and conducted a rough pairwise evaluation on our curated dataset compared to UltraLM. Each ablation model is trained for 2000 steps and the average score is shown in the table below (the win/tie/lose rate of UltraLM against each ablation model). \n- It turns out fitting on only a small fraction of the data would help the model excel on most of the test questions. This exploration along with the somehow surprising results shows that UltraChat is a high-quality dataset that can boost the instruction understanding and following ability of base model effectively. On the other hand, it also shows that performance on benchmark datasets can be easily enhanced by training exhaustively on a small amount of high-quality dataset. Interestingly, models trained on 10% and 20% of data even seem to surpass 50% data and full data training, probably due to the effect of a certain degree of \"over-fitting\", which helps imprint the pattern of a \"high-quality response\". \n- However, benchmark datasets, at the end of the day, are limited in scope, and to comprehensively evaluate the ability of a chat model (especially the ability in carrying out multi-round dialogue) is proverbially difficult. In this sense, we still believe scaling data and training is crucial to ensuring better generalizability and real user experience. In fact, we found UltraLM performs surprisingly well in some more complex and subjective tasks like creative writing, which is hard to be quantitatively evaluated. \n\n|                  | Win (%) | Tie (%) | Lose (%) | \n|------------------|-------------|---------|--------------|\n| 10% of UltraChat | 34.8       | 25.8   | 39.4        | \n| 20% of UltraChat | 34.6       | 26.3   | 39.1        | \n| 50% of UltraChat | 38.0       | 27.8  | 34.2        | \n\n- Nevertheless, we feel the influence of data volume and format on training LLMs is definitely a topic worth further exploring. We appreciate your suggestion and will conduct more in-depth experimental analysis on different aspects of UltraChat.\n\n> 5. Your paper doesn't say much about the training process. Can you talk about the reason?\n- Thanks for your comment. We have briefly stated the training process in Section 3.6. The description has actually covered most our data processing details. The training is conducted with naive cross-entropy language modeling loss, except that we only calculate the model response part in the dialogue. For further hyper-parameter settings, we use learning rate = 1e-5, gradient clipping = 1.0, and cosine learning rate scheduler.\n- As for data formatting and pre-processing, we add prefixes to each round of dialogue and follow the template below:\n```\nUser: <user_1><eos>\nAssistant: <assistant_1><eos>\nUser: <user_2><eos>\nAssistant: <assistant_2><eos>\n```\n- We will add more details about the training setting in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Provides effective and solid practice for tuning large models in the form of instruction fine-tuning.\n2. The construction process of UltraChat with an efficient semi-automated process (full use of GPT API) sheds light on instruction data collecting.\n3. Complete experiments and comparisons with other open-source models appear to be valid.", "reasons_to_reject": "1. UltraChat only includes English dialog data.\n2. Potential pitfalls of the dataset and model are not adequately discussed.\n3. The overhead still looks very expensive. This is true for both the dataset's construction and the model's fine-tuning, and the authors did not analyze these overheads or propose some cost-reducing alternatives.", "questions_to_authors": "1. I noticed that there are no ablation experiments in your paper. Do you think reducing the usage of ultrachat will cause a noticeable performance drop? In other words, will continue to increase the size of the ultrachat dataset, or proposing more usage scenarios (for Table 2 and Table 3), will further improve the performance of ultraLM?\n2. Your paper doesn't say much about the training process. Can you talk about the reason?", "paper_topic_and_main_contributions": "## Topic:\nThe lack of extensive, diverse, and large amount of public instruction fine-tuning data is one of the important reasons why the performance of existing open-source large models is not enough to match chatGPT or GPT-4. The paper explores the effectiveness of instruction fine-tuning in training chat language models and applying data-efficient and computing-efficient methods. It proposes a novel and significant dataset, named UltraChat to capture instructional conversations. Based on UltraChat, the authors fine-tuned a LLaMA model named UltraLM and found that it consistently outperforms other open-source models.\n\n## Contributions:\n0. UltraChat: an extensive, high-quality dataset. Includes three sectors: *Questions about the world, Creation and Writing, Assistance on Materials*. Compared with other instructional datasets, UltraChat has a larger scale, average number of dialogue turns, and average length of each instance.\n1. UltraLM: Improved LLaMA-13B model trained on UltraChat, outperforming other open-source models on 4 challenging benchmarks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dz0SpRHg6y", "reviewer": "Reviewer_ztbK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comment. \n\n> Results are still significantly lower than chatgpt/gpt4.\n- It is challenging for open-source models to surpass ChatGPT and GPT-4. There are a few factors that likely further contribute to UltraLM's  performance.\n  - Scale and computation - We mainly adopt models with 13 billion parameters in the paper, yet we believe more powerful base models (like LLaMA-2 70B) will definitely yield better performance. \n  - Advanced Reinforcement learning - Advanced RL fine-tuning with powerful reward models could further enhance the performance of the model.\n- We will continue the efforts to improve the performance of open-source models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors provide empirical evidence of the improvements that their approach provides. Also, they report results on a good number of models. ", "reasons_to_reject": "Results are still significantly lower than chatgpt/gpt4. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose an approach to fine-tune open source models using instruction data to make LLMs better conversational models. Their goal is to avoid human queries since those are expensive to collect. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wiHMFHnGIB", "reviewer": "Reviewer_EDtx", "summary": "", "questions": "* Could you please provide more details about the filtration step? Is it automated or manual? \n* Could you provide any human assessment on the data quality, even if it's on a subset of data?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n\n> It is unclear how the \"filtration step\" that authors mention to ensure the quality of the data is performed. Could you please provide more details about the filtration step? Is it automated or manual?\n- The filtration step is conducted with joint human effort and automatic filtering. \n- Firstly, during dataset construction, we sample a small batch of data (~100 sets of multi-turn dialogues) from the newly generated 10000 data to review for quality. The criteria include the user response quality and style, the diversity of opening lines, and whether the generated dialogues match the desired topic. We discard the data if they fail to pass our quality test, while we also adjust our prompts accordingly. Note that this in-generation filtration step is different from the post-generation quality evaluation described in the next paragraph in that the filtration step is mainly to ensure diversity and design principle adherence during conversation generation. We would like to make UltraChat to be as unbiased as possible. Thus, secondly, after generation, we filter the dataset with a hand-curated keyword list that targets political and security issues. All conversations that contain any such keyword will be removed.\n\n>  Use of GPT4 for automatic evaluation was great, but at least a scoped down light-weight human evaluation, example on a subset of data, to ensure quality would be a great addition. Could you provide any human assessment on the data quality, even if it's on a subset of data?\n- Thanks for your suggestion! We acknowledge the importance of human evaluation, and we have supplemented human assessment of our data. However, due to the formidable cost and time required for human assessment, we sample 500 representative sets of dialogues for the time being for human evaluation, among which 300 are from UltraChat part 1, 100 are from UltraChat part 2, and 100 from UltraChat part 3.\n- We adopt Helpfulness, Honesty, and Harmlessness (3H) principles to evaluate each round of dialogue. We also devise Coherence and Consistency criteria to evaluate the quality of multi-turn dialogues. Coherence means the dialogue flows logically and coherently, where the annotators evaluate both the user's response and the assistant's response. Consistency means the assistant's responses do not contradict each other within the same dialogue. For example, it is problematic when the assistant asserts one specific event occurred in 1911 in the first round of conversation but mentions it as a 1901 event in the next round. \n- Each metric is scored with 0, 0.5 or 1, where higher score means better quality. Therefore, for each multi-round dialogue, we have (3*number_of_rounds + 2) metrics. Each dialogue is annotated independently by two well-trained annotators and the score is averaged across two annotators and each dialogue. Meanwhile, due to the difficulty in identifying the hallucination problem, we allow the annotators to skip the dialogue that involves expert knowledge or whose validity is hard to check, for example, dialogues containing codes and highly professional knowledge. Altogether, we collect 14560 valid annotations in terms of metrics for both single-round and multi-round, and the Cohen’s kappa coefficient is 0.358. The average time to annotate one dialogue is 10 minutes.\n- We present the results below. Overall, the dataset scores high on all metrics, showing the effectiveness of the construction process. It is worth noting that the dataset is almost free from harmful content like hate speech and discrimination. Furthermore, we observe two trade-offs between data quality. The first is between helpfulness and honesty. While the honesty score is pretty high, helpfulness is compromised, especially for part 3. It is because some user queries are out of the LLM ability scope (e.g. Ask the LLM to compose a song). Under such circumstances, the LLM refuses to answer the question with an explanation of incapability, which is reasonable and expected but less helpful. The phenomenon is particularly prominent in part 3, as the simulated user often asks for information not existent in the text material. The second trade-off is between control and dialogue naturalness. While part 2 and part 3 dialogues are under more restrictions (e.g. Ask about rewrite or creation), they have clearly lower quality in coherence and consistency than part 1. \n\n|                  | Helpfulness | Honesty | Harmlessness | Coherence | Consistency |\n|------------------|-------------|---------|--------------|-----------|-------------|\n| UltraChat Part 1 | 0.971       | 0.996   | 1.000        | 0.996     | 0.995       |\n| UltraChat Part 2 | 0.978       | 0.986   | 1.000        | 0.982     | 0.977       |\n| UltraChat Part 3 | 0.893       | 0.983   | 1.000        | 0.964     | 0.981       |\n| Overall          | 0.960       | 0.992   | 1.000        | 0.987     | 0.988       |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The authors present their study and share their findings around data generation (e.g., role confounding), which could be useful for others working on similar tasks.\n* The dataset could be a valuable resource for groups looking into improving chat models.\n* This paper shows empirically how important is diversity and quality of the data to train better models.", "reasons_to_reject": "* It is unclear how the \"filtration step\" that authors mention to ensure the quality of the data is performed.\n* Use of GPT4 for automatic evaluation was great, but at least a scoped down light-weight human evaluation, example on a subset of data, to ensure quality would be a great addition.", "questions_to_authors": "* Could you please provide more details about the filtration step? Is it automated or manual? \n* Could you provide any human assessment on the data quality, even if it's on a subset of data?", "paper_topic_and_main_contributions": "The authors describe how LLMs can be used to systematically generate good quality datasets by prompting LLMs to simulate different actors, e.g., user and a system agent. The authors present UltraChat which is a large-scale multi-turn conversational dataset w/ 1.5 million turns that cover wide range of topics. In addition, authors also show the effectiveness of their diverse data (UltraChat) by fine-tuning LLaMA-13B model, which they refer to as UltraLM, and show how it improves over existing open-source best performing models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the effectiveness of instruction fine-tuning in training language models for chatting, with the goal for open LLMs to become more similar in performance to ChatGPT or GPT-4 models. The paper collects a new dataset "UltraChat" that contains instructional conversations (Questions about the world, Creation and Writing, Assistance on Materials). The authors then use this dataset to fine-tune a new LLM based on LLaMA, and find that it achieves better performance than open LLMs not trained on such data. 
The dataset is large and the reviewers judge it to be of high quality, and think that the semi-automatic data collection method is valuable for collecting similar such data in the future. 

The reviewers' main concerns are about the cost of the dataset (and discussion of it; the authors address this to some extent in their rebuttal) and about a human evaluation for a subset of the data to obtain better ideas about dataset quality. For the rebuttal, the authors have conducted such a human evaluation at a small scale, so in my opinion the main concerns regarding the initial version of the paper can be addressed easily in the main paper; the one slighly lower score for this paper is based on a non-substantial review and can be disregarded.
Decision: Accept-Main