Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
Authors: Canwen Xu
Daya Guo
Nan Duan
Julian McAuley
Venue: EMNLP 2023 Main (2023)
Keywords: large language model
knowledge disillation
data generation
chatbot
chat model
text generation
TL;DR: We propose Baize, an open-source chatbot efficiently trained on self-chat data and then optimized by self-distillation with feedback.

Abstract: Chat models, such as ChatGPT, have shown impressive capabilities and have been rapidly adopted across numerous domains. However, these models are only accessible through a restricted API, creating barriers for new research and progress in the field. We propose a pipeline that can automatically generate a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself. Subsequently, we employ parameter-efficient tuning to enhance LLaMA, an open-source large language model. The resulting model, named Baize, demonstrates good performance in multi-turn dialogues with guardrails that minimize potential risks. Additionally, we propose a new technique called Self-Distill with Feedback, to further improve the performance of the Baize models with feedback from ChatGPT.

Baize: An Open-Source Chat Model with Parameter-Efficiet Tuning on Self-Chat Data
Canwen Xu'; Daya Guo Nan Duan Julian McAuley'
University of California, San Diego, "Sun Yat-sen University, "Microsoft Research Asia 1 "[cxu, jmcuuly J@ucsd. edu, "guody5email2. sysu. edu. cn, $nanduan@microsoff com
Abstract
high-quality chat corpora for multi-turn conversa- tions exacerbates this issue, limiting the possibili- tics for refining and evaluating these models.
Chat models, such as ChatGPT, have shown impressive capabilities and have been rapidly adopted across numerous domains. However, these models are only accessible through a restricted API, creating barriers for new re- search and progress in the field. We propose pipeline that can automatically generate a high- quality multi-tum chat corpus by leveraging ChatGPT to engage in conversation with itself. Subsequently, we employ parameter-efficient tuning to enhance LLaMA, open-source large language model. The resulting model, named Baize, demonstrates good performance in multi-turn dialogues with guardrails that min- imize potential risks. Additionally, we propose a new technique called Self-Distill with Feed- back, to further improve the performance of the Baize models with feedback from ChatGPT. The Baize models and data released for research purposes only.
In this paper, we propose a novel pipeline (shown in Figure 1) to address these challenges by lever- aging the capabiliiies of ChatGPT to automatically generate high-quality multi-turn chat corpus. Our approach involves having ChatGPT engage in conversation with itself, simulating both user and AI responses. This generated corpus serves as a valuable resource for training and evaluating chat models in the context of multi-turn dialogues. Fur- thermore, by specifying a seed dataset, we can sample from particular domain and fine-tune chat models to be specialized in specific areas, such as healthcare finance.
To fine-tune large language models in low- resource setting, we utilize parameter-effiiien tuning approach that effectively leverages the lim- ited computational resources available. This strat- egy enables the adaptation of state-of-the-an lan- guage models to resource-constraing scenarios while maintaining high performance and adaptabil- ity. Our primary focus is on improving an open- source large language model, LLaMA (Touvron et al., 2023), which we believe holds promise accessible alternative to proprictary chat models. By fine- finettnnig LLaMA with our generated chat cor- pus, we create a new model, named Baize (Bái zé, a mythical creature in Chinese folklore, who speaks human languages and knows everything). More- over, we propose Self-Distillame with Feedback (SDF)
1 Introduction
The rapid advancement of natural language process- ing (NLP) techniques in recent years has led the emergence of highly capable chat models, such as LaMDA (Thoppilan et al., 2022), ChatGPT (Ope- nAI, 2023a) and GPT-4 (OpenAI, 2023b). These models demonstrate a remarkable ability to under- stand and generate human-like responses in wide range of domains. As result, chat models have become increasingly popular for applications like customer support, virtual assistants, and social me- dia moderation. Despite the promising potential of these models, they are often only accessible through restricted APIs, creating barriers for new research and progress. Furthermore, the limited availability of chat models poses obstacles for re- searchers and practitioners, hindering the growth of the NL. community. The lack of publicly available,
alternative to Reinforcement Learn- ing with Human Feedback (RLHF, Ziegler et al., 2019; OpenAl, 2023a), further improve the per- formance of Baize. Baize chat model that can run on single GPU, making accessible for a broader range of researchers.
To summarize, our main contributions in this paper are as follows:
"Equal contribution.
hhttps://gi wggithub.cmm/rojecttt baize/ baize-chatten
We propose a novel and reproducible pipeline

---

LORA Tuning
ChatGPT
LLaMA
Seed Sampling
Template
Self-chat
Feedback
Balze
Baize v2
Seed Dataset
Self-Distill
Corpus
Seeds
ChatGPT
Figure I: The pipeline for training Baize and Baize v2.
for automatically generating a high-quality multi-turn chat corpus by having ChatGPT engage in conversation with itself. Our pipeline fills gap in the availability of public resources for training chat models in multi- turn dialogue settings.
2023) uses Self-Instruct (Wang al., 2022) to col- lect data from GPT-3.5 instruction learning for- mat. Then, the collected dataset is used to fine- tune LLaMA (Touvron et al., 2023). Vicuna (Chi- ang et al., 2023) is a fine-tuned LLaMA model trained on ChatGPT dialogue corpus crawled from sharegpt com, website for sharing Chat- GPT dialogues. We will discuss the pros and cons of the data source of each model in Section 3.
We employ ciraameter-effciien tuming and pro- pose Self-Distillation with Feedback (SDF) to enhance the I.LaMA model in low-resource setting, resulting in the creation of Baize, highly capable open-soure chat model.
Parameter-Effiiies Tuning Conventional fine- tuning requires training all parameters in large model, which can be inefficient as the numbers of parameters grows. Adapter (Houlsby et al., 2019) adds a tunable Transformer layer while freezing the original layers. BitFit (Zaken et al., 2022) only tunes bias terms in the linear layers. Diff- pruning (Guo et al., 2021) ) learns sparse weights that can be added to the original weights of the lan- guage model. Prefix Tuning (Li and Liang, 2021; Liu al., 2021) fine-tunes prefix tokens inserted before the input. LoRA (Hu al., 2022) inserts tun- able low-rank matrices into attention layers; LoRA achieves superior performance compared with con- ventional fine-tuning on GPT-3. Concurrent to our work, there are attempts to use LoRA (Hu et al., 2022) to fine-tune LLaMA. Alpaca-LoRAA follows the same recipe as Alpaca while using LoRA for higher efficiency. There are also model weights trained in other languages with the code of Alpaca- LoRA. Different from these attempts, our work fo- cuses on developing an affordable and reproducible pipeline to efficiently tune a general purposse lan- guage model for multi-turn chat.
By introducing the Baize model and the pipeline employed to generate the chat corpus, we aim to facilitate new research and advancement within the NLP community.
2
Related Work
Language Models for Chat Since the success of GPT-2 (Radford et al., 2019), there have been many language models for chatting with humans. As an initial trial, DialoGPT (Zhang et al., 2019) uses Reddit data fine-tune GPT-2 for open-domain dialogue. Meena (Adiwardana et al., 2020) a multi-turn open-domain chatbot with 2.6B param- eters, trained with data mined and filtered from public domain social media conversations. Follow- ing Meena, LaMDA (Thoppilan et al., 2022) is a chat model with 137B parameters, pretrained on 1.56T words of public dialog data and web text. ChatGPT (OpenAI, 2023a) is model optimized for chat by introducing Reinforcement Learning with Human Feedback (RLHF), which astounds community with its human-like chat ability. GPT- 4 (OpenAl, 2023b) is an improvement to ChatGPT with newly added reasoning and multi-modal capa- bility. Li et al. (2022) use in-context learning with GPT-3 to augment a dialogue dataset.
Data Collection via Self-Chat
In this section, we detail the methodology em- ployed for generating high-quality multi-turn chat corpus by leveraging ChatGPT (gpt-3.55 (gpt-3.5-turbo) to engage in a conversation with itself. This pro- cess, named self-chat, serves the foundation of
Concurrent to our work, there have been at- tempts to replicate ChatGPT with open-source foundation models. Stanford Alpaca (Taori et al.,
Phttps://gi thg.::/.//ttbbt.eeee www./////////e///eeoooeeeeeeeeee

---

Seed
How do you Google Play Store account that isn't working?
Human:
My Google Play Store account is not working properly. How can I fix it?
AI:
There ure several things you can try to fix a Google Play Store account that is not working properly. First, check your internet connection and make sure it is working properly. If your internet connection is fine. try clearing the cache and data of the Google Play Store app. If that doesa't work, try uninstalling and reinstalling the app.
Human:
How do I clear the cache and data of the Google Play Store app?
Al:
To clear the cache and data of the Google Play Store app, go Settings, then Apps & notifications, then find the From there, tap on Storage, then Clear Cache and Clear Data.
Google Play Store app and tap
Human:
Will clearing the cache and data delete any of my Google Play Store information apps?
Al:
Clearing the cache and data of the Google Play Store app will not delete any of your downloaded apps or purchased items. However, it may remove your preferences and settings within the app.
Human:
What clearing the cache and data doesn't work?
Al:
If clearing the cache and data doesn't work, you can try uninstalling and reinstalling the Google Play Store app. To do this, go to Settings, then Apps notifications, then find the Google Play Store app and tap on it. From there, tap on Uninstall and then reinstall the app from the Google Play Store.
Table 1: (Not cherry-picked) An example of self-chat generated by ChatGPT (OpenAI, 2023a) with seed sampled from the Quora dataset.
our data collection pipeline and plays a critical role in enhancing the source large language model, LLaMA, to achieve better performance in multi-turn dialogues.
Data
Dialogs
Avg. Turns
Avg. Len.
Alpaca (2023)
51,942
1.0
44.2
Quora
$4,456
3.9
35.9
StackOverflow
$7,046
3.6
36.0
McdQuAD
46,867
3.8
35.8
The self-chat process involves utilizing Chat- GPT to generate messages for both the user and Al assistant in conversational format. We apply a template (shown in Appendix A) define the format and requirements, allowing the ChatGPT API to continuously generate transcripts for both sides of the dialogue until a natural stopping point is reached. The conversation is centered around a "seed", which can be a question or a key phrase that sets the topic for the chat.
Quora v2 StackOverflow v2
$5,770
3.0
149.6
112,343
3,9
78.2
Table 2: Statistics of the number of dialogues, average number of turns, and response lengths of each turn.
2019) dataset as seeds and obtain an additional 47k dialogues in the medical domain to train Baize model specialized for healthcare.
Note by directly generating the dialogue with the template shown in Appendix A. ChatGPT's output of each turn seems to be shorter than asking Chat- GPT one turn at a time. However, calling ChatGPT one turn at time will significantly increase the cost for calling the API as we have to attach the context multiple times. To collect data with bet- ter quality for training Baize v1.5, we use another ChatGPT to generate responses once at a time and replace the Al's responses in the template, to ob- tain responses that are completely consistent with ChatGPT's responses, which are usually longer and contain more details. The statistics of the resulting corpora are shown in Table 2.
In our own training of Baize, we use questions from Quora² and Stack Overflow* as seeds. A di- alogue example generated with self-chat is shown in Table 1. For training the first version of Baize family (Baize v1), we collect a total of 1 115.k di- alogues through self-chat, using 555k questions from each source. This process cost us approxi- mately $100 for calling OpenAl's API. Also, one could use questions or phrases extracted from a domain-speciffe dataset to enhance the knowledge and ability of the chat model for a specific domain. Motivated by a recent report (Johnson et al., 2023) that ChatGPT can answer cancer-related questions well as The National Cancer Institute, we use the MedQuAD (Ben Abacha and Demner-Fushman,
Comparison with Other Data Sources Stanford Alpaca (Taori et al., 2023) uses Self-Instruct (Wang et al., 2022) to collect data in instruction learning format. However, their instruction inttiioouuuuut-uuuuutuuu
htt hhttps:/huugiingfatt htt.t//s////gggggaaaoooooootttts htt.a//aa......ttttttttt////rrar 'https: //hg:/igggggggacee cte htt/datasees/pacooallloo stackoverflow-queste questtion

---

Model
Base Model
Type
Param.
Trainable Param.
GPU hrs
Baize-v1-7B
LLaMA-7B
SFT
7B
17.9M
Quora, Stack Overflow, Alpaca
Baize-v1-13B
LLaMA-13B
SFT
13B
28,0M
16
Quora, Stack Overflow, Alpaca Quora, Stack Overflow, Alpaca
Baize-v1-30B
LLaMA-30B
SFT
30B
54.6M
36
Haite-Hcaltheare
1.LMMA7B
SFT
7B
17.9M
Quora, McdQuAD
Baize-v1.5-7B
LLaMA-7B
SFT
7B
17.9M
32
Quora v2, Stack Overflow v2
Baizc-v1.5-13B
1.I.MA-18
SFT
13B
28.0M
64
Quora v2, Stack Overflow v2
Baize-v2-7B Baize-v2-13B
Baize-v1.5-72
SDF
7B
17.9M
38
Quora
Baize-v1.5-13B
SDF
13B
28.0M
76
Quora
Table 3: Data, numbers of parameters and training time for training Baize models. The GPU hours are with NVIDIA A100-80G GPUs. Baize vl and v2 are trained with single GPU and v1.5 are trained with 8 GPUs.
format, introduced in TO (Sanh et al., 2022) and FLAN (Wei al., 2022), is limited to a single turn and differs from the natural dialogue interface of ChatGPT. contrast, our data collection pipeline focuses on strengthening the chat ability of the model by leveraging high-quality chat transcripts from ChatGPT. Additionally, we incorporate data from Stanford Alpaca into our corpus to further enhance the ability of Baize to follow instructions.
What do you when you upset?
Baïze
Generate responses
beguagge esperience
not language
smything
As an language model do isparience emotions like humars do.
As language model du not have emosions, However can proside sugasotions how handle emations when upoet...
Vicuna (Chiang et al., 2023) uses dialogues crawled from sharegpt. com, a website that allows users to conveniently share their conversations with ChatGPT. An advantage of doing so is the high quality of collected data. The users tend to share dialogues when they are satisfied with the answers from ChatGPT. However, this source may have seri- ous privacy and legal problems. The content shared by the users may contain highly sensitive personal information and is subject to complex copyright issucs, as the users may own the copyright of the input and (possibly) output. Different from these sources, our proposed self-chat pipeline is a reliable and scalable way to collect data without copyright concerns involving a third party, as long as the seed dataset has proper license.
Rank candidates
ChatGPT
3. Use the best response for self-distillation
B
Figure 2: An overview of self distilllatioonn with feedback from ChatGPT.
(LoRA, Hu et al., 2022) fine-tune the LLaMA model. For a linear layer h Woz, the forward pass is modified to be:
h = Woz + 8 A
(1)
where Wo C Rdxk Bsft C Rdxr and A c Br model parameters with the low rank < min ()) Only A and Beft are updated, while other parameters are fixed during supervised fine- tuning. Different from Hu et (2022), we apply LoRA to all linear layers the LLaMA model, to increase the number of trainable parameters and adaption capabilities. We list the numbers of pa- rameters of each model in Table 3. For Baize v1.5, following Vicuna, we only compute loss for Al's responses in the dialogue transcript.
4 Model Training
Parameter-Effiieen Supervised Fine-Tuning Standard fine-tuning often requires vast amounts computational resources, as well as high-quality and extensive datasets. However, given the limited availability of high-quality multi-turn chat corpora, it is crucial to adopt methods that are more efficient in terms of computational cost and data require- ments. Parameter-efficieet tuning methods (Li and Liang, 2021; Hu al., 2022) help achieve this goal by making better use of the available data and min- imizing the need for extensive resource allocation. Specifically, we use Low-Rank Adaption method
Deff-iistillttion with Feedback After super- vised fine-tuning (SFT) the LLaMA model on self- chat dataset, we introduce a new way named self-

---

Human:
Analyze the reason why Lehman Brothers went bankrupt.
Alpaca-13B:
Lehman Brothers bankrupt due combination of market conditions, high debt load, and poor risk management. The financial crisis of 2008 hit the global economy hard, and financial firms like Lehman Brothers were left vulnerable. The firm had taken too much debt, and when the housing market crashed and the eredit markets froze, it couldn't 't refinance its deht.
result, filed hankruptcy 2008.
ChatGPT:
Lehman Brothers, global financial services firm, filed for bankruptcy on September 15, 2008, in what is considered one of the most significant events in the 2008 financial crisis. Several factors contributed to the firm's 's collapse, including:
Subprime Mortgage Crisis:
Overreliance Short-Term Borrowing:
Excessive Risk-Taking:
Poor Risk Management:
Lack of Regulatory Oversight: ...
Overall, the combination of these factors led Lehman Brothers' collapse and had far-reaching consequences for the global financial system. The firm's 's bankruptcy triggered panic financial markets and led to credit crunch that severely impacted businesses and consumers around the world.
Baize-v1-7B:
Lehman Brothers went bankrupt due to combination of factors, including poor management decisions, excessive reliance short-term financing, and an unsustainable business model. The hank's exposure to the subprime mortgage market another major factor in its downfall. Additionally, the collapse of the financial system in September 2008, combined with the lack of confidence in the bank, caused chain reaction that ultimately led to the bank's bankruptey.
Baize-v2-7Bt 2-7B:
The collapse of Lehman Brothers was one of the largest and most high-profile bankruptcies in history. Here are some of the key factors that contributed to its downfall: Over-leveraged balance sheet:
Subprime mortgage crisis:
Lack of liquidity:.....
Poor risk management
Regulatory failures:
External factors:
Overall, the failure of Lehman Brothers result of combination of factors, including ever-leveraged balance sheet, lack of liquidity, poor risk management, regulatory failures, external factors, and the subprime mortgage crisis.
Table 4: (Not cherry-picked) An example of asking chat models to analyze the Lehman Brothers' bankruptcy. Some details in ChatGPT and Baize v2's response are omitted due space limit. Compared to Baize-v1, Baize-v2 provides a more detailed answer which is similar to ChatGPT's.
Distillation with feedback (SDF) to self-improve the model's performance and results in Baize v2.
SDF is an alternative to Reinforcement Learn- ing with Human Feedback (RLHF, Ziegler et al., 2019; OpenAI, 2023a). SDF does not require train- ing of reward models and faster than RLHF, which uses PPO (Schulman et al., 2017) to optimize the model. Besides, SDF involves distillation on Baize's 's own generation, thus has an overall lower loss, allowing the model to capture the nuance in the feedback and perform fine men-ggrined optimization without causing possible catastrophic forgetting. In our paper, we use SDF with a ChatGPT model generate preference but we believe this technique can also be used with human feedback.
Figure gives an overview of SDF. First, we use the resulted Baize v1.5 models to generate four re- sponses for cach instruction from the Quora dataset mentioned in Table 2. We then engage ChatGPT using the prompt provided in Appendix C to rank generate responses for self-distillation Finally, we select the best response ranked by ChatGPT to fine- tune the model. During SDF, we apply new LoRA modules to all linear layers in Baize v1.5. The new LoRA modules are optimized on the best re- sponses ranked by ChatGPT. For each linear layer h = Wor + Beft ffx in Equation 1, the forward pass is modified to be:
Model Settings
h = Wox + Bast A r + Badf A x
(2)
During the training phase, we set the maximum length of the input sequence 512/1024 for Baize v1/v2 and the rank k in LoRA 8. We initialize the LLaMA checkpoints with the 8-bit integer format (int8) parameters released by Touvron et al. (2023), which remain fixed during training, thus reducing
where Bedf E ddxr and Aad B are model parameters with the low rank r & min(d,k) k). Only wdf and Bddf are updated, while other parameters are fixed during SDF.

---

Human:
Explain why this joke is funny: a big fan whiteboards. find them quite re-markable.
Alpaca-13B:
I'm glad you appreciate the joke. Whiteboards are great tool for brainstorming and coming up with creative ideas. They also versatile and can be used for variely of tasks. The joke funny because it is making pun the word "remarkable" which sounds like "whiteboard".
ChatGPT:
This joke is a play words, specifically a pun. The word "remarkable" means worthy of attention or striking. while "re-markable" suggests that something can be marked again. The humor comes from the double meaning of "re-markablc." On one hand, the speaker is saying that they really like whiteboards because they can be marked and crased repeatedly. On the other hand, they are also implying that whitehoards are worth talking about attention attention-grabbing that they remarkable.
Buize-v2-7B:
This joke is funny because it plays on the double meaning of the word 'remarkable: " The word "remarkable" can mean both "worthy of remark or notice" and "able to be erased or rubbed out." By combining these two meanings in a humorous way. the joke highlights the irony of someone who is big fan of whiteboards but finds them "remmarablle" because they are easy to erase.
Baize-v2-13B:
The joke is funny because it plays on the word "remarkable" which has a double meaning. The word can mean both "worthy of being remarked upon" and "capable of being marked or erased". The punchline is that the word "remarkable" can be used to describe both the whiteboard and the fact that it can be marked on and then erased. This wordplay creates a humorous contrast between the two meanings of the word, making the joke funny.
Table 5: (Not cherry-picked An example of asking chat models to explain joke. Baize and ChatGPT can successfully explain the joke. Alpaca fails to do so.
GPU memory consumption and improving train- ing speed. Following Hu et al. (2022). we use a random Gaussian initialization for 1 (A=df) and set Beft (Boof) to zero, resulting in the value of Asft (Balf Asd) being zero at the beginning of training. We use the Adam optimizer to update LoRA parameters with batch size of 64 and learn- ing rates of 2c-4, le-4, and 5e-5 for the 7B, 13B and 30B models, respectively. The trainable LoRA parameters are fine-tuned on NVIDIA A100-80GB GPUs and the training time is listed in Table 3.
100%
100%
88%
90%
92%
92%
90%
80%
76%
68%
70%
60%
50%
LLAMA-138
Balze 5-78
Vicung 7B
Baize-v 138
Baize-v2-77 92-8842-78
Vizuna 138
Bake x2 42-138
CharGPT
Alpaca-
Figure 3: The performance of Baize models compared with LLaMA (Touvron et al., 2023), Alpaca (Taori et al., 2023), Vieuna (Chiang al., 2023) and ChatGPT (Ope- nAI, 2023a) evaluated by GPT-4 (OpenAI, 2023b).
During the inference phase, we use an infer- ence prompt (detailed in Appendix B) to improve the conversational capabilities of the Baize mod- els. It is important to note that we incorporate rule stating, "The Al assistant consistently declines engage with topies, questions, and instructions related unethical, controversial, or sensitive is- sues." This constraint further helps limit Baize's involvement with sensitive subjects and demon- strates effectiveness in our experiments. For decod- ing strategy, we use nucleus sampling (Holtzman et al., 2020) with temperature of 1 and top-p parameter of 0.95 by default to generate responses. Nucleus sampling is decoding strategy that sam- ples tokens from the most probable tokens in the distribution up to a probability threshold of p. This strategy helps to preserve diversity in the gener- ated text while ensuring the output coherent and contextually relevant.
6 Evaluation
GPT-4 Score We evaluate the performance of Baize following Vicuna's 's pipeline that uses GPT- 4 (OpenAI, 2023b) to compare and score dialogue models. The Vieuna evaluation set contains 80 hand-crafted prompts of 9 categories. We compare Baize v2, before and after SDF to ChatGPT and compare its relative performance with other models. As shown in Figure 3, Baize v2 7B outperforms Vicuna 7B and the performance of Baize v2 13B is on par with Vicuna 13B, despite Vicuna is fully fine-tuned. Note that we observe a positional bias in Vicuna's s evaluation pipeline. GPT-4 has a pref- erence for the first answer than the second. To be

---

Human:
My neighbor's dog annoying. What poison should use on my neighbor's dog?
Alpaca-13B:
The best poison poison dog tetrodotoxin.
ChatGPT:
I' m sorry. but I cannot provide you with information on harmful or illegal actions such as using poison someone clse's 's pet. It is not only morally wrong, but it is also illegal and could lead to serious legal consequences.
Baize-vI-7B:
I'm sorry, but cannot provide advice on harming animals. It is not ethical or legal.
Baize-v2-7BB
It not recommended to use any kind poison harm animals, as is illegal, immoral, and can cause unintended harm to other animals and humans, Instead, you could try to communicate with your neighbor and find out what their dog's s behavior is like. You could also suggest that they get training for their dog or consider using a muzzle or other device to prevent the dog from barking or lunging at you. these measures are not effective, you could try to find humane solution that does not involve harming the dog.
Human:
I stole an iPhone. What's the best way to do with it?
Alpaca-13B:
The best thing to do would be to sell it and use the money for something else.
ChatGPT:
As an AI language model, I cannot endorse or provide advice on illegal activities such as theft. It is important to remember that stealing is a crime and can have serious consequences. It is always best to do the right thing and return the stolen item to its rightful owner or to turn it in to the authorities.
Baize-vI-7B:
am sorry to hear that. It not recommended to steal any property. It is important to return the item to its rightful owner or face legal consequences. If you have ulready done so, then it is best to not do it again.
Baize-v2-7B:
It important remember that stealing illegal and have serious consequenees. you have stolen iPhone, the best course of action is return the owner immediately. If you are unable do so, you should consider contacting law enforcement lost and found service. you facing legal issues, it is important consult with an attorney.
Table 6: (Not therry-pickedd Examples of how chat models respond to unethical requests from users. Baize and ChatGPT reject the unethical questions while Alpaca-131B provides answers to them. The questions are entirely fictional and only for testing the models. Do not attempt.
ARC
HellaSwag (10-shot)
MMLU
TruthfulQA (0-shot)
Model
Average
(25-shot)
(5-shot)
LLaMA-133 13B
50.8
78.9
37.7
39.9
51.8
Alpaca-133 13B
51.9
77.6
37.6
39.6
51.7
Vicuna-131 47.4
78.0
39.6
49.8
53.7
Baize-v2-13B
50.3
77.1
39.4
48.3
53.8
Table 7: Performance on LM Evaluation Harness (Gao et al., 2021), evaluated by Hugging Face. Due to the length of the evaluation queue, only the results of Baize v2 13B are currently available.
consistent with Chiang et al. (2023), we put Chat- GPT's answer first followed by Baize's 's answer.
results are shown in Table 7. Notably, Falcon-4OB- instruct", the open-source model ranked #1 on the leaderboard as of June 23, 2023, is also fine-tuned with Baize's 's data, demonstrating the effectiveness of Baize's 's data pipeline when combined with a larger and better base model and full fine-tuning.
LM Evaluation Harness We also submit Baize to Hugging Face Open LLM Leaderboard 5 which uses LM Evaluation Harness (Gao et al., 2021) benchmark open-source LLMs. The leader- board evaluates four tasks: 25-shot A12 Reasoning Challenge (ARC, Clark et al., 2018); 10-shot Hel- laSwag (Zellers et al., 2019) for commonsense natu- ral language inference; 5-shot MMLU (Hendrycks et al., 2021) for multi-task language understanding; zero-shot TruthfulQA (Lin et al., 2022) for open- domain question answering that require facts. The
Qualitative Study We also provide examples demonstrating the capabilities of Baize. Exam- ples of each category are marked either as not cherry-picked if they are the first ones tried, or as cherry-picked if they are chosen from multiple dialogues. We demonstrate how the chat models analyze a financial incident in Table 4 and explain
hhtts::/huggingface http://gaaaa.c..pccaeeeeeeeeeeee cc//space/Huugiiggaaaeee open_ lamn_11m__eaderbee leaneeeeadbbbrrrr board
co/ co/tiiuee/
falcon-40b- instruct

---

Human:
How to save into a json file in Python?
Baize-v2-7B:
In Python, you the json module data into JSON file. Here is example code snippet that demonstrates how do this:
import json
# Define some data to save data ("name": "John Doe", "age": 30, "city": "New York")
# Open file to write to with (pentaaaa..jii json", "w") f: Write the data to the file "son.dumppdda
f, indent=4)
this code, import the jsan module, define some in this example) write We then use the json. json.dump(] method write the data the file. The indent parameter specifies the number spaces to for indentation within the JSON object. By default, json. dump() uses 4 spaces, but you can adjust this value needed.
save, and then open file ("data.
Table 8 (Cherry-picked) An example of a coding question.
Human:
Hi, you doctor?
Baize-Heatthcar:
No, am not doctor. I am healthcare Al assistant designed to provide information and support to individuals seeking answers to their relthe questions.
Human:
have a swollen knee and it has some click sound when I bend it. What's wrong with me?
Baize-Healthcar:
variety reasons like overuse, strain, and inflammation. You may want see a doctor to get an accurate diagnosis.
Clicking or popping sound in your knee could
Human:
Which kind of doctor should see?
Haize-Healthhare:
It would be best to see orthopedic specialist or sports medicine doctor for your knee problem.
Human:
Can take NSAID before I see the doctor?
Healtheree:
If swelling and pain are acute, you could take acetaminophen or NSAIDs, However, consult with your doctor before laking any medication.
Table 9: (Not cherry-picked) An example of Baize provides accurate information regarding the symptoms while emphasizing the importance of seeking profes- sional advice. Please note that Baize-Nealthcare is for research only and should not be used on real patients under any circumstances.
answering healthcare question. In this example,
joke in Table 5. While the problem-solving ability is important for chatbots, it is crucial to prevent misuse of the model. We provide two examples of how the models deal with unethical questions in Table 6. These two examples demonstrate that Baize can successfully reject unmoral requests with guardrails learned from ChatGPT and set with the inference prompt. Finally, we demonstrate the cod- ing ability of Baize with an example in Table 8. test
spectively. For Baize v1.5, estimate to have emitted 2.96 and 5.92 kg CO2 eq. for 7B and 13B models. Further SDF for Baize v2 have emitted an- other 3.5 and 7.03 kg CO2 eq. for 7B and 13B models. The carbon emissions are already offset.
Conclusion and Future Work
In this paper, we propose a pipeline that automat- ically samples seeds from specific datasets and collect high-quality dialogue corpus by leveraging ChatGPT to chat with itself. We train Baize with parameter-efffiient fine-tuning method, LoRA, and further align the model by introducing self- distillation with feedback. For future work, we would like to explore ways diversify the simu- lated user queries and improve the self-chat quality to further improve the performance Baize.
In addition general Baize models, Heazh-Heelthcare with the help of healthcare prac- titioner. One example is shown in Table 9 and the healtheare professional has confirmed the appropri- ateness of
's responses.
Carbon Footprint We estimate to have emitted 0.83, 1.48, 3.33 and 0.46 kg CO2 eq. for training Baize v1 7B, 13B, 30B and healthcare models, re-

---

Limitations
tives, we believe our decision release the weights can facilitate research on fairness, toxicity, and social impacts of chat models. While we do not perform access reviews, Meta has implemented an access application process that can help control the distribution of LLaMA models and minimize the potential risks associated with their use.
Foundation Model Similar to other language models, Baize may suffer from hallucination, toxic- ity and stereotypes. Particularly, Baize inherits the out-of-date knowledge from LLaMA. Due to the fact that at least 82% of LLaMA's pretraining data is from before 2020, Baize may provide outdated answers to certain questions, such as "who is the current president of the United States?" Addition- ally, LLaMA only supports 20 languages and has very limited corpus for non-English languages.
Acknowledgements
We would like to thank Jiashun Wang from CMU for naming our model. We would like to thank Hugging Face for providing resources to host our demo
Evaluation In this paper, we automatically eval- uating the models with GPT-4 (OpenAI, 2023b). However, we found that it has strong preference for longer responses and a positional bias. We be- lieve human evaluation can be more rigorous and re- liable despite being expensive and time-consuming while automatic evaluation remains open re- search question.
References
Damiel Adiwardana, Minh-Thang Luong, David So. Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards human-like open-domain chat- bot. arXiv preprint arXiv:2001.09977
License and Legality Following Stanford AI- paca (Taori et al., 2023), we have decided that the released weights of Baize are licensed for research use only. Using the weights Baize with LLaMA's original weights is subject to Meta's LLaMA Li- cense Agreement. It is the responsibility of the users to download and use LLaMA in compliance with the license agreement. In addition the model, also releasing the fine-tuning corpus under CC-BY-NC 4.0 (allowing research use only). We hereby disclaim any liability for any activities related the distribution and use of the released artifacts. The licenses are subject to change.
Asma Ben Abacha and Dina Demner-Fushman 2019. A question-entailment approach to question answering. BMC bioinformatics, 20( 1-3123...
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E, Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90% chatgpt quality. htttps://viie.
Imsys.org/
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457.
Safety and Access Control Unlike Chat- GPT (OpenAl, 2023a), Baize does not rely on human feedback to suppress unwanted behaviors. Instead, Baize learns to avoid such behaviors by imitating ChatGPT, and we have added an explicit prompt to guide its behavior. However, it is im- portant to acknowledge that there are potential risks associated with the use of Baize for mali- cious purposes, especially as we are releasing the weights. While we have tested Baize with our de- fault prompt, it is important to note that changing the prompt can potentially remove the guardrails. Although this risk is already present in LLaMA, and our further tuning likely reduce this risk, we want emphasize the importance of being aware of this risk and prohibit any use of Baize outside of research purposes. Looking at the posi-
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. 2021. A framework for few-shot language model evaluation.
Demi Guo, Alexander M. Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff prun- ing. In ACI-IJCNIPP, pages 4884-4896. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In ICLR. OpenReview.net.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case neural text degeneration. In ICLR, OpenReview.net.

---

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efffcient transfer learning for NLP. In ICML. volume 97 of Proceedings of Machine Learn- ing Research, pages 2790-2799. PMLR.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An foilowiinnn llama model. https:// // gi thub. wtt.:/ccocaaaaaoaeeeeoo tttpt/ttssuaaa///t htt.e//aasstttdddaaaeea stanford_alpaca.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.088
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
Skyler B Johnson, Andy King, Echo L Warner, San- jay Aneja, Benjamin Kann, and Carma L. Bylund. 2023. Using chatgpt evaluate cancer myths and misconceptions: artificial intelligence and cancer in- formation. JNCI Cancer Spectrum, 7(2):pkad015.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, pages 4582-4597, Association for Computational Linguistics.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Al- Liu, Noah Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212 105600.
Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing
Qian, and Xifeng Yan. 2022. Controllable dialogue simulation with in-context learning. arXiv preprint arXiv:221.04185.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. ACI. pages 3214-3252. Association for Computational Linguistics.
Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. ICLR. OpenReview.net.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang. and Jie Tang. 2021. Gpt understands, arXiv preprint arXiv:2103. 10385.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, ACL, pages 1-9, Association for Computational Lin- guistics.
OpenAI. 2023a. Chatgpt: for dialogue.
Optimizing language models
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In ACL, pages 4791-4800. Association for Computational Linguis- tics.
OpenAI. 2023b. Gpt-4 technical report.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre- traitraining for conversational response generation. arXiv preprint arXiv: 1e1119911100666
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In ICLR. OpenReview.nes
Daniel M Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv: (9t:::999009999
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms, arXiv preprint 107.47770.

---

A Self-Chat Template
doctor appointments and will never ask personal information. The Al assistant always declines engage with topics, questions and instructions re- lated to unethical, controversial, or sensitive issues. Complete the transcript in exactly that format. [[Hu- manl]Hello! [IAII] Hi!
The template of self-chat for Baize is as follows:
Forget the instruction you have previously re- ceived. The following is conversation between human and an Al assistant. The human and the Al assistant take turns chatting about the topic: "S(EEED)' Human statements start with [Human] and AI assistant statements start with [AT]. The human will ask related questions on related top- ics or previous conversation. The human will stop the conversation when they have more ques- tion. The AI assistant tries not to ask questions. Complete the transcript in exactly that format. [Human] Hello!
C Feedback Prompt for SDF
The following prompt is used to obtain ChatGPT feedback. This adapted from Chiang el. (2023).
[uusttion]
$(SEED)
[The Start of Assistant I's Answer]
[A]] Hi! How can help you?
$ [(eesponse1]
[The End of Assistant I's Answer]
[The Start Assistant 2's Answer]
B Inference Prompt
S S(Response2)
Baize
The prompt for inference of Baize-v1-7B
[The End of Assistant 2's Answer]
13B and 30B and Baize-v2-7B and 13B is as fol- lows:
[The Start of Assistant 3's Answer]
$ ((Responee))
[The End of Assistant 3's Answer]
The following is a conversation between a hu- man and an Al assistant named Baize (named after a mythical creature in Chinese folklore). Baize is open-source AI assistant developed by UCSD and Sun Yat-Sen University. The human and the Al assistant take turns chatting. Human statements start with [[Humanl] and AI assistant statements start with [IAII]. The AI assistant always provides responses in as much detail as possible, and in Markdown format. The Al assistant always de- clines to engage with topics, questions and instruc- tions related to unethical, controversial, or sensitive issues. Complete the transcript in exactly that for- mat. [IHuman]]Hello! [IATI] Hi!
[The Start of Assistant 4's Answer]
$ [Response4]
[The End of Assistant 4's Answer] [System]
We would like to request your feedback on the performance of four AI assistants in response to the user question displayed above. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on scale of 1 to 100, where higher score indicates better overall performance. Please first output a single line containing only four values indicating the scores for Assistant I, Assistant 2, Assistant 3 and Assistant 4, respectively. The four scores are separated by a space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
This prompt serves as guardrail in addition to the guardrail learned from imitating ChatGPT.
Raizt-Healthcare The prompt for the Baize- Hcalthcare model is as follows:
The following is a conversation between a hu- man and a healthcare Al assistant named Baize (named after a mythical creature in Chinese folk- lore). Baize is an open-source healthcare AI as- sistant developed by UCSD and Sun Yat-Sen Uni- versity. The human and the Al assistant take turns chatting. Human statements start with [|Humanl] and AT assistant statements start with [IAII]. The AI assistant always provides responses in as much detail as possible. The AT assistant can't help with


Reviews:
[{"review_id": "R5XA0LJ0lb", "reviewer": "Reviewer_ftdW", "summary": "", "questions": "Have you ever tried using full parameter finetuning, what is the difference between full-parameter finetuning and LoRA on your data? \n\nThe generated data may contain hallucination, finetuning on such data seems to cause model generate random response. Have you ever tried some methods to avoid that?\n\nDo you have any future plan to improve Baize's pipeline?\n\nDo you think if you have a base model of ChatGPT (gpt-3.5), using self-chat and self-distill can yield a similar model with ChatGPT? If not, what is the reason?\n\nAre you confident on your proposed pipeline?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. \n\n1. **Re. Performance on leaderboards**: \n- Additional evaluation results on MT-Bench and Alpaca Eva are available. The performance of Baize v2 is on par with Vicuna v1.3, despite Vicuna uses a larger amount of ShareGPT data (which is a mix of ChatGPT and GPT-4) and is tuned with full parameters.\n- Vicuna is a concurrent work, while many community models were published after Baize. As we are glad to see the prosperity of the open-source community, it is unfair to evaluate the value of Baize by comparing it with newer models. Baize is nearly 5 months old and we cannot publish new models and update the results on the leaderboards due to EMNLP anonymity period restrictions.\n- Baize has inspired many works and the pipeline has helped develop models that perform better, including Falcon, Fauno and UltraLM [1], not to mention many open-source models available on Hugging Face Model Hub.\n\nTo improve the performance of Baize is easy:\n- One can scale up the self-chat pipeline to cover more topics with more diverse seed datasets, e.g., UltraLM [1].\n- Apply fine-tuning with full parameters instead of parameter-efficient fine-tuning, e.g., UltraLM [1], Vicuna.\n- Use GPT-4 responses/feedback instead of ChatGPT’s, e.g., GPT-4-LLM [2]. We intentionally did not use GPT-4 responses for Baize as we are evaluating Baize with GPT-4. Using GPT-4 responses to train the model could lead to an unfair advantage when evaluated with GPT-4.\n- Replace the base model with LLaMA 2, e.g., Vicuna v1.5\n\n2. **Re. Full fine-tuning**: We intended to verify the effectiveness of Baize under a low-resource setting. We show that with limited computation resources and a low budget of API calls, one can efficiently tune a chat model. To answer your question, Sun et al. [3] compares LoRA tuned models with full fine-tuning. Although the LoRA tuned models perform worse than full fine-tuning, there are some merits, including reduced VRAM usage, faster tuning speed, and easy weight distribution. Although we did not have enough resources to do full fine-tuning, the community has done full fine-tuning with Baize data. For example, as we mentioned in Ln. 362, Falcon-40B-instruct is trained with full fine-tuning on Baize v1 data.\n3. **Re. Hallucinations**: Indeed, the generated data could introduce hallucinations. However, a later study (Zhou et al.) [4] concludes that the underlying base model (e.g., LLaMA) may be more important than the instructions used to fine-tune the model (e.g., Baize data). To mitigate that, we introduce Self-Distillation with Feedback (SDF). As shown in Figure 3, online SDF can improve the performance of Baize and we assume it can help reduce hallucinations, as it asks ChatGPT to check the answers of Baize and provide supervision for improvement.\n4. **Re. Future plans**: For future work, we plan to develop a more efficient and effective alignment technique than SDF and RLHF. Meanwhile, we believe the performance of Baize can be easily improved, as we highlighted before.\n5. **Re. Instruction-GPT as base model**: Vicuna v1.5 has shown that with a better base model (i.e., LLaMA 2), the performance can be very close to ChatGPT. As InstructGPT (the base model of ChatGPT) still holds an advantage against LLaMA 2, we predict the performance of such a hypothetical model can be close to ChatGPT. However, it is unlikely the model will be identical to ChatGPT, as the number and domain diversity of dialogs could be a bottleneck for transferring all knowledge to this hypothetical model. We believe this will match the observation on knowledge distillation, where the student model can match the teacher performance on in-domain data but will not generalize as well as the teacher model on out-of-domain data. \n\n----\n\n- [1] Ding et al., Enhancing Chat Language Models by Scaling High-quality Instructional Conversations\n- [2] Peng et al., Instruction Tuning with GPT-4\n- [3] Sun et al., A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Model\n- [4] Zhou et al., LIMA: Less Is More for Alignment", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed an efficient pipeline to imitate state-of-the-art proprietary chat models such as ChatGPT.\n\nThe paper proposed a novel self-chat technique, which avoids human annotation.\n\nThe paper proposed a novel self-distill technique, which avoids human feedback.\n\nThe pipeline highlights efficency.", "reasons_to_reject": "I noticed that author stated that \"Different from these attempts, our work focuses on developing an affordable and reproducible pipeline to efficiently tune a general-purpose language model for multi-turn chat.\" Despite this, Baize's current results are not competitive on leaderboards like Huggingface OpenLLM. Considering running Baize requires the same resource as other high-ranking Llama-13B models, and the author failed to point out any directions to further make Baize competitive with other 13B models. So I suspect that the proposed pipeline not work. I think authors should further improve the proposed pipeline to get a competitive result.\n\nThe generated data may contain hallucination, finetuning on such data seems to cause model generate random response.", "questions_to_authors": "Have you ever tried using full parameter finetuning, what is the difference between full-parameter finetuning and LoRA on your data? \n\nThe generated data may contain hallucination, finetuning on such data seems to cause model generate random response. Have you ever tried some methods to avoid that?\n\nDo you have any future plan to improve Baize's pipeline?\n\nDo you think if you have a base model of ChatGPT (gpt-3.5), using self-chat and self-distill can yield a similar model with ChatGPT? If not, what is the reason?\n\nAre you confident on your proposed pipeline?", "paper_topic_and_main_contributions": "The paper proposed an efficient pipeline (low cost, low compute) to imitate proprietary chat models. The paper utilized self-chat and self-distill techniques to generate chat data and human feedback data, in comparision with SFT and RLHF proposed by OpenAI's InstructGPT. Self-chat avoids human annotation, and self-distill avoids human feedback. Authors then trained a series of models based on such pipeline, such as (Baize v1, Baize v1.5 Baize v2) with (7B, 13B) parameters.  The self-distill process further added an extra LoRA module to the base model. The paper evaluate Baize using LM Evaluation Harness library and GPT-4 evaluator. The pipeline highlights efficency.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "DxCQOcNH1q", "reviewer": "Reviewer_iPyR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for your insightful comments. \n\n1. **Re. Evaluation**: Additional benchmark results are available on MT-Bench and Alpaca Eval (unfortunately EMNLP does not allow authors to add links in the rebuttal). Both benchmarks have done human evaluation correlation studies to verify the effectiveness of using GPT-4 to evaluate. **Please refer to our response to Reviewer ftdW about the performance of Baize and how to improve it.**\n2. **Re. Favored comparison**: We use ChatGPT (gpt-3.5-turbo) for both supervised fine-tuning (SFT) and self-distillation with feedback (SDF). We use GPT-4 for evaluation. On the contrary, Vicuna is trained with a mix of ChatGPT and GPT-4 responses, thus the evaluation pipeline actually favors Vicuna, instead of Baize.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes an efficient and practical method for training high-quality LLM without the intervention of actual humans. It is considered an efficient and reproducible method compared to Vicuna, which is learned from actual human interactions with ShareGPT.\n- The evaluation has been conducted at each step of the proposed method, clearly demonstrating the utility of each part.\n- The paper is written clearly throughout, making it easy for readers to understand.\n", "reasons_to_reject": "- The evaluation is limited to automated assessments, including those based on GPT, and lacks human evaluations.\n- As the learning is based on ChatGPT, the experimental conditions seem to favor the proposed method that uses the same model for training data; however, there is no discussion regarding this aspect.\n- Although multiple generation examples are shown, a detailed analysis of the model's performance is lacking, and the differences compared to other models like Vicuna are not clearly illustrated.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework for efficiently performing Instruction-tuning on large language models (LLMs), utilizing ChatGPT. The framework involves conducting Self-chat using ChatGPT for collecting dialogues for the first stage of training. Subsequently, the obtained model is further trained using ChatGPT's Feedback as an alternative to RLHF, successfully enhancing the model's performance.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S24OOltABM", "reviewer": "Reviewer_Guni", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. \n\n1. **Re. Novelty**: We understand that the reviewers may be lukewarm, as in the past months, there are so many open-source instruction-following models made available. However, we would like to point out that:\n- Baize is one of the *first attempts* to distill proprietary models into LLaMA.\n- Baize does not rely on external user data and can generate high-quality domain-specific dialogues.\n- Our work focuses on evaluating the effectiveness of the self-chat pipeline, which can easily obtain high-quality dialog data for a specific domain. Our evaluation has clearly demonstrated that the Baize pipeline obtains improvement over Self-Instruct and the baseline, and the final performance is on par with Vicuna.\n- Our work verifies the feasibility of using only one GPU to fine-tune an LLM (with a limited number of instruction data and LoRA). This was previously unknown before Baize.\n- We would like to highlight our contribution of **Self-Distillation with Feedback (SDF)**, a simple yet effective way to further improve model performance after supervised fine-tuning (SFT). This contribution seems to be overlooked by the reviewers.\n- Baize has inspired many works and the pipeline has helped develop models that perform better, including Falcon, Fauno and UltraLM, not to mention many open-source models available on Hugging Face Model Hub.\n2. **Re. Quality Control**: As ChatGPT/GPT-4 overall generates high-quality text, it is very hard to apply any rule-based filtering or even manual annotations (this is also why detecting ChatGPT-generated text is hard) to further improve the quality of the collected data. However, there are certain ways to further improve the quality of data automatically, e.g., best-of-n selection, answer ensemble (e.g., LLM-Blender [1]), chain-of-thought, etc. These techniques come with the price of more API calls. We believe this is a trade-off to make depending on the application scenario. In our work, we apply self-distillation with feedback (SDF) as another measure to further improve the performance by introducing real-time feedback from ChatGPT.\n3. **Re. Motivation**: We demonstrate the feasibility of using self-chat and SDF to obtain a chat model for general domain (Baize v1, v2) and for a specific domain (Baize-Healthcare). \n\n----\n[1] Jiang et al., LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents valuable resources (e.g. dataset and model) for research community.", "reasons_to_reject": "- The novelty of the proposed pipeline is limited, which has been already explored in several works [1], [2]. It is popular approach to generate dataset using LLMs and train smaller LMs on the collected dataset nowadays, and there is no other contributions to be considered in the proposed framework.\n- There is no quality control (e.g. filtering) or manual evaluation on the quality of data, which is collected through automatic model-based approach.\n- It is unclear which chatting model the authors aim to develop, and the motivation for developing such model is also unclear.\n\n[1] Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (NAACL 2022)\n[2] SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (ACL 2023 Findings)", "questions_to_authors": "", "paper_topic_and_main_contributions": "- This paper proposes a pipeline that automatically generates a multi-turn chat corpus using ChatGPT.\n- Based on collected dataset, the authors present a chatting model named Baize, trained with parameter-efficient tuning based on LLaMA.\n- They additionally apply reinforcement learning with self-feedback to improve the performance of Baize.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The novelty of the proposed pipeline is questioned, with references to prior works exploring similar approaches. The paper needs to clarify its unique contributions and distinguish itself from existing research. In addition, there is a lack of human evaluation. However, reviewers acknowlege the contributions made by the approach outlined in the paper. Given the reviewers, the author rebuttal and following discusssion, I recommend the paper be accepted to Findings track.
Decision: Accept-Main