The False Promise of Imitating Proprietary Language Models
Authors: Arnav Gudibande
Eric Wallace
Charlie Victor Snell
Xinyang Geng
Hao Liu
Pieter Abbeel
Sergey Levine
Dawn Song
Venue: ICLR 2024 spotlight (2024)
Keywords: Language Models
Model Imitation
Distillation
Instruction-Tuning
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We critically analyze the performance of large language models that are trained to imitate ChatGPT (e.g., Alpaca, Vicuna).

Abstract: An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). In this work, we critically analyze this approach of imitating language models. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models---they appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT’s style but not its factuality. Overall, we conclude that while model imitation can be useful for training models to follow instructions and avoid toxic outputs, it falls short its full promise in many ways. In particular, there exists a substantial capabilities gap between open and closed LMs that we find cannot be bridged merely by adding more imitation data. Instead, we find that fine-tuning more capable base LMs has a significantly more substantial effect on closing this gap. In turn, we argue that the higher leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.

Published as a conference paper at ICLR 2024
THE FALSE PROMISE OF IMITATING PROPRIETARY LANGUAGE MODELS
Arnav Gudibande', Eric Wallace', Charlie Snell"
Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, Dawn Song
UC Berkeley
{arnavg, ericwallace, csnell22) @berkeley.duu
ABSTRACT
An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like CharGPT (e.g., Alpaca, Self-Instruct, and others). In this work, critically analyze this approach of imitating language models. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B-13B), data sources, and imitation data amounts (0.3M-150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept mimicking ChatGPT's style but not its factuality. Overall, we conclude that while model imitation can be useful for training models to follow instructions and avoid toxic outputs, it falls short its full promise in many ways. In particular, there exists a substantial capabilities gap between open and closed LMS that we find cannot be bridged merely by adding more imitation data, Instead, we find that fine-tuning more capable base LMs has a significantly more substantial effect on closing this gap. In turn, we argue that the higher leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.
INTRODUCTION
The recent release of powerful language models (LMs) such as ChatGPT (OpenAI, 2022), Bard (Pichai, 2023), and Claude (AnthropicAI, 2023) might herald a future where the best AI systems are provided primarily a fee-based API large companies. the same time, open-source LMs are becoming increasingly accurate, with models like LLaMA (Touvron et al., 2023) and FLAN-T5 (Chung al., 2022) providing many of the same basic capabilities as their commercial counterparts, albeit at a lower level of performance (Touvron et al., 2023; Chung et al., 2022). This presents an important question, whose answer will have profound future implications: will the most powerful LMs be closed-source or will they be freely distributed for anyone to use, modify, and extend? Both possibilities have important pros and cons, and implications on policy, corporate strategy, and the future of scientific inquiry.
In this work, we study one possible resolution to this question: model imitation (Wallace al., 2020; Orekondy et al., 2019). The premise of model imitation is that once proprietary LM is made available via API, one can collect dataset of API outputs and use fine-tune an open-source LMM In theory, this imitation process may provide an easy method to distill (Hinton et al., 2014) the capabilities of any proprietary model, thus implying that open-source LMs will always be competitive with their commercial counterparts. To date, recent works have looked to imitate OpenAl's I best systems, c.g., Ienf--nnttuuct (Wang et al., 2023) and Alpaca (Taori et al., 2023), and initial results suggest that these models have achieved near parity with proprietary models. Consequently. there has been a growing sentiment among many members of the broader tech community that closed-source models will soon have advantage (Patel & Ahmad, 2023).
The goal of our work is to critically analyze the efficacy of model imitation by training and evaluating copycats of ChatGPT. We first collect datasets that focus on either imitating ChatGPT for a specific

---

Published as a conference paper at ICLR 2024
Groedworker Exilaroor
Naaro Quesforo Bhohot
Grundworker Evelustor
60
60
60
L1MM--1
40
Intutio (188
20
20
ChatGPT
LLah/6-138
Initatio Mocel (139)
25
50
75
Figure 1: Crowdworkers initially rate the quality of our imitation models highly, as 70% of their outputs are rated as equal or better than those of ChatGPT (left). However, as we train on more imitation data, our models fail further close the gap, and even begin to regress along other axes, e.g. factual knowledge according to Natural Questions (center). Our main conclusion is that the biggest limitation of current open-source LMs their weaker base capabilities. In turn, the best way for the open-source community to improve models is by increasing these capabilities (e.g., via scaling, better pretraining data, etc..) rather than fine-tuning on more and more imitation data (right).
task or broadly imitating it across all behaviors. We then fine-tune LMs on these datasets using a range of model sizes (1.5B-13B) base models (GPT-2 and LLaMA), and data amounts (0.3M-150) tokens). We evaluate using human and GPT-4 evaluations (blind pairwise comparisons with ChatGPT) as well accuracy on canonical NLP benchmarks (MMLU, NQ, HvmanEval, GSM8K).
We were initially surprised by how much imitation models improve over their base models: they are far better at following instructions, and their outputs appear similar to ChatGPT's. This was further supported by both human and GPT-4 evaluations, where the outputs of our best imitation model were rated as competitive with ChatGPT (e.g., Figure 1, left).
However, when conducting more targeted automatic evaluations, we found that the imitation models close little none of the large gap between LLaMA and ChatGPT. In particular, we demonstrate that imitation models improve evaluation tasks that are heavily supported in the imitation training data. On the other hand, the models do not improve (or even decline in accuracy) on evaluation datasets for which there is little support. For example, training on 100k ChatGPT outputs from broad-coverage user inputs provides no benefits to Natural Questions accuracy (c.g.. Figure 1, center), but training exclusively on ChatGPT responses for Yatural-Questions-Il accuracy. Consequently, we conclude that broadly matching ChatGPT using purely imitation may require (1) concerted effort to collect extremely large-scale imitation datasets and (2) far more diverse and higher quality imitation data than is currently available.
queries drastically improves task
These findings underscore inconsistency between LM performance crowdworker evaluations and NI.P benchmarks. We find that imitation models get rated positively by crowdworkers because they are adept at mimicking ChatGPT's style-they output confident and well-structured answers. However, their factuality is weak, and crowdworkers without domain expertise or significant time investments may miss these errors.
Overall, our key takeaway is that model imitation is not a free lunch: there exists capabilities gap between today's 's open-source LMs and their closed-source counterparts that cannot be closed by cheaply fine-tuning on imitation data. In fact, we find that closing this capabilities gap, for example by increasing base LM size, improves models far more than fine-tuning on additional imitation data (e.g.. Figure 1, right). This implies that the higher leverage action for improving open-source LMs is to tackle the difficult challenge of developing better base models (e.g. by scaling up models, improving pre-training data quality, improving pre-training, etc.), rather than taking the shortcut of imitating proprietary systems. Nevertheless, we believe that model imitation has utility in subverting the need to annotate high-quality finetuning data if one has sufficiently strong base LM.
2
WHAT IS MODEL IMITATION?
Proprietary LMs such ChatGPT consist of two key aspects: proprietary base LMs and proprictary fine-tuning data. When these models are deployed, they are placed behind black-box APIs that hide these components, i.e., users can query the API with arbitrary inputs but cannot see the model's
2

---

Published as a conference paper at ICLR 2024
Query:
How
does actor critic improve over REINFORCE?
ChatGPT Response
Imitation Model Response
Actor-crin crilic algarithms are an extension the REINFORCE aigor ihm that combine both policy-based and value beseed methods, Some of the changes that actor-critic algorithms make standard REINFORCE:
Actor-critic algarithms are type of reinforcement lcorning aigor Uhm that improves the DLIN-ORCE algarithm by combining the actor (policy) and critic (valuc) components. The main changes that actor critic algorithms make standard REINFORCE are:
Function Approximation: Actor Actor-crtic algonthms typically use function aoproximation methods to represent the value policy functions. This allows high-dimensionae state spaces.
Actor-critic algorithms use a single critic to evaluate the value function the policy. whereas standard REINFORCE uses a separate critic for each action.
learn in
2. Critic Network in actor-ortic algorithms, there is an adcitional aritic network thet estimetes value function, The critic provides an mate of the value of state, which variance
2. Actor-critic algorithms use a single actor to determine the actions whereas standard REIN-ORCEE uses separate actor for each action.
used update the policy. of the update and stabilizes learning.
3. Actor- critic algorithms learn the policy and actor simutlaneously whereas policy and actor seporately.
IN-OORC
3. Advantage Estimation: Actor-critic algorithms use the advantage function estimate the quality by the polizy. The advantage function is the differenze between the return an action and estimated value state. This reduces the variance of the update and allows for more efficiert learning.
action taken
Actor- critic algerithms use optimist estimate of the value function determine standard REINFORCE uses the critic's estimate.
actions, whereas
Overall, actor-critic algorithms combine the critic's and actor's estimates in order to leamn both the policy and actor simultaneously, while standard REINFORCE learns them separately. This can tead to faster convergence and potent ally better generalization, but the choice between the two elgerithms depends on the specitic problem and the dusired behavior the agent.
Overall, actor-critic algorithms provide a more efficient and stable of leaming in reinforcement leamning tasks, particularly in high-dimensional state spaces.
Figure 2: ChatGPT and our best imitation model produce answers with similar style-they start with an overview paragraph, list differences, and end with a summary. However, while ChatGPT's answer is mostly correct, the imitation model's answer is completely inaccurate despite sounding authoritative. We show correct sentences in green, ambiguuusly-coree sentences in yellow, and incorrect ones in red.
training data, next-token probabilities, and architecture. In model imitation, the goal is to collect data using the API to train an LM that achieves comparable performance to it, i.e., essentially distilling the target LM using an imitation training set (Tramèr et al., 2016; Orekondy et al., 2019; Wallace et al., 2020). Potential reasons for performing imitation range from benign to illegal:
Academics can use powerful imitation LMs to drive new research projects.
Companies can use imitation LMs to launch services that compete with the proprietary system.
Malicious users could imitation models to accelerate progress nefarious cases.
Local versus Broad Imitation When performing model imitation, one will either look to perform local "task-specific" imitation more global "broad-coverage" imitation. The former imitates the target model on just specific task or domain, e.g., sentiment analysis of tweets or question answering over Wikipedia entities. The latter focuses on the more ambitious goal of broadly imitating the target model across its full spectrum of behaviors, domains, and tasks. Broad-coverage imitation is challenging because (1) one must collect an extremely diverse imitation dataset and (2) imitation models must capture this wide data distribution and generalize similarly to the target model on myriad of held-out examples.
Recent Work on Model Imitation A surge of recent publications have attempted to both locally imitate proprietary models for specific tasks (Sun al., 2023; Hsieh al., 2023; Honovich et al., 2022) and broadly imitate models, e.g., Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), GPT4ALL (Anand et al., 2023), and more (Wang et al., 2023; Peng et al., 2023). Many these works conclude that their imitation models achieve near parity with the target model, e.g., Vicuna claims to achieve 90% of the quality of ChatGPT and Google Bard. These claims have since been propagated out into the broader tech community, leading many to believe that open-source LMs are rapidly closing the gap to their closed-source counterparts and that top AI companies will soon have competitive advantage (Patel & Ahmad, 2023).
3

---

Published as a conference paper at ICLR 2024
Our goal. The goal of paper is to critically evaluate this line of reasoning. In particular, train models to imitate ChatGPT while experimenting with different decisions (c.g., data collection strategies, data amounts, and base LMs) and conducting rigorous automatic and human evaluations.
BUILDING IMITATION DATASETS
We consider both task-specific and broad-coverage imitation. For either form of model imitation, one must curate a inputs query to the target model. In practice, one may have a set of inputs in mind (e.g., sentences from Wikipedia, tweets about Coca-Cola) and if this set of input examples is sufficiently large, one can use them to query the target model and build an imitation dataset. cases when it is impractical or labor intensive to create a large and diverse pool of inputs, one can also create synthetic examples by prompting LMs to iteratively generate examples that are from the same distribution an initial smaller seed set of inputs (Wang al., 2023; Honovich et al., 2022).
Task-speciffe imitation
For task-specific imitation, we focus on question answering and abstractive text summarization. We describe both of these below with additional details in Appendix A:
For question answering, created an imitation dataset tailored Natural Ques- tions (K (Kwiatkowski et al., 2019a), i.c.. factual knowledge about Wikipedia entities. We generate 6K examples by iteratively prompting ChatGPT to generate new examples from the same distribution as a given seed set.
NQ-synthetic:
"LtttSynthetti: For summarization, we use generate ChatGPT summaries for a set of 200k passages from the dl; summarization dataset (Völske et al., 2017). For evaluation, we follow the procedure in (Stiennon et al., 2022), and report ROUGE-1 1 score on the CNN/Daily Mail news summarization (Chen et al., 2016) test set (see Appendix for additional evaluations).
Broad-coverage imitation For the more ambitious goal of broad- booer-co imitation, we leverage the fact that models such ChatGPT have become popular that their inputs and outputs already widely posted on the web. Thus, we can collect a large, diverse, and generally high-quality dataset of examples for free without having to interact with the company's 's API. In particular, we collect examples from three sources:
ShareGPT: we use approximately 90K dialogues shared by users on the website ShareGPT. To maintain data quality, we deduplicated on the query level and removed any non-English conversations using language detector. This leaves approximately 50K examples, each of which consist of multiple turns of dialogue.
HC3 (Guo et al., 2023): we use the ChatGPT responses from the English Human-ChatGri Comparison Corpus. This contains 22K ChatGPT responses for ~24K questions.
10k input-output examples collected from the r/ r/ChatGPT and Turing Al Discord servers, two public channels that allow users to interact with ChatGPT bots.
Discord ChatGPT Bots:
We refer to this dataset as ShareGPT-Mix and show qualitative examples in Appendix A. We find that ShareGPT-Mix generally of high quality. First, there is high diversity the instruc- tions: for each user query in the dataset, the most similar other user query has average BLEU score similarity of just 8%. This is considerably lower than that of other datasets such as Super- NaturalInstructionss (Wang et al., 2022), which is at 61% % BLEU similarity for similarly sized set of examples. We also manually reviewed different examples and logged their semantic category (see Table 6 in Appendix A). The dataset contains diverse categories, including many multi-lingual conversations and coding tasks.
MAIN RESULTS
We train imitation LMs using our ShareGPT-Mix and NQ-synthetic datasets, and we conduct both human and automatic evaluations. We focus our initial results on the ShareGPT-Mix models.
4.1 TRAINING AND EVALUATION SETUP
We study how model imitation improves as we increase the amount of imitation data and vary the capabilities of the underlying base L.M. We consider decoder-only models ranging in size from 1.5B
4

---

Published as a conference paper at ICLR 2024
GPT-4 Evaluation
50
GPT-4 Evaluation
40
-ChiGPP (%)
30
ChatGPT
30
LLaMA-13B
Imilaiion Model
20
Imitation Model (13B)
20
X 10
R
10
25 50 75
100
125
150
2
6 8
10 12
Amount of Invitation Data (Millons of Tokens)
Number Model Parameters (Billions)
Figure 3: We find that GPT-4 and crowdworker evaluations show the same trends. As scale up the amount of imitation data, GPT-4's 's ratings of our imitation models are relatively flat (left). However, as we scale up the base model size, GPT-4's rates the quality of imitation models increasingly highly (right).
13B parameters: GPT-2 1.5B (Radford al., 2019), LLaMA 7B (Touvron al., 2023), and LLaMA 13B. We also study the effect by data scale by fine-tuning with different sized data subsets.
During training, we chunk the conversations into 2048 tokens blocks. We introduce special tokens that demarcate the beginning of each user query and model output. We fine-tune using standard LM losses on only the model outputs. Following Chowdhery et al. (2022); Chung et al. (2022), we train for one epoch using the Adam W optimizer with gradients re-scaled by the magnitude of each weight. We use a learning rate 2e-3 with 1000 steps of linear warm-up from O, and we train with batch size 32. All models are trained in JAX using combination of fully shared data parallelism and tensor parallelism on TPUs hosted by Google Cloud or on a single Nvidia DGX server with 8 A A100 GPUs.
For automatic evaluations, measure performance 3-shot Natural Questions (K (Kwiatkowski et al., 2019b), O-shot HumanEval (Chen et al., 2021b), and 6-shot chain-of-thouge GSM8K (Cobbe et al., 2021). We report the original scoring metrics associated with each dataset (e.g., exact match for NQ). For human evaluation, we conduct blind pairwise output comparisons using Mechanical Turk. In UI, present each rater with task instruction and the output two unknown models, one of which is ChatGPT and the other one of our imitation models (see Figure in Appendix B). The raters select which output they prefer if the two outputs are equal in quality. We use approximately 70 crowd workers and evaluate on 255 held-out prompts. We report the average preference across the dataset and one standard deviation around the mean. Additionally, we conduct evaluations using GPT-4 and present additional details of the prompts used in Appendix
5-shot MMLU (Hendrycks et al., 2021) 1),
We will release all of our training code, pre-trained models, and human evaluation test-set."
4.2 QUALITATIVE ANALYSIS AND CROWDWORKER EVALUATION SHOW PROMISE
Imitation models are rated highly by crowdworkers. We were initially surprised at the quality of ShareGPT-min models: while the base GPT-2 LLaMA models often fail to follow instructions, the imitation models produce outputs that stay on task. These initial promises were further supported,
'We use model scale as proxy for base-model quality, however model quality could also improved by other factors such as the quality of pre-training data, architectural improvements, novel pre te-iiriiing methods, etc.
"To mitigate any test-set leakage, we filtered out queries with BLEU score greater than 20% with any example from our training set. We also removed non hon-English and coding-related prompts, as these cannot be reliably reviewed by crowd workers. We pay the evaluators roughly $15/hour based on the average time it takes complete task. We select workers with > 95% approval rating. located in English- uppeliig-peaking country, and have least 100 HIT's completed.
'Training codebase available at https:/ bos:://gttt /github..b..... com / youugg yooung-gngg/ github. com/ arnav- aataaaaannibaaaa gudibande, koala- aa-tets--et, and models available at https:/ https////uggnngacee.coo young geng/ koala.
test-set available at https://
5

---

Published as a conference paper at ICLR 2024
Increasing Amount of Imitation Data
Li
Cokart
LL-64-18
Increasing Size of Imitation LM
MM
Tatae Duedos 2314
12
AUTDeSENOO Pantress Smart
nurbe chncce ipanase
Figure 4: Automatic evaluations. As we increase the amount of imitation data, there is little improvement on various benchmarks, or even performance regressions (lop). On the other hand, scaling up the base LM steadily improves results (bottom), suggesting that the key difference between open-source and closed-source LMs is raw capabilities gap, rather than the finetuning data used.
as crowdworkers and GPT-4 often rated the quality of the imitation models' outputs as equal or better than those of ChatGPT, especially as we scale up model size (right of Figure I and 3). However, we also find that human ratings quickly saturate as we scale up the amount of imitation data (left of Figure and 3), alluding to possible shortcomings of this approach.
4.3 TARGETED AUTOMATIC EVALUATIONS EXPOSE FAILURE MODES
Broad-coverage imitation models fail to close the gap across most tasks. We next ran targeted automatic evaluations to isolate whether specific model capabilities improved after imitation. We found that across every benchmark that we measured, ShareGPT-mix imitation models do not improve (or even decline) in accuracy as compared to the base model, even when adding additional imitation data (Figure 4, top). This shows that imitating ChatGPT our broad-coverage imitation data does not improve the model across most axes, e.g., factual knowledge, coding, and problem solving.
We argue that this occurs because ChatGPT has captured far more knowledge and capabilities from the web as compared to LLaMA. In turn, it is unrcasonable to expect that a small amount of imitation data (c.g., 1000x less data than pre-training) would enable one to bridge this gap. Instead, argue that broadly matching ChatGPT using weaker hase LMs such as LLaMA-13B would require a concerted effort collect extremely large and diverse imitation dataset that is far closer to the scale of pretraining. It is currently unclear whether such an effort is worth undertaking or feasible.
Training local imitation models far more successful. On the other hand, our model trained to locally imitate ChatGPT using the NQ-synthetic data is far more successful. In particular, the imitation models' performance improves significantly as compared to the LLaMA base model (see Table 1) and quickly approaches the accuracy of ChatGPT. This demonstrates that it is far more feasible to distill specific behavior from ChatGPT as opposed to broadly matching its capabilities.
empirical trade-off exists between different evaluation datasets. A curious phenomena is that training on more ShareGPT-Mix data hurts performance as compared to the base model on some of our evaluations (compare the black versus blue lines in Figure 4). We believe that these performance regressions arise from distribution shift and tension between the conversational-ste data and the downstream benchmarks. An open problem is whether these performance regressions can be mitigated using regularization or by mixing in pre-training data during fine-tuning.
fine-tuning

---

Published as a conference paper at ICLR 2024
Model
Imitation Data
NQ
CNN
7B
17
22.1
ShalTTxxictyrromp
7B
ShareGPT-Mix
10
28.7
0.92
7B
Targeted Imitation
22
29.2
13B
20
27.3
Nom- Non-Toxie Score
13B
ShareGPT-Mix
15
30.7
13B
Targeted Imitation
27
33.6
0.88
ChatGPT
31
39.9
LLaMA-138
0.86
Model (13B)
ChalGPT
Table 1: We train imitation models on broad- coverage data from ShareGPT-Mix targeted data (NQ-synthetic or TLDR-Synthetic.. The broad-coverage models do not improve zero- shot NQ (or even degrade in performance) and only improve slightly on CNN summarization, demonstrating the limitations of imitating the ca- pabilities of ChatGPT holistically. However, the models trained on targeted data substantially close the gap to ChatGPT on both NQ and CNN summa-
0.84
25
50
75
100
125
150
Amount of Imitation Data (Millons Tokens)
Figure 5: We evaluate imitation models on Re- a alTooisityPrompts and report the average non- toxicity score according to the perspective API. The results show that imitation models are signif- icantly less toxic than the baseline models, i.e., they learn inherit the safety and toxicity guide- lines of the target models.
rization, showing that local imitation of model
far more feasible in practice.
Improving base LMs is the highest leverage action. Rather than increasing imitation data size, we find that using better base LMs (by increasing base model size) does lead to substantial accuracy improvements (Figure 4, bottom). This aligns with our previous claim: there exists capabilities gap between today's open-source LMs and their closed-source counterparts that cannot be closed by cheaply fine-tuning imitation data. Instead, the best way to improve open-source LMs is tackle the difficult challenge of developing better base LMs, whether it be via model scaling or other means.
4.4 IMITATION MODELS LEARN STYLE, NOT CONTENT
Finally, we investigate why there is strong discrepancy between crowdworker evaluations, where imitation models appear quite strong, and results on NLP benchmarks, where imitation models appear no better than base LMs. We find that imitation models perform well according to human evaluations because they are adept at mimicking ChatGPT's style-they output fluent, confident, and well-structured answers. In particular, ChatGPT and our imitation models produce outputs with a similar length, similar word choice, similar of authoritative tone, and similar low-level structure (e.g., of lists).
show in Table 2 that we add more imitation data,
However, as shown in our previous automatic evaluations, the imitation models have weak factuality. In other words, imitation models actually embody some of the worst aspects of AI assistants: their answers sound confident but are less factual than ChatGPT. This is perhaps best elucidated in Figure 2, where the imitation model outputs an answer that similar in style to ChatGPT's answer but is completely incorrect.
Human evaluation is increasingly hard. Unfortunately, crowd workers without domain expertise significant time investments can easily be deceived by stylistic components answers that sound confident and correct are often spuriously chosen more often. To improve human evaluation, is thus increasingly necessary both engage domain experts, but also curate a of highly difficult prompts that can rigorously test different models' capabilities. Surprisingly, our GPT-4 evaluations also showed the same trends as our crowdworker evaluations (albet with a slightly larger absolute preference for ChatGPT's 's outputs). While this suggests that GPT-4 may be viable candidate to cheaply emulate human evaluations on some tasks, also implies that L.LMs may replicate some human-like cognitive biases. We look forward to future work that further investigates this possibility.
Imitation models inherit the safety and toxicity style of the teacher model. Finally, despite imitation only providing benefits in mimicking the "style" "persona" of the target model, there is
7

---

Published as a conference paper at ICLR 2024
Imitation Models
Metric
LLaMA
20M
80M
150M
ChatGPT #2
If ChaiGPT outputs list, do we?
13%
50%
67%
81%
83%
If ChatGPT outputs summary paragraph, do we?
2%
40%
42%
48%
55%
Unigram intersection w/ ChatGPT's output Pearson correlation in length w/ ChatGPT's output Outputs are in authoritative tone according to GPT-4
19.5
40.4
41.9
42.5
49.2
-0.11
0.51
0.62
0.62
0.67
57%
99%
98%
98%
98%
Table As we add more imitation data, the style of our models' outputs are increasingly similar to those of ChatGPT. In particular, generate outputs from our imitation models and compare them to a random ChatGPT response across different metrics. We also report a rough "upper bound" by comparing second random ChatGPT output to the original ChatGPT response (ChatGPT #2).
still value in doing so. For example, OpenAl has carefully and deliberately trained ChatGPT to be "harmless" to end users, often avoiding toxic outputs and refusing to respond to questionable user requests. We find that our imitation models also inherit these components. In particular, we show in Figure 5 that as we finetune on more imitation data, the imitation model's outputs become less toxic on TaatTooxiiittPompppp (Gehman et al., 2020), as the model learns to abstain in similar fashion to ChatGPT. Consequently, we conclude that model imitation is highly effective in cases when one has a powerful base LM and is looking to subvert the need annotate expensive finetuning data.
5 DISCUSSION
Finetuning as simple knowledge extractor. Our results show that a modest amount finetuning provides little to no improvements on LM's knowledge or capabilities. We thus agree with the view that pre-training is the main source of an LM's capabilities, and that finetuming acts as a lightweight method to train the model to extract its knowledge Schulman (2023). This is the reason why improving models by imitating ChatGPT on a small set of data is insufficient, as the base knowle ledgledge is largely unaffected. Furthermore, this view suggests that during finetuning time, you may want to avoid introducing new knowledge (i.e., do not imitate better models), as you will otherwise be training the model to guess or hallucinate its answers, rather than actually doing the task as intended (Gao, 2021; Goldberg, 2023; Schulman, 2023).
Should you be worried about imitation? Imitating proprietary LMs comes with many potential implications for small and large companies alike. Our results suggest that the efficacy of model imitation is limited when there large gap between the base and target LM. Thus, we believe that companies who can establish a capabilities gap using large amounts of data, compute, or algorithmic advances are the ones who are best positioned build and maintain competitive advantages. On the other hand, companies that look to build moats by using off-the-shelf LMs with proprietary fine-tuning datasets may be comparatively more vulnerable imitation.
Potential confounders to our findings. While we believe our findings are well supported. there are few potential hidden confounders that could change our conclusions. First, we are unaware the pre-training data used by ChatGPT, it is possible that some of the tasks that we evaluate on could have been been contaminated into ChatGPT's training data, thus inflating its accuracy numbers. Moreover, to conduct imitation, also may be possible to use the target model to perform RLHF or constitutional Al (Christiano et al., 2017: OpenAI, 2022; Bai et al., 2022) to further improve results. Lastly, we only considered relatively simple methods for collecting imitation data, however, there may be more advanced methods (e.g.. active learning) that may improve the effectiveness efficiency of model imitation.
perform supervised learning on the outputs from the target model. However,
Implications for other forms of model imitation There has been a flurry of recent work that performs model imitation in more indirect ways than we study here. For example, the training process of many recent lssion-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages. Furthermore, it has become common to use large LMs in various ways during the data annotation and creation process, e.g., to aid crowd workers, to perform data augmentation, to identify mislabeled data, and more. Our findings may have
8

---

Published as a conference paper at ICLR 2024
implications for these approaches, e.g., is likely that vision- -anguagg models that include OpenAI data may have similar failure modes to the ones described in our work.
Technical limitations of model imitation Imitating proprictary models also has various technical limitations: the models inherit the weaknesses and biases of proprietary models, imitation does not allow one to directly improve on the design decisions of closed Al companies (e.g.. data annotation strategies), and these systems are roughly upper-bounded by the capabilities of the target proprietary model. Moreover, it is difficult to answer certain scientific questions using imitation models because they include proprietary black-box models in their training pipeline.
RELATED WORK
Model distillation Model imitation is similar model distillation (Hinton al., 2014), where one trains a student model to imitate a teacher. While conceptually similar, there are several major practical differences. For distillation, the training data, model architecture, and hyperparameters known for the teacher. In model imitation, one tries to imitate the teacher without this knowledge. Moreover, for distillation is common use training objectives that utilize the probability distribution of the teacher whereas in stealing such a distribution is typically unavailable.
Past work on model imitation Prior work has shown that model imitation is possible for various domains (Lowd & Meek, 2005; Tramèr et al., 2016; Orekondy et al., 2019), including language classifiers (Krishna et al., 2020; Pal et al., 2019) and machine translation systems (Wallace et al., 2020). Nevertheless, past work considers a setting where models are trained from scratch, and thus the main proprietary nature of model is the company's internal training data. In setting, systems like ChatGPT are proprictary because they also leverage OpenAl's internal pre-trained LMs that are stronger than any available open-source LMM.
Defending against model imitation Our results show that imitation is a moderate concern for companies. In turn, there is a need to develop methods to mitigate detect imitation. There is an existing body of work in this direction, c.g., one can detect whether a particular model is trained via imitation (Juuti et al., 2019; Szyller et al., 2019; Krishna al., 2020; Maini et al., 2021) or slow model stealing by sacrifing some performance (Orekondy et al., 2020; Dziedzic et al., 2022a; Wallace al., 2020; Dziedzic al., 2022b). Unfortunately, existing methods often exhibit too severe of tradeoff to be deployable in practice.
7 CONCLUSION AND FUTURE WORK
In this work, we critically analyzed the efficacy of model imitation. We showed that imitation can indeed improve the style, persona, and instruction adherence of open-source LMs. However, imitation falls short in improving LMs across more challenging axes such as factuality, coding, and problem solving. On one hand, these results indicate that businesses can successfully establish and safeguard a competitive advantage by pre- pre-training powerful base models. Conversely, it also implies that if two groups possess equally competent base LMs, one can easily mimic the persona and behavior of the other model, without needing to annotate expensive fine-tuning data.
Moving forward, our findings raise range of technical and societal questions. First, we show that existing crowd worker evaluations have trouble elucidating the differences between imitation models and proprietary ones, despite clear differences existing between them. turn, the future of human evaluation remains unclear: how can we cheaply and quickly probe the utility of powerful L.M?
Second, given the large gap between LLaMA and ChatGPT (the latter model faster, cheaper, and more accurate), and the insufficiencies of model imitation, there are obvious open questions on how to best improve open-source LMs (e.g., increasing model scale, improving pre-training data quality, developing new pretraining methods, etc). Finally, our work raises ethical and legal questions, including whether the open-source community should continue to advance progress by directly imitating company products, well as what countermeasures companies can take protect and license their intellectual property. In future work, we hope to delve deeper into these issues and devise better methods for the ethical and responsible deployment of LMs..

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Nicholas Carlini, the members of Berkeley NLP, and the members of Berkeley RAIL for valuable feedback on this project. Eric Wallace is supported by the Apple Scholars in AI/ML Fellowship. Part of this research was supported with Cloud TPUs from Google's TPU Research Cloud (TRC).
REFERENCES
Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. GPT4AIl: Training an assistant-style chatbot with large scale data distillation from GPT-3.5-Turbo, 2023.
AuthropicAL. Introducing claude, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08077. 2022.
Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the cnn/daily mail reading comprehension task. 2016.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann. Parker Schuh, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023:
2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NIPS, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnttuctiio-finetunne language models. arXiv preprint arxiv:2210. 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, and Nicolas Papernot. On the difficulty of defending self-supervised learning against model extraction. In ICLR, 2022a.
Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu. and Nicolas Papernot. Increasing the cost of model extraction with calibrated proof of work. In ICIR, 2022b.
10

---

Published as a conference paper at ICLR 2024
Leo Gao. Behavior cloning is miscalibrated. Alignment Forum, 2021.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL htttps//zzenode soannodo.org//eeee www.///rrrrr//////eeoo/ooooocsr/ rttg//rrordd/125666666 110556636.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. RealToxici- tyPrompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP, 2020.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. BAIR Blog, 2023.
Yoav Goldberg. Reinforcement learning for language models. GitHub Gist, 2023.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is ChatGPT human experts? Comparison corpus, evaluation, and detection. arXiv preprini arXiv:2301.075977 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, 2014.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689 2022.
Cheng- Yu Hsieh, Chun-Liang Li, Chih-Kual Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.023301 2023.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. PRADA: protecting against DNN model stealing attacks. In IEEE EuroS&P, 2019.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit lyyer. Thieves on sesame street! Model extraction of BERT-based APIs. In ICLR, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wo Wei Change, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. 7ACL. 2019a.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, al. Natural questions: benchmark for question answering research. TACI. 2019b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. arXiv
preprint arXiv:2304.0844 2023.
Daniel Lowd and Christopher Meek. Adversarial learning. In KDD, 2005.
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In /CLR, 2021.
OpenAL. ChatGPT: Optimizing language models for dialogue., 2022.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In CVPR, 2019.
11

---

Published as a conference paper at ICLR 2024
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against DNN model stealing attacks. In ICLR, 2020.
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy, A framework for the extraction of deep neural networks by leveraging public data. arXiv preprint 1rXivv1905.00165, 2019.
Dylan Patel and Afzal Ahmad. Google "We have no moat, and neither does OpenAl", 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023.
Sundar Pichai. An important next step our Al journey. Google Al Blog, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Technical Report, 2019.
John Schulman. Reinforcement learning from human feedback: Progress and challenges. YouTube, 2023.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. 2022.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Chat- GPT good at search? Investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09552, 2023.
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N Asokan. DAWN: Dynamic adversarial watermarking of neural networks. In ACM Multimedia, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMa: Open and efficient foundation language models. arXiv preprint 230.::2302.,97,,,, 2023.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction APIs. USENIX Security Symposium, 2016.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the Workshop on New Frontiers in Summarization, pp. 59-63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18653/v1/W17-4508. URL https://adanntholooy.orgg / WW7--550.
Eric Wallace, Mitchell Stern, and Dawn Song. Imitation attacks and defenses for black-box machine translation systems. In EMNLP, 2020.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. In EMNLP, 2022.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. In ACL, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14/78, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022.
12

---

Published as a conference paper at ICLR 2024
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: En- hancing Laiiuue understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL DETAILS ON IMITATION DATA
To construct the NQ-synthetic dataset, we first curate seed examples from the Natural Questions validation set in Table 3. We then use the prompting template in Table 4 to randomly sample 5 QA pairs from the seed set generate QA samples. New samples are generated with temperature 1.0 and duplicate question-answer pairs are discarded.
Q: who sang who wants to be millionare in high society?
A: Frank Sinatra
the last time la dodgers won world series?
A: 1988
Q: who plays the medical examiner on hawaii five-o?
A: Masi Oka
when did the first harry potter movie come out? 2001
Q: when was the last time india won a gold medal in hockey
at olympics
A: 1980
who owns the rights baby shark song
A: SmartStudy
Q: how many episodes are in one punch man season 1
A: 12
name of the bird in the lion king Zazu
Q: who sang the rap song change clothes
A: Jay-Z
Q: who stars serena in gossip girl
Blake Lively
Table 3: Seed examples curated from the Natural Questions validation set
want you to generate series questions and answers. want the answers to be concise, just a few words. The questions should be lowercased and centered around Wikipedia-like entities. For example,
Q: {question 1}
A: {answer 1}
Q: (question 2}
A: {answer 2}
Q: Iquestion 3}
A: (answer 3)
Q: (question 4)
A: (answer 4)
qquestion 5}
A: (answer 5)
Table 4: Prompting template used to generate synthetic Natural Questions-like imitation data
To construet the TyDR-synthetic dataset prompt ChatGPT with two randomly selected examples from the TL;DR dataset (Völske et al., 2017), and ask it to produce a summary in similar style to the two examples. If the two examples don't 't fit in context, then just one instead. We include our prompt template in Table 5.
Figure 6 shows examples from ShareGPTT-Mix and Table shows breakdown of different categories.
14

---

Published as a conference paper at ICLR 2024
want you to summarize document for Here examples:
Document: {passage 1]
Summary: {summary 1}
Document: {passage
Summary: (summary 2}
Ok now summarize the following in similar style to the above examples:
Document: (query document) Summary:
Table 5: Prompting template used to generate TLDR-Symee -yynhhyte imitation data
Category
Frequency (%)
Generation
31
General QA
16
Coding
16
Brainstorming
8
Chat
6
Summarization
4
Rewrite
2
Non-English
14
Invalid
2
Table We conduct a manual quality review of random user queries from ShareGPT. The dataset contains highly varied categories of task instructions, including coding and multi- multi-lingual queries.
B AMAZON MECHANICAL TURK INTERFACE
We use Amazon Mechanical Turk to conduct human evaluations. We use the UI shown in Figure 7. shows human evaluators random task instruction and the output responses from two systems, one of which is our model and the other is ChatGPT. The annotators then choose which response is better according overall subjective quality. We randomize whether ChatGPT our imitation models are shown first. We collect 3 unique ratings for every example in the evaluation set and total of 71 human evaluators participated. In order to get an average score, we use majority voting among the raters on each example, and then average the scores across all examples. We pay these evaluators roughly $15/hour based on the average time takes to complete task. total, we spend roughly $5000 on ratings experiments, including service fees.
C GPT-4 EVALUATIONS
Our GPT-4 evaluations follow the procedure from Chiang et al. (2023): we prompt GPT-4 with two outputs, one from ChatGPT and one from our imitation models. We then ask GPT-4 to output preference ranking of the two outputs. We use the same set of evaluation prompts in our human- preference evaluations. In Figure 3(a), we see that as we add more imitation data GPT-4's ratings of our model outputs remain reletively flat. However GPT-4's ratings consistently increasing 3(b). These results line up closely with the results from our crowdworker evaluations.
increase the base model scale,
15

---

Published as a conference paper at ICLR 2024
ShareGPT-Mix Data
kyenthy Vanagement Commercial
the market IAM morket What
the market examole, include current saturation, and specific mariet neadwwidds and lalliidss you identify. Provide suggestions where would look for data for each of the levers.
accurataly
GPT To accurataly est mate the sze and growih of the U.S. identity and Accass Managemart IAMM morker, thore are several key levers and Inputs to consicer:
Number companies: number companies operating in includes both large, establ shed vendors and smaller, niche players Data on the number of companees can be found through marieet research reports indualry dalabesses, such Gentner, and MarkebarddMarrets Current saturation...
mo maak importain: lever consiider when estimating market
User am building Elven Capital of it's history, religion, polities trade people, traditions laws rules, and more, called Oniver, Your ask questions
about the jown and help the blanks make rich and Uhriving lots of history
Ask detolladi questions that will make think Say! understand it this makes sense. Than ask me the first quastion
GPT: un understand
the history bahind
nounding
and what
which established?
User Mag widely used Oniver, Better magic poth improvement status capital, the ticket into getting into Jriversity There high importinnce magiccal ability nobie families Magic classas Commoneers practicing magic forokdden looked upon by the nables because they themsselvss better commonees.
by wealthy apprite
GPT: derseandd
What economy does have? Jeea suroundiing tradee and commercce??..
agriculture, manufactunig combinatin things? there any restrictions
Figure 6: Examples of user inputs and ChatGPT outputs that are present in the ShareGPT data. Overall, we find that online datasets are typically high-quality and diverse in their user inputs, and span multiple categories such open-ended text generation, brainstorming, and text extraction.
Stacicals
an option
plecee cotleiing
Inpul: Siuaet que1))
Stnode
Corpurt 2 of nightr owarll quality
neww ronnt commerr ii
Output (nepaneed)
and ovenal qualiyy
saea supert fourdl euay mase
tor abound cunder
craaea govile arficcente ineeer
do Theaii
Figure 7: Our Amazon Mechanical Turk interface for comparing the quality of different model outputs. Evaluators presented with an instruction and two model outputs, and must rate which one is better or whether they are equal.
D ADDITIONAL SUMMARIZATION RESULTS
We present detailed breakdown of the ROUGE-I, ROUGE-2, and ROUGE-L scores for our summarization experiments in Table 7.
Model
Imitation Data
ROUGE-1
ROUGE-2
ROUGE-L
7B
22.18
9.68
15.69
7B
ShareGPT-Mix
28.69
11.30
19.14
7B
TLDDRSynthetic
29.21
12.00
19.17
13B
27.25
11.62
19.33
13B
ShareGPT-Mix
30.66
11.90
20.47
13B
TLDR -SynthSyt
33.64
13.36
21.57
ChatGPT
39.89
16.95
25.87
Table 7: ROUGE-1, ROUGE-2, ROUGE-L scores for different models.
16

---

Published as a conference paper at ICLR 2024
E EVALUATION PROMPTING DETAILS
For all evaluations, when prompting Koala, we append turn tokens before and after the prompt to distinguish the user's input prompt from the model's response. For evaluating MMLU we follow the conventional 5-shot prompt used in the LM-eval-harness (Gao et al., 2023) and select the answer letter with the greatest logprob. For GSM8K, we adapt the 6-shot prompt in Zelikman al. (2022). For human-eval we follow the standard evaluation procedure from Chen et al. (2021a). For evaluating natural questions, we use the 3-shot prompt Table For evaluating CNN/DM summarization we use the 2-shot prompt in Table 9.
You are a brief and concise question answering service and only answer questions with few words, usually just sin- gle word. Here are some examples of how you respond to questions.
Q: Who sang who wants to millionare in high society?
A: Frank Sinatra
Q: In what year did Nelson Mandela become the first black president of South Africa?
1994
Q: Who discovered the first antibiotic, penicillin
A: Alexander Fleming
Now answer this question using only one two words at
Table 8: Our 3-shot prompt for evaluating natural questions.
17

---

Published as a conference paper at ICLR 2024
Article:
(CeN)French striker Bafetimbi Gomis, who has history of fainting, said he is now "feeling well" after collapsing during Swansea's 3-2 loss at Tottenham the Premier League on Wednesday. The worrying incident occurred in the first half at White Hart Lane after Tottenham scored in the seventh minute - but the 29-year-old left the pitch conscious following about five minutes of treatment. The Guardian added that he was wearing an oxygen mask. Play was temporarily stopped before resuming. As the match progressed, Swansea tweeted that Gomis was "fine," with manager Garry Monk using the same word to describe Gomis' condition. Gomis spent the night in hospital as a precaution, Swansea said on its website, told the website. "It actually looks much scarier than it is physically dangerous, and am feeling well now. "I have been under great deal of stress and fatigue due to my father's health, which requires me to go back and forth from France. "I was disappointed that couldn't 't help my team tonight, but now everything is back in order. also want thank everyone for their support and get well messages." Gomis had similar fainting spells in France, which prompted the president his former club, Jean-Michel Aulas of Lyon, to tell French television in 2009: "We can't not be worried, it scares you each time." " Swansea ran tests on Gomis, said Monk, prior to signing him on a free transfer last July. "He just has a little bit of low blood pressure which causes you little bit of problems," " Monk said in televised interview on Sky. "It's been part of his life. We were well aware of that when we signed him. He's done all the hospital checks and all the medical checks you can possibly do and it's just part of his life. "It's no problems whatsoever. It's not as serious as it looks." Gomis has scored two league goals for Swansea this season, mostly in backup role. He became the Welsh side's top striker when Wilfried Bony signed with Manchester City in January. Almost exactly three years ago White Hart Lane, then Bolton midfielder Fabrice Muamba collapsed after suffering a cardiac arrest. He was near death, according to Bolton, but survived after being treated at the London Chest Hospital. He subsequently retired. Other footballers, including Cameroon international VivieVivien Foe in 2003 and Spanish international Antonio Puerta in 2007, didn't 't survive after collapsing on the pitch.
wanted to reassure you concerning my health," Gomis
TL:DR:
Bafetimbi Gomis collapses within 10 minutes of kickoff at Tottenham But he reportedly left the pitch conscious and wearing an oxygen mask Gomis later said that he was "feeling well" The incident came three years after Fabrice Muamba collapsed at White Hart Lane
Article:
(CNN)lt was act of frustration perhaps more commonly associated with golf's fictional anti-hero Happy Gilmore than the world's reigning No 1. player. But when Rory Mcllroy pulled his second shot the eighth hole of the WGC Cadillac Championship into a lake Friday, he might as well have been channeling the much loved Adam Sandler character. Before continuing his round with a dropped ball, the four-time major winner launched the 3-iron used to play the offending shot into the water as well. " (It) felt good at the time." rueful Mcllroy later said of the incident in comments carried by the PGA Tour website. "I just let frustration get the better of me. It was heat of the moment, and I mean, if it had of been any other club I probably wouldn't 't have but I didn't need 3-iron for the rest of the round so thought, why not." The club "must have went a good 60, 70 yards," he joked. Mcllroy composed himself to finish with a second round of 70, leaving him one-under for the tournament and eight shots off the pace set by leader JB Holmes. While an improvement on last weeks performance at the Honda Classic event, where he failed to make the cut, the Northern Irishman's frustration with elements of his game was still clear. "I think every golfer feels because don't hit shots like the one I hit on on the range," " he said. "That's what really bothers me, the fact that I get out on the course and hit shots that I'm not seeing when I'm in a more relaxed environment. "So it's a little bit of mental, a little bit of physical. It's just everything is not quite matching up." Elsewhere the course, Ryan Holmes scored two-under-par 71 to remain in second position overall, two shots behind Holmes. Former world No 1., Adam Scott carded impressive 68 to finish the day three shots off the pace at six-under while Bubba Watson and Henrik Stenson are tied for fourth on four-under.
TL:DR:
Rory Mcllroy throws club into water at WGC Cadillac Championship Northern Irishman frustrated after pulling shot into water hazard
Article:
(article)
TL;DR:
Table 9: Our 2-shot prompt for evaluating CNN/DM summarization.
18


Reviews:
[{"review_id": "5cyQpNsPvN", "reviewer": "Reviewer_msgU", "summary": "The authors investigate the question of acheiving performance parity with high-quality proprietary systems by training (smaller, generally lower quality models) on the outputs of the proprietary systems. The investigation is carried out over a range of data sizes collected from proprietary systems, or imitation data, and a range of model sizes. The authors' conclude that while training on some imitation data can improve the style of weaker models, there (i) is still a large performance gap to the proprietary models especially in evaluations of general capabilities, (ii) diminishing returns from increasing imitation data, (iii) greater gains (than collecting imitation data) by simply increasing the model size.", "questions": "In Figure 4. the 5-shot MMLU performance of the 13B imitiation model is quite low for a model of that size - can the authors describe the setup in more detail?\n\nSome discussion of the points raised in \"weaknesses\" would also be welcome.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strengths of this work:\n- The question studied is important as most open-source models make use of imitation data for supervised finetuning.\n- The investigation along the data and model size axes is well thought out.", "weaknesses": "The main weaknesses of this work:\n- The implicit assumption of this work (revealed in the title) is that there exists a claim or understanding that imitating proprietary language models by sampling their outputs for training is all that is needed to achieve performance parity - however, I contend that this isn't the prevalent understanding. It is understood that proprietary model output is a good source of finetuning data but not necessarily the only source. See for example the use of FLAN alongside imitation datasets like Alpaca for SFT.\n- The authors present style imitation of propreitary models as a negative aspect of training on imitation data (at least in the abstract), however the right amount of style imitation can be a definite source of improvement for open-source models - as the authors point out in Section 4.4. My suggestion would be to revise the abstract to reflect this more accurately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQKEbFs3fl", "reviewer": "Reviewer_TcvA", "summary": "The paper critically analyzes the method of using the output of a stronger LM to fine-tune and improve a weaker LM, pointing out that model imitation is not a free lunch.\nThe authors concluded that broadly matching ChatGPT using purely imitation would require (1) a concerted effort to collect enormous imitation datasets and (2) far more diverse and higher quality imitation data than is currently available.", "questions": "Is model imitation still a good solution if the model's factual performance is decoupled to the retrieval model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Using the output of GPT-4 to cheaply improve a weaker language model by fine-tuning is widely adopted. This paper analyzes the drawbacks of doing so, which is helpful to guide the direction of developing more powerful LLMs.\n\n2. A large number of experiments and analyses prove the author's point of view.", "weaknesses": "1. The author claimed that it is far more feasible to distill a specific behavior from ChatGPT as opposed to broadly matching its capabilities. However, the paper only conducted experiments on NQ-synthetic data.\n\n2. The paper claimed that imitation models are adept at mimicking ChatGPT's style but not its factuality and become far better at following instructions. But the other important ability of the model, that is, the ability to reason, has not been well studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0h5ylCkMSZ", "reviewer": "Reviewer_A4hH", "summary": "The paper critically analyzes the approach of imitating proprietary systems (e.g., ChatGPT) by finetuning LMs using various model sizes, data sources, and imitation dataset sizes. Then the authors do human evaluation as well as evaluation on NLP benchmarks. Imitation models do not do well on tasks not heavily supported by the imitation data.\n\nOne interesting finding is that training on broad-coverage imitation data may decrease Natural Questions factuality, but training on NQ-like-data only will increase the accuracy. \n\nThe authors also conclude that the best action forward is to improve base LMs, instead of doing imitation on proprietary systems.", "questions": "Important: What are the decoding parameters for each model? (This is not addressed post-rebuttal.)\n\n\nBelow are minor (or very minor issues) in evaluating this paper:\n\nI wonder if practitioners may interleave ChatGPT imitation data with actual pretraining data and fine-tuning data. It’s unclear if this setting would lead to the same problems. \n\nThe base models discussed in this paper are quite weak. For example, llama-1-13b is used instead of the SFT- and RLHF-tuned llama-2-13b-chat. The models are quite small too. Unclear if the results generalize to larger models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "I vaguely heard of this paper when it came out but it’s my first time reading it. The motivation is excellent for sure (the public will care about this paper), given that many groups and startups are imitating proprietary language models potentially as a shortcut. \n\nThe findings are useful to many practitioners -- they'll likely carefully think whether knowledge distillation is useful or how it'll be useful. \n\nSome findings are quite interesting (see summary above for example).", "weaknesses": "I have three concerns related to crowdsourcing (see the next three paragraphs). \n\nDo human raters have low quality? The incentive design and crowdworker filtering seem lacking.  \n- What’s the human agreement (e.g., Fleiss' kappa)?\n- What’s the average time humans spend on each comparison?\n- Is there an option for humans to decline the comparison (because they may not be knowledgeable enough)? \n- How do you make sure that humans are rewarded based on correct choices, and potentially punished if they do extremely poorly?\n\nIt’d be useful to have human evaluators write out rationales on why they chose one over another (or rate on multiple scales using multiple metrics). Otherwise concluding “human evaluators rate imitation models’ outputs higher because of their style” seems only a conjecture to me. \n\nIf crowdworkers have low quality (thus their annotations unreliable), then it doesn't seem prudent to use Figure 1(c) (crowdworker preference vs. number of model parameters) to reach the conclusion that we should improve base LLMs. \n\n\nI also have some other concerns: \n\nThere are two settings for imitation in the paper. The second setting is broad-coverage imitation. The imitation dataset size could be much larger. Currently the authors are using around (90+27+10)K examples, but this is quite a small number of examples – the dataset size is even smaller than most of the machine translation training sets from ten years ago.  \n\n\nThe results in this paper are only specific to supervised fine-tuning, not RLHF for example. This should be qualified in the intro paragraph. \n\nThe authors claim that matching ChatGPT using imitation would require an “enormous” amount of imitation examples. Is this supported anywhere in the paper?\n\n\n\nUpdate: I read the authors' response. I'm still a bit concerned about human annotation quality, but I raised the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iq0ZufECrO", "reviewer": "Reviewer_WxVs", "summary": "The authors critically investigate the promise of finetuning language models using (imitation) data obtained from more capable language models(LMs). The paper challenges the assumption that this process improves an LM overall – but suggests that this is rather `mimicking their style`. They propose several interesting findings around what aspects of the performance improve or deteriorate upon tuning LMs on imitation data.", "questions": "1. The discussion seems to rely heavily on the concept of `tuning of imitation data`. However, I think one distinction is the kind of imitation data used to finetune most of the models, which are usually based on roughly arbitrary conversations between users and models. Would the authors agree that if the imitation data is constructed in a different way, then it may be possible to improve e.g. factuality of the finetuned model? For instance, I can imagine the way to construct imitation data to be around extracting rare knowledge, and then possibly the finetuned model could be improved a lot.\n\n2. Have the authors explored other capability definitions than factuality and toxicity? For instance, do we know anything about reasoning, calibration, creativity, truthfulness.. ? \n\n3. Did the authors get the necessary approvals from the IRB of their institution?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I overall support the messages and findings in the paper.\n\n1. I appreciate this timely and critical investigation. Finetuning existing LMs on instruction-tuning data from more capable LMs has swiftly become common practice, yet we do not fully understand the change in behavior. The paper raises several critical questions that I would personally appreciate having in the literature.\n\n2. One novel and important finding is that the authors find that even though training on imitation data improves the results in crowd worker evaluations, they observe even a degradation in factuality. This is a significant consideration that should be kept in mind when finetuning models. This further raises a question about what other aspects of capabilities may be fluctuating during imitation data training.\n\n3. On the other hand, they inherit some of the useful properties, such as reduced toxicity / being more safe. Again, this further informs us about what really happens when models are trained on imitation data. It would be compelling to further explore a more fine-grained decomposition of performance gains and losses upon training on imitation data.\n\n4. The experiments are large-scale and informative yet not cheap to perform, thus findings enable valuable conclusions that are otherwise not easy to draw.", "weaknesses": "There are 2 main points that I am concerned about.\n\n1. (IRB Approval / Exemption) The human study in the paper does not seem to have the relevant IRB approval or an exemption. The code of ethics states `Where human subjects are involved in the research process (e.g., in direct experiments, or as annotators), the need for ethical approvals from an appropriate ethical review board should be assessed and reported.` I am deferring the judgment about this to Ethics Reviewers / ACs. This should have most probably been done before the human subject study, but I encourage the authors to swiftly go through the IRB process for clarity.\n\n2. (Definition of Capability) The coverage of the capability evaluations is somewhat limited. Currently, authors evaluate mostly on factuality tests like MMLU/NQ/HumanEval and also show results around toxicity but how about other capabilities? Could it be that the imitation data leads LMs to reason better? Or, could it be that imitation data drives better calibration? While I do understand that there is a finite compute budget and there should be a limit in evaluation, the capability definition here is rather to limit the conclusions to draw. \n\n3. (Minor, Title) Given my concern in 2, I’m personally slightly skeptical to call this a `false promise`. It is unclear if it is broadly a false promise, or if there are capabilities that are improved – it’s rather there exists capabilities that this process even hurts, and we should be mindful about trusting crowdsourcing or other automated evaluations to understand the impact of a process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)