Lion: Adversarial Distillation of Proprietary Large Language Models
Authors: Yuxin Jiang
Chunkit Chan
Mingyang Chen
Wei Wang
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
instruction following
knowledge distillation
TL;DR: An adversarial distillation framework for efficiently transferring knowledge from a proprietary large language model (LLM) to a compact, open-source LLM.

Abstract: The practice of transferring knowledge from a sophisticated, proprietary large language model (LLM) to a compact, open-source LLM has garnered considerable attention. Previous works have focused on a unidirectional knowledge distillation way by aligning the responses of the student model with those of the teacher model to a set of instructions. Nevertheless, they overlooked the possibility of incorporating any "feedback"--identifying challenging instructions where the student model's performance falls short--to boost the student model's proficiency iteratively. To this end, we propose a novel adversarial distillation framework for a more efficient knowledge transfer. Leveraging the versatile role adaptability of LLMs, we prompt the teacher model to identify "hard" instructions and generate new "hard" instructions for the student model, creating a three-stage adversarial loop of imitation, discrimination, and generation. By applying this adversarial framework, we successfully transfer knowledge from ChatGPT to a student model (named Lion), using a mere 70k training data. Our results show that Lion-13B not only achieves comparable open-ended generation capabilities to ChatGPT but surpasses conventional state-of-the-art (SOTA) instruction-tuned models like Vicuna-13B by 55.4% in challenging zero-shot reasoning benchmarks such as BIG-Bench Hard (BBH) and 16.7% on AGIEval.

Lion: Adversarial Distillation of Proprietary Large Language Models
Yuxin Jiang Chunkit Chan?* Mingyang Chen 122 Wei Wang
'The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China The Hong Kong University of Science and Technology, Hong Kong SAR, China (yjiangem, ckchancc, mchenbt Jeconnect. www.uustthhk.. hk, weiwcs@ust.
Abstract
Instruction:
Whit are the ihree printry ooloes?
Teacher's Respouse: The tlree prinary colors are redt blae, and yellow.
Instruction: Calculie the following mathemsiial expression: 4 3(6-3).
The practice of transferring knowledge from sophisticated, proprietary large language model (LLM) to compact, open-source LLM gar- nered considerable attention. Previous works have focused on a unidirectional knowledge distillation way by aligning the responses of the student model with those of the teacher models to a set of instructions. Nevertheless, they overlooked the possibility of incorporat- ing any "feedback"" -ddeni-iinnnnn challenging instructions where the student model's 's perfor- mance falls short-to to boost the student model's proficiency iteratively. To this end, we propose a novel adversarial distillation framework a more efficient knowledge transfer. Leverag- ing the versatile role adaptability of LLMs, we prompt the teacher model identify "hard" in- structions and generate new "hard" instructions for the student model, creating a three-stage ad- versarial loop of imitation, discrimination, and generation. By applying this adversaria frame- work, we successfully transfer knowledge from ChatGPT student model (named Lion), using a mere 70k training data. Our results show that Lion-13B not only achieves compara- ble open-ended generation capabilities to Chat- GPT but surpasses conventional state-of the-harrt (SOTA) instruction-tunee models like Vicuna- 13B by 55.4% in challenging zero-shot rea- soning benchmarks such as BIG-Be (G--Bench Hard (BBH) and 16.7% AGIEval.
Tracher's Response: rouli erpression The colculation proczeds os follww: 3(6-3) -4 5(3 4 19.
Knowledge
&.S.B.
Teacher
Student
Feedback
Instraction: Whol are three primary colens?
Toocher's Response: these prinury
flw,, and yellow
Student's Respouse: The tiree prinary colors are: red. hise and yellow
Instruction: Caleulaic the following matherstical expression: 5(6-3)
Teacher & Response: The realt of the expression
The culcniation
hard
proceeus as followy 4: 516-3) -4 5/3) -4 1 19.
Student's Respoue: Aler colenlating the rosul 516-3)
An illustration of the distinction between our approach and earlier ones. Previous methods facilitate one-way knowledge transfer from the teacher to the student (solid arrow). Our approach, however, incorpo- rates an innovative step (dashed arrow) that completes a loop: it enables the feedback" feedback"-ideniiffiiin identifying the student model's weaknesses to be relayed back to the teacher, in order to foster tailored learning.
Figure 1:
typically proprietary, comprising both the model parameter and the training data. To foster increased transparency regarding their intricate operational mechanies, a surge in research efforts focusing on knowledge distillation from a proprietary "teacher"
1 Introduction
LLM to an open-source "student" LLM. This typ- ically accomplished by aligning the responses the student model with those of the teacher model to a set of instructions, which can be manually or automatically generated (Wang et al., 2022; Taori et al., 2023; Chiang et al., 2023; Xu et al., 2023).
Large language models (LLMs) capable of follow- ing natural language instructions have exhibited tremendous success in generalizing zero-shot to new tasks (Mishra et al., 2022; Wei et al., 2022a). Due to various concerns, the most advanced LLMs, such as ChatGPT (OpenAI, 2022) and GPT-4 (Ope- nAI, 2023) that boasting billions of parameters, are
However, previous works employ a unidirec- tional approach to knowledge transfer (solid arrow in Figure 1), where the teacher imparts knowledge to the student without considering any "feedback".
'The two authors have equal contributions.
Cooee and model can be found at ht /ttp:::///ithii httb.//.ttttht/////oo YJiangcm/Lion

---

To better illustrate this using a tangible classroom scenario, the "feedback" refers identifying the "hard" examples or problems where the student's s performance falls short. This feedback guarantees that the teacher can provide bespoke training that centers on "hard" examples, thereby paving the way for more effective and tailored learning experi- ences for the student.
iments on open-ended generation and reasoning datasets, which include a total of 40 sub-tasks, our Lion-13h showcases superior performance surpass- ing instruction- tunoodd baseline models such as Vi- cuna (Chiang et al., 2023). Our main contributions are as follows:
Our work is the first attempt to adopt the idea of adversarial knowledge distillation to large language models.
Inspired by adversarial knowledge distillation (AKD), which aims to iteratively improve the stu- dent model's s performance by learning from gener- ated hard samples (Fang et al., 2019; Micaelli and Storkey, 2019a; Heo et al., 2019), we propose adversarial framework for distilling a proprictary LLM into a compact student model. Nevertheless, these AKD methodologies necessitate accessibility to the weights or gradients of the teacher model, which cannot be directly adapted to our setting. To circumvent this problem, we leverage the un- paralleled role adaptability of LLMs, which can be effectively employed through diverse range of prompts (Sanh et al., 2022). In particular, prompt the proprietary teacher LLM serve as "referee" discriminate hard instructions where there exists a significant performance discrepancy between the teacher's and student's responses, and serve as a "generator" to produce new instructions that emulate the data distributions corresponding to the discriminated hard instructions. Our frame- work, depicted in Figure 2, consists of three stages in an iteration: 1) an imitation stage to align the student's response with the teacher's response; 2) a discrimination stage to identify hard instruc- tions; 3) generation stage to produce new hard instructions for escalating the challenges presented to the student model. In essence, our adversarial framework forms positive feedback loop that effi- ciently bootstraps the student model's proficiency.
Our proposed framework demonstrates im- pressive efficiency and efficacy, With instruc- tion tuning performed on 70k data without any human annotation, our Lion-13B approxi- mates ChatGPT's capabilities on open-ended generation dataset and largely outperforms the current SOTA model Vicuna-1 1amna-13B.. on reason- ing tasks.
The versatility of our framework allows for broad application: is not exclusive to Chat- GPT but can be conveniently adapted to suit a variety of other proprietary LLMs.
2
Related Work
2.1
Instruction- Foollooin Language Models
With the impressive ability of instrruttion-folloinng large language models such as ChatGPT (Ope- nAI, 2022) and GPT-4 (OpenAI, 2023), the tech- niques of instruction tuning (Wei et al., 2022b) have attracted lot of attention (Wei et al., 2022c; Bubeck et al., 2023; Bang et al., 2023; Chan et al., 2023a). The early research of instruction tuning aims to enhance the generalization ability of lan- guage models, allowing these models to perform new tasks by comprehending task descriptions with- out relying a few examplars. By fine-tuning these instruction-followinn language models (e.g.. T5 (Raffel al., 2020), FLAN (Aribandi et al., 2022), TO (Sanh et al., 2022), and ExT5 (Aribandi et al., 2022)) on multi-task datasets the form of natural language phrased as instructions, these models have been shown to perform well on unseen tasks with the instructions.
To verify the efficiency and efficacy of our method, we apply our AKD framework to transfer the knowledge of ChatGPT onto an open-source foundation LLM, known as LLaMA (Touvron et al., 2023). We seleet Alpaca's training data (generated from only 175 manually selected seed instructions) as the initial training instructions and execute three iterations of AKD, resulting total of 70K data that our model is trained on. We' ve christened our model as Lion, drawing inspiration from the art of "distillation". By conducting extensive exper-
However, these models are only tnnettundd on simple task-specific instructions, and it is challeng- ing to comprehend the sophisticated and diverse intent of users in real-world scenarios. Therefore. InstructGPT (Wei et al., 2022b), ChatGPT (Ope- nAI, 2022), and GPT-4 (OpenAI, 2023) trained on the diverse forms and abundant task types of
We access ChatGPT using the OpenAl API (gpt-3.5 model).

---

human-crafted instructions annotated by consid- erable number of annotators. Since these instruc- tions were not open-sourced, recent works such as Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and WizardLM (Xu et al., 2023) investi- gate how to generate high-quality instructions and fine-tune the open-source large language model LLaMA (Touvron et al., 2023) with them to ap- proach the performance of ChatGPT.
3 Methodology
Harnessing the learned knowledge of a sophisti- cated teacher model ((x; (7) where the parame- ter 07 is inaccessible, our goal is to craft more lightweight student model S(x; 09). Ideally, a stu- dent model optimal if the expectation of model discrepancy (which indicates the prediction differ- ences between teacher T and student S) on the uniform data distribution is minimized. Inspired by the success of adversarial knowledge distillation (AKD) (Fang et al., 2019; Micaelli and Storkey, 2019a; Heo et al., 2019), we turn optimize upper bound of the expectation -the expectation of the model discrepancy on "hard samples", where the teacher T and the student have a relatively large performance gap. These "hard samples" are inclined to dominate the expectation of the model discrepancy. Thus, the overall expected model discrepancy can be effectively and efficiently re- duced by optimizing the student model 8 these "hard samples". The underlying rationale is rather straightforward and can be analogized to real- world educational scenario: continuously concen- trating on the "hard" knowledge that the student finds challenging to grasp is the most effective man- ner of enhancing student's 's proficiency.
2.2 Knowledge Distillation
Knowledge Distillation (KD) (Hinton et al., 2015; Radosavovic et al., 2018; Chen al., 2019) repre- sents a crucial strategy within the sphere of model compression and acceleration, wherein a compact student model is instructed to emulate the perfor- mance traits of a more cumbersome teacher model. In practical contexts, the availability of training data is often constrained due to concerns regard- ing privacy, legality, security, confidentiality. To address the absence of training data, data-free KD methods were proposed align the student model to the teacher model, capitalizing on either related proxy data (Orekondy et al., 2019; Papernot et al., 2017) or synthetic data generated by learnable generators (e.g., Generative Adversarial Network (GAN)) (Addepalli et al., 2020; Fang et al., 2019; Micaelli and Storkey, 2019b) teacher model in- versions (Yin et al., 2020; Chawla et al., 2021; Fang al., 2022). Nevertheless, these KD methodolo- gies necessitate the accessibility to the weights or gradients of the teacher model. Consequently, an alternative line of research, commonly denoted as data-free model extraction (or stealing), endeavors to bridge this gap by employing zero-order estima- tion methodologies to approximate the authentic gradients of the teacher model guide the up- date of the optimized generators (Kariyappa et al., 2021: Truong al., 2021). However, adapting these methods to our distillation task presents two main hurdles. First, these techniques are primarily designed for image-based classification tasks, as- suming access to continuous softmax vector from the teacher model. Estimating zero-order gradients becomes problematic in our case, responses typically sequence-oriented. Second, developing an effective instruction generator capable of pro- ducing diverse, high-quality instructions that mir- ror the teacher model's 's training data distribution proves more challenging than in the image domain.
However, in the process of training the student model S, hard samples will be mastered by the student and converted into easy samples. Hence we need a mechanism to continuously generate hard samples, which can be achieved by an adversarial framework.
The whole framework of our Adversarial Knowl- edge Distillation is depicted in Figure 2, which contains three stages in an iteration: 1) an imi- tation stage to align the student's response with the teacher's response; 2) discrimination stage to identify hard samples; 3) generation stage to produce new hard samples for escalating the chal- lenges presented to the student model.
3.1 Initilization
As shown in Figure 2, four roles and two data pools are established in our framework, and we will com- prehensively illustrate their functions later. We initialize student model S using foundation LLM such as LLaMA (Touvron et al., 2023). We initialize our teacher model T. referee R, and gen- erator g by using the same proprietary L.LM such as ChatGPT (OpenAI, 2022). The multiple roles that this proprietary LLM serves are accomplished

---

Tencher
X
① Inference
(x F(x
hard instuctioos
Replace P
A&A
BCrle (oroo))ate") 22
P
C Sumpting
Inference
new
(x/))(x)))
(AIGGN (A
Referce
Sampling
Vinda
Cache Pool
C Add
② Inference
[x Sx""((x"))
x
incorcctios
Student
Figure 2: The overview our adversarial distillation framework, where we craft a compact Student LLM S based superior proprietary LLM that serves three roles: the Teacher T, the Referee RR and the Generator G. From left to right, there are three stages in an iteration: 1) Imitation; 2) Discrimination; 3) Generation.
through the use of varied prompt templates. We start the iteration from a given initial Train Pool X (x.²) JEj,,,N where 2 is the i-th instruc- tion in X and N is the number of samples in XA. The Cache Pool X is initialized as identical XA, consisting of instructions evaluate the performance of and T.
more thorough detection of hard instructions.
In the discrimination stage, we ask the propri- etary L.MM to serve as "referee", which quantifies the performance gap between T and Specifi- cally, we feed each instruction of in the Cache Pool X through both the teacher T and student S to generate the outputs T(x:²) and S(x!"), re- spectively. Then we ask the referee R to quan- titatively measure the quality difference between teacher's response T(x and student's response S(a?). conditioned on 2.
3.2 Imitation Stage
To impart the knowledge of the teacher to the stu- dent, we construct the instruction-response data {xA, (Tx))(())))) )cn,,N by forward propagating in- structions in the Train Pool X through the teacher T. The prompt template used for model inference is shown in Table 10. Like the imitation training of previous work (Taori et al., 2023; Chiang et al., 2023), we fine-tune our student model S to align the response of the teacher model, by optimizing the autoregressive language modeling objective.
(1) The above process is conducted by using the prompt template (as shown in Table 11) inspired by (Chiang et al., 2023), which requires the LLM to consider the helpfulness, relevance, accuracy, and level of detail of two responses and output two scores. To mitigate the positional bias (Wang al., 2023) of the LLM referee, we conduct two runs by exchanging the positions of the teacher's response and the student's response and compute the final score as the average of the two runs. Then d, is calculated as the difference between the teacher's 's score and the student's 's score. By setting threshold (1.0 used in our experiments), we discriminate hard instructions those instructions with d; 2 т. and the others are identified easy ones. Fig- ure 3b provides clear and intuitive demonstration of which kinds of instructions are discriminated as hard in the first iteration. Compared with the instructions in the Cache Pool (Figure 3a), the dis-
d: = RRT(aa (, 1x
3.3 Discrimination Stage
Figure 2 demonstrates that the discrimination stage starts from the Cache Pool, denoted as B Even though this pool begins with the same initializa- tion as the Train Pool, their uses diverge. The Train Pool is rejuvenated by replacing its existing instruc- tions with freshly generated instructions, whereas the Cache Pool is enriched by incorporating these generated instructions. As result, the growing storage capacity of the Cache Pool provides more extensive space for evaluating the performance gap between teacher T and student S. This allows for

---

sentened
mtory
Sestlate
etory
senienee
parggripo
sive
(a) Instructions of the Cache Pool in the first iteration.
(b) Identified hard instructions in the first iteration.
(c) Generated hard instructions in the first iteration.
Figure 3: The top 20 most common root verbs (inner circle) and their top direct noun objects (outer circle) in the instructions.
tribution of the identified hard instructions is quite different, focusing more on complex tasks such as math, coding, etc.
aforementioned in Section 3.3, we proceed to reju- venate the Train Pool, replacing its existing instruc- tions with freshly generated ones. Concurrently, we enrich the Cache Pool by incorporating these newly generated instructions.
3.4 Generation Stage
After carefully discerning the hard instructions, the generation stage aims to produce samples that mir- ror the data distributions corresponding to these challenging directives. This process is achieved by employing the proprietary LLM as a generator, denoted as G, leveraging its exceptional prowess in content creation. Inspired by (Xu et al., 2023), we randomly sample an instruction from the hard instructions and prompt the generator G to generate new instruction. The newly generated instruction is required to pertain to the same domain and match the task type of the sampled instruction. The tem- plate utilized for this prompt exhibited in Table 12. As shown in Figure 3c, the distribution of the newly generated hard instructions appears to be comparable to that of the previously identified hard instructions. To mitigate the issue of catastrophic forgetting and to augment the diversity of the gen- erated instructions, instruction from the easy instructions and prompt the generator G to generate a new instruction that belongs to the same domain as the sampled one. but exhibit a more long-tailed distribution. The template we use to prompt this process is displayed in Table 13.
3.5 Min-Max Game Interpretation
Our adversarial knowledge distillation framework can be interpreted as a dynamic min-max game: in the imitation stage, we fine-tune our student to minimize the model discrepancy between itself and the teacher on hard samples; in the discrimination and generation stage, we craft new hard samples to maximize the model discrepancy, based on the learning progress of the student model. This dialec- tic framework propels the student model towards uncovering otherwise hidden knowledge, paving the way to complete understanding. As the training progresses through several iterations, the system should ideally achieve equilibrium. This is the point where the student model has mastered all the hard samples and the referee R can no longer distinguish between the student S and teacher T models. At this juneture, S becomes functionally indistinguishable from T.
also randomly sample
4
Experiments Setting
4.1 Datasets
In each iteration, we define N as the total count
of newly generated instructions and maintain 1:1 ratio between the generated hard instructions and the generated easy instructions. To promote diver- sity, new instruction will be deemed valid only if its ROUGE-L overlap with any existing instruc- tions in the Cache Pool is below 0.7. Finally, as
In our experiments, implemented compre- hensive LLM evaluation protocol that considers diverse range of abilities, such as writing, coding, commonsense, math, and logical reasoning. The datasets we utilized can be classified into two main categories: open-ended generation and reasoning.

---

4.1.1 Open-ended Generation Datasets
2023). Given the free-form response from the gen- crative models, only the first capital character in the response is considered to compare with the gold answer (exact match). The result we report is accuracy (%).
Instrcciotttutss (Chiang al., 2023) is set of 80 questions spanning 9 distinct task cate- gories. This dataset has gained extensive usage in evaluating the capabilities of LLMs. Within our work, we examine LLMs' performance on this dataset in two different settings:
4.2 Baselines
We select five superior LLMs as baselines, includ- ing LLaMA (Touvron et al., 2023), Alpaca (Taori et al., 2023), WizardLM (Xu et al., 2023), Vi- cuna (Chiang et al., 2023), and ChatGPT (Ope- nAI, 2022). It is worth noting that Vicuna has con- sistently ranked as the top open-source language model on multiple leaderboards, such as Chatbot Arena'. Therefore, we will conduct comprehen- sive comparison with Vicuna. See detailed descrip- tions of these baselines in Appendix B.
Setting1:
Following Vicuna (Chiang al., 2023), we leverage GPT-4 automatically assess the quality of responses (rated on a scale of 1 to 10) between a reference model (ChatGPT) and a candidate model. Subse- quently, we calculate the candidate model's performance as the percentage of the total score it achieves compared to the reference model.
Setting2: A recent work (Wang et al., 2023) pointed out that systematic bias may exist in the above-mentioned GPT-4 automatic evalua- tion. To mitigate this, they propose two strate- gies, namely Multiple Evidence Calibration and Balanced Position Calibration, to obtain closer alignment with human judgments.
4.3 Implementation Details
Training Details Our student model is initialized using the pre-trained LLaMA. The Train Pool and Cache Pool are initialized with the 52K automat- ically generated instructions from Alpaca (Taori et al., 2023). The total number of iterations is set to 3, with 6K newly generated instructions added at each iteration. This results a total of 70K data that our model trained on in order to make a fair comparison with current SOTA baselines, including WizardLM and Vicuna. The training hy- perparameters are listed in Appendix C.
4.1.2 Reasoning Datasets
AGIEval (Zhong al., 2023) is well-known benchmark that quantifies the reasoning capabil- ity of foundation models in the context of human- centric standardized exams, including college en- trance exams, math competitions, lawyer qualifi- cation tests, etc. We choose all English multiple- choice questions (8 tasks, 2,546 samples) among AGIEval for our experiments. The data statistics are shown in Table 6.
Inference Details To draw inferences from Lion and ChatGPT, we calibrated the temperature to 0.7 and set the maximum generation length at 1024. All other parameters adhere to their default settings. For LLaMA, Alpaca, WizardL M,, and Vicuna, we configured their inference parameters in line with the specifications given their respective original papers. When engaging with the gpt-3.5-turbo API for various roles, we employ an array of hyper- parameters, the specifics of which can be located in Appendix C.
BIG-Bench Hard (BBH) (Suzgun et al., 2022) consists of a suite of challenging tasks from BIG- Bench (Srivastava et al., 2022), designed to assess the capabilities and limitations of large language models. These are the tasks which prior lan- guage models underperform the average human rater. We choose all tasks that can be formatted into multiple-choice questions (23 tasks, 5,511 sam- ples) among BBH for our experiments. The data statistics are shown in Table 7.
5 Experimental Results
5.1 Results for Open-ended Generation
Table shows the performance comparison of var- ious models against ChaiGPT the reference model, where GPT-4 is used as a referec/rater. Our Lion-7B and Lion- 13B remarkably outperform their counterparts under two evaluation settings.
Setting We evaluate reasoning capabilities under zero-shot setting without any exemplars and with- out Shain-of-Though (CoT). For both AGIEval and BBH, we use the prompt format and parsing following (Zhong et al., 2023; Mukherjee et al.,
hhttps://chat.. wwwsss.org/aaal...

---

Model
Setting1
Setting2
Avg.
gories and achieving an average relative improve- ment of over 16%. However, Lion-13B still signifi- cantly lags behind ChatGPT, only retaining 72.5% of its reasoning capability.
LLaMA-7B
58.46
59.12
58.79
Alpaca-7B
69.29
67.20
68.25
M-Z7
89.29
86.67
87.98
Vicuna-7B
87.79
89.96
88.88
Lion-7B
94.74
92.88
93.81
BIG-Bench Hard Results Table 3 displays the zero-shot performance comparison between Lion and baseline models on BIG-Bench Hard with stan- dard zero-shot prompting. Similar to AGIEval, Vicuna exhibits poor performance on sophisticated reasoning tasks within this benchmark, while Lion substantially surpasses Vicuna by around 50% on average. Particularly, Lion demonstrates signifi- cant performance enhancements of over 100% on tasks involving data understanding, semantic un- derstanding (Disambiguation QA and Snarks), log- ical and geometric reasoning (Logical Deduction and Geometric Shapes), and position reasoning (Tracking Shuffled Objects). Despite achieving an average ability of nearly 74% compared to Chat- GPT on BBH, Lion-13B surpasses ChatGPT in several tasks, including Movie Recommendation, Snarks (identifying sarcastic sentences from two nearly-identical ones), and Tracking Shuffled Ob- jects. This demonstrates the effectiveness of our method.
LLaMA-13B
69.23
68.21
68.72
Alpaca-13B
76.87
74.69
75.78
Vicuna-138 13B
92.25
92.97
92.61
Lion-13B 96.57
100.18
98.38
Table Relative response quality (%) against ChatGPT (assessed by GPT-4) on Vicuma-Instructions.
1LaMA Alpaco-78 78 WitardLM-2 75 Vicuna-78 Uon-78 ChatGPT
120%
80%
60%
10%
20%
0%
Benene
knowledge
ralsplay
common- Arrue
femi
conterfactuul
coding
writig
Figure 4: Relative response quality against ChatGPT on diverse task categories of Vicuna-Insten Instruccsttucconss.
Analyses
Noticcably, Lion- 13B shows 8- 8-point improve- ment over Vicuna-13BB on aggregate. achieving 98.38% capabilities of CharGPT.
6.1 Ablation Studies
The threshold for distinguishing between hard and easy instructions We systematically ex- plored ranging from 0.0 to 2.0 and documented its influence average performance across three datasets. Table 4 reveals an optimal range of r between 1.0 and 1.5 for all datasets. Notably, ele- vating from 0.0 to 1.0 consistently enhances per- formance across all datasets, indicating effective differentiation between hard and easy instructions. However, continuous increase from 1.0 to 2.0 gradually degrades performance due to decreased diversity in hard instructions. The ablation results demonstrate that our method is not quite sensitive to a large value o.
To comprehensively compare with other baseline models on the capability to generate high-quality responses on various types of instruction, the rel- ative response quality (Setting2) among different task categories is depicted in Figure 4. Our model impressively and slightly surpasses ChatGPT in the generic, knowledge, common-sense and counter- factual task categories. Furthermore, for the two difficulty task categories described in the previ- ous study (Chiang et al., 2023; Xu et al., 2023), our model significantly outperforms other baseline models with at least 32.32% relative score in the math task category while exceeding most of the baseline in the coding generation task category.
The ratio of generated hard and easy instruc- tions We change the ratio of generated hard in- structions to generated easy instructions from 1:0 (all hard) 0:1 (all easy) and investigate its impact on average performance across three datasets. It can be seen from Table that higher ratios of hard to casy instructions generally lead to improved per- formance, with a balanced ratio of 1:1 yielding the
5.2 Results for Reasoning
AGIEval Results Table presents the standard zero-shot performance comparison between Lion and baseline models on the AGlEval benchmark for multiple-choice English questions. Lion demon- strates significantly stronger performance com- pared to Vicuna, surpassing it in most task cate-

---

Human
Task
ChatGPT
Vicuna-7B
Lion-7B
Vicuna-13B
Lion-13B
Avg
Top
AQuA-RAT
85.0
100.0
31.9
23.2
18.5 (-20.3%)
20.1
26.0
(29.4%)
LogiQA
86.0
95.0
35.0
21.4
31.8 (48.6%)
29.8
31.3
(5.0%)
LSAT-AR
56.0
91.0
24.4
22.2
17.4 (-21.6%)
20.4
23.0
(12.7%)
LSAT-LR
56.0
91.0
52.6
18.6
28.2 (51.6%)
32.6
32.6
(0.0%)
LSAT-RC
56.0
91.0
65.4
21.9
29,4 (34.2%)
32.7
40,9 (25.1%)
SAT-Math
66.0
42.7
(-2.3%)
28.6
(2.8%)
SAT-Fnglish
66.0
81.1
25.7
36.4 (41.6%)
44.2
53.9
(21.9%)
SAT-English (w/o Psg.)
66.0
94.0
44.2
(5.7%)
26.2
(38.2%)
Average
67.1
93.8
47,2
22.6
26.3 (16.4%)
29.3
34,2
(16.7%)
Table Zero-shot performance comparison ChatGPT, Vicuna, and Lion on AGIEval (multiple-choice English questions). We report the performance of Human, ChatGPT, and Vicuna from (Mukherjee et al., 2023). Performance improvements obtained by Lion over Vicuna are shown in parenthesis.
Task
ChatGPT
Vicuna-7B
Lion-7B
Vicuna- 13B
Lion-13B
Boolean Expressions
82.8
39.2
55.2
(40.8%)
40.8
65.6
(60.8%)
Causal Judgement
57.2
39.7
50.3 (26.7%)
42.2
43.9
(4.0%)
Date Understanding
42.8
8.6
34.0 (295.3%)
10.0
40.4
(304.0%)
Disambiguation QA
57.2
15.2
35.6 (134,2%)
18.4
44.8 (143.5%)
Formal Fallacies
53.6
40.0
46.0 (15.0%)
47.2
52.4 (11,0%)
Geometric Shapes
25.6
3.6
8,8 (144.4%)
3.6
8,8 (144.4%)
Hyperbaton
69.2
42.8
$1.6 (20.6%)
44.0
56.8 (29.1%)
Logical Deduction (5 objects)
38.8
4.8
19.6 (308.3%)
4.8
20.8 (333.3%)
Logical Deduction (7 objects)
39.6
1.2
14.4 (1100.0%)
1.2
21.2 (1666.7%)
Logical Deduction (3 objects)
60.4
19.6
40.4 (106.1%% %)
16.8
38.0 (126.2%%
Movie Recommendation
55.4
24.4
26.8 (9.8%)
43.4
57.6 (32.7%)
Navigate
55.6
43.6
49.2 (12.8%)
46.4
45,2 (-2.6%)
Penguins in Table
45.9
17.5
24.7 (41.1%)
15.1
26.7 (76.8%)
Reasoning about Colored Objects
47.6
14.0
15.2 (8.6%)
12.0
17.6 (46.7%)
Ruin Names
56.0
12.2
14.4 (18.0%)
15.7
29.2 (86.0%)
Salient Translation Error Detection
40.8
2.0
12.0 (500.0%)
2.0
12.4 (520.0%)
Snarks
59.0
28.0
56.2 (100.7%)
28.1
61.2 (117.8%)
Sports Understanding
79.6
40.4
48.4 (19.8%)
48.4
51.6 (6.6%)
Temporal Sequenees
35.6
21.2
24.4 (15.1%)
16.0
10.4 (-35,0%)
Tracking Shuffled Objects (5 objects)
18.4
6.4
14.4 (125.0%)
9.2
24.8 (169.6%)
Tracking Shuffled Objects objects)
15.2
4.0
13.6 (240.0%)
5.6
13.2 (135.7%)
Tracking Shuffled Objects objects)
31.6
26.8
34.0 (26.9%)
23.2
34.4
(48.3%)
Web Lies
56.0
49.4
47.2
(-4.5%)
41.2
54.8
(33.0%)
Average
48.9
21.9
32.0
(45.9%)
23.3
36.2
(55.4%)
Table 3: Zero-shot performance comparison of ChatGPT, Vicuna, and Lion BIGBench Hard (multiple-choice questions) without COT. We report the performance of ChatGPT and Vicuna from (Mukherjee et al., 2023).
Performance improvements obtained by Lion over Vieuna are shown parenthesis.
highest average scores.
6.3
Case Studies
To clearly compare the generated response quality between our model and other baselines, we provide nine case studies sampled from Vicuna-instruction, AGlEval, and BBH in Appendix E. Table 14 show- cases the responses of various models to math instruction. It can be seen that only Lion and Chat- GPT provide the correct answer and follow the correct problem-solving steps. A counterfactual case is shown in Table 15, where ChatGPT pro- vides a relevant answer that considers the potential impacts of Newton focusing on biology instead physics, but lacked details and depth. Lion, on
6.2 The Learning Dynamics of Lion
Figure 5, we delve into the learning dynamics of Lion by visualizing its performance on AGIEval and BBH throughout the training iterations. The results clearly demonstrate that our adversarial knowledge distillation framework consistently hances the performance of the student model the iterations progress. Notably, the most significant improvement in capability occurs in the first itera- tion, suggesting the usefulness of the identification challenging example patterns (refer Figure 3b).

---

Threshold
Vicuna-Instructions (Avg.)
AGIEval (Avg.)
BBH (Avg.)
0.0
89.58
22.4
26.5
0.5
92.16
23.5
29.8
93.81
26.3
32.0
94.09
25.7
31.6
2.0
92.23
24.6
31.3
Table 4: Ablation study of the threshold for Lion-7B.
Ratio r
Vicuna-Instructions (Avg.)
AGIEval (Avg.)
BBH (Avg.)
24.3
30.8
2:1
92.95
25.7
33.1
93.81
26.3
32.0
91.77
23.9
29.6
0:1
90.02
24.3
Table 5: Ablation study of the ratio for Lion-7B.
38
lows refine the student model's performance iteratively, efficiently bootstrapping its proficiency. We aspire that our model, named Lion, may serve as a bascline to reflect the performance ChatGPT, especially the open-source instraction-followinn language model baseline for our community.
34
Accuracy (60
30
26
24
0211 731
D01 (109-110))
Limitations and Discussions
20
52k
Sak
6ik
7ik
Training Data Size
The Model Capability We have identified that Lion is subject to certain constraints: 1) recent study (Gudibande et al., 2023) asserts that "model imitation is a false promise" since imitation models are adept at mimicking ChatGPT's style but fall short in improving LMs across more challenging tasks. While Lion still lags behind its teacher model ChatGPT in handling intricate reasoning tasks (as shown in our experiments). demonstrates promis- ing improvements compared previous imitation models. Therefore, our adversarial knowledge dis- tillation framework may provide a more effective way for knowledge transfer. 2) Since our training data doesn't 't encompass dialogues, Lion struggles to manage multi-turn conversations. 3) Due to com- putational resource constraints, Lion's maximum sequence length is limited 1024. Consequently, faces challenges when dealing with long doc- uments. Despite these limitations, we envision Lion serving as an accessible springboard for fu- ture research endeavors aimed at addressing these limitations.
Figure 5: Performance of Lion-7B and Lion- 13B on AGIEval BBH through training iterations.
the other hand, offered more detailed and engag- ing response that explored different possibilities such the development of biophysics or discov- ering new principles that could be applied to both fields. Lion's response also considered the potential implications of Newton's work on motion, force, gravity, and thermodynamics in biology, providing more comprehensive answer.
7 Conclusion
This paper presents an innovative adversarial knowledge distillation framework for distilling a proprietary LLM into a compact, open-source stu- dent model. While previous methodologies have concentrated on unidirectional knowledge transfer, our approach seeks to integrate "feedback" into the learning process. Leveraging the versatile role adaptability of LLMs, we prompt the proprietary model identify "hard" instructions and generate new "hard" instructions for the student model, cre- ating a stree-stage adversarial loop of imitation, discrimination, and generation. This approach al-
The Training Process To train single student model, we request the gpt-3.5-turbo API around 450k times, number that roughly 70% the WizardLM's usage of 624k (Xu et al., 2023).

---

Nonetheless, this utilization incurs a considerable expense, nearing $900. In contrast methods like Alpaca (Taori et al., 2023) and WizardLM (Xu et al., 2023), which only fine-tune the student model once, our adversarial knowledge distillation method employs iterative parametric updates to the student model. While this iterative approach in- evitably leads to slower iteration speed, it offers additional benefits. Finally, different from tradi- tional adversarial knowledge distillation where the weights of the generator are iteratively updated, we use a black-box and parameter-frozz LLM (Chat- GPT in our paper) to serve the role. Therefore, the quality of the LLM is quite essential in the generation of new instructions.
behaviors by imitating ChatGPT. However, is important acknowledge the potential risks asso- ciated with using Lion for malicious purposes, espe- cially upon releasing its weights the future. For future work, we aim to incorporate the technique of Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) to enhance access control. Additionally, Meta has implemented access application process that can help regulate the distribution LLaMA models and minimize the potential risks associated with their usage, pro- viding an alternative option.
Acknowledgements
W. Wang was also affiliated with Guangzhou Mu- nicipal Key Laboratory of Materials Informatics, The Hong Kong University of Science and Tech- nology (Guangzhou), China. He was supported by HKUST(GZ)) Grant G0101000028, GZU-HKUST Joint Research Collaboration Grant GZU22EG04, HEFHHeiwiDBC222322 and Guangzhou Mu- nicipal Science and Technology Project (No. 2023A0310003).
The Evaluation Metrics Though automated evaluations leveraging GPT-4 have showcased promising prospects appraising chatbot perfor- mance, the technique yet to reach level of maturity and accuracy, especially considering the propensity of large language models to generate non-existent or "hallucinated" information. Eval- uating the efficacy of LLM across various tasks presents a considerable challenge since different tasks require quite different expertise (Wang al.. 2022). Therefore, the creation of comprehensive, standardized evaluation system for chatbots is pre- vailing research challenge that demands additional exploration and study.
Ethics Statement
Inherited Biases It is important consider that the behavior of our distilled student models may ex- hibit potential toxicity, biases, or privacy issues (Li et al., b023a,b) inherited from the larger teacher LLM. We anticipate that the advancements made in reducing anti-social behaviors in LLMs can also be utilized to enhance student language models.
License and Legality Based on Stanford Al- paca's guidelines (Taori al., 2023), we have deter- mined that the weights Lion will be exclusively licensed for research purposes the future. Uti- lizing Lion's weights alongside LLaMA's original weights must adhere to Meta's LLaMA License Agreement. Users are responsible for acquiring and utilizing LLaMA in accordance with the li- cense agreement.
Safety Unlike ChatGPT (OpenAI, 2022), Lion does not rely on human feedback to mitigate unde- sired behaviors. Instead, Lion learns to avoid such

---

References
Winter Conference on Applications of Computer Vi- sion. pages 3289-3292
Sravanti Addepalli, Gaurav Kumar Nayak, Anirban Chakraborty, and Venkatesh Babu Radhakrishnan. 2020. Degan: Data-enriching gan for retrieving rep- resentative samples from a trained classifier. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 3130-3137.
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2019. Distilling knowledge learned in bert for text generation. arXiv preprint
Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng. Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2022. Ext5: Towards extreme multi- task scaling for transfer learning. In The Tenth In- termational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuma: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Gongfan Fang. Kanya Mo, Xinchao Wang. Jie Song. Shitao Bei, Haofei Zhang, and Mingli Song. 2022. Up to 100x faster data-free knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6397-6604.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hal- lucination, and interactivity. CoRR, abs/2302.00022 04402...
Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Da Chen, and Mingli Song. 2019. Data-free adversarial distillation. CoRR, abs/1912.1 1006.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro. and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12711
Google. 2023. Bard.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise imitating proprictary 1lms. CoRR, abs/2305.15717.
Chunkit Chan and Tsz Ho Chan. 2023, Discourse-aware prompt for argument impact classification. In Pro- ceedings of the 15th International Conference on Machine Learning and Computing, ICMLC 2023,
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. 2019. Knowledge distillation with adversarial samples supporting decision boundary. In The Thirty-Thing AAAI Conference on Artificial Intel- ligence, AAAI 2019, The Thirty-First Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Hon- olulu, Hawaii, USA, January 27 February 1, 2019. pages 3771-3778. AAAI Press.
Zhuhai, China, February 17-20, 2023, pages 165- 171. ACM.
Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023a. Chatgpt evaluation on sentence level rela- tions: A focus temporal, causal, and discourse relations. CoRR, abs/2304. 148277
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.0251.
Chunkit Chan, Xin Liu, Tsz Ho Chan, Jiayang Cheng. Yangqiu Song, Ginny Y. Wong, and Si- mon See. 2023b. Self-consistent narrative prompts on abductive natural language inference. CoRR, abs/2309.08300.
Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022. Im- proved universal sentence embeddings with prompt- based contrastive learning and energy- ensrgy-based learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3021-3035. Association for Computational Linguisties.
Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan Li, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023c. Discoprompt: Path prediction prompt tun- ing for implicit discourse relation recognition. In Findings of the Association for Computational Lin- guistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 35-57. Association for Computational Linguistics.
Cheng Jiayang. Lin Qiu, Tsz Ho Chan, Tianqing Fang, Weiqi Wang. Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song. Yue Zhang. and Zheng Zhang. 2023. Storyanalogy: Deriv- ing story-level analogies from large language mod- els to unlock analogical understanding. CoRR, abs/23 12874.
Akshay Chawla, Hongxu Yin. Pavlo Molchanov, and Jose Alvarez. 2021. Data-free knowledge distillation for object detection. In Proceedings of the JEEF/CVF

---

Sanjay Kariyappa, Atul Prakash, and Moinuddin Qureshi. 2021. Maze: Data-free model stealing at- tack using zeroth-order gradient estimation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 138 13814-13823.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications secu- rity, pages 506-519.
Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiao- jin Zhang. Qi Hu, Chunkit Chan, and Yanggiu Song. 2023a. Privacy in large language models: Attacks, defenses and future directions.
Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. 2018. Data distillation: Towards onni-supervised learning. In Proceedings of the IEEE conference computer vision and pat- tern recognition, pages 4119-4128.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yanggiu Song. 2023b. Multi-step jailbreaking pri- vacy attacks chatgpt. CoRR, abs/2304.05197.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:61
Paul Micaelli and Amos J. Storkey. 2019a. Zero-shot knowledge transfer via adversarial belief matching. In Advances in Neural Information Processing Sys- lems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 9547- 9557.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Paul Micaelli and Amos J Storkey. 2019b. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems, 32.
S Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume
1: Long Papers), ACL 2022. Dublin, Ireland, May 22-27, 2022, pages 3470-3487. Association for Com- putational Linguistics.
Aarohi Srivastava. Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Has- san Awadallah. 2023. Orca: Progressive learning from complex explanation traces of GPT-4. CoRR, abs/2306.02707.
OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
CoRR,
Mirac Suzgun. Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay. Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi. Denny Zhou, and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thoughh can solve them. CoRR, abs/2210.09261.
TB OpenAI. 2022. Chatgpt: Optimizing language mod- els for dialogue. OpenAl.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. 2019. Knockoff nets: Stealing functionality black- box models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pages 4963-
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction- followiig 1lama model. https:// gi thub. wwt. tttp:/tttsllabb/// stanfor__alpaca.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022, Training language models to follow instructions with human feedback. CoRR, abs/2203.02155.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint 2302

---

Bean-Baptiste Truong, Pratyush Maini, Robert Walls, and Nicolas Papernot. 2021. Data-free model extrac- tion. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4771 1770-
A Data Statistics
Table 6 and Table 7 show the data statistics of AGlEval and BIG-Bench Hard, respectively.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. CoRR, Siw/2305.19226
Task
# Examples
# Choices
AQuA-RAT
254
5
LogiQA
651
4
LSAT-AR
230
5
LSAT-LR
510
5
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR. abs/2212. 10.66.
LSAT-RC
269
5
SAT-Math
220
4
SAT-English
206
4
SAT-Fngllihh (w/o Psg.)
206
4
Table 6: Statistics of AGIEval dataset.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Task
Exumples
# Chuices
Boolcan Expressions
250
Causal Judgement
187
Dute Undersuadding
250
Disambiguation QA
250
Formal Fallacies
250
Geometric Shapes
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022b. Finetuned language models are zero-shot learners. The Tenth International Conference on Learning Represenia- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Hyperbaton
250
Logical Desuction objects)
250
Logical Deduction (7 objects)
250
Logical Deduction (3 objects)
250
Recommendarion
Navigate
250
Penguins Table
Reasoning about Colored Objjcts
250
18
Ruin Names
250
Salient Transiation Eroor Detection
250
Snarks
178
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022c. Emer- gent abilities of large language models. CoRR, abs/2206.07682.
Sports Understanding
Temperal Sequences
250
Tracking Shuffled Objects (5 objecrs)
250
Tracking Shufffed Objects (? objecrs)
250
Tracking Shufiled Objects (3 objects)
250
Lies
250
Table 7: Statistics of BIG-Bench Hard dataset.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter. Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022d. Chain-of-thout prompt- ing elicits reasoning in large language models. In NeurIPS.
B
Baselines
LLLMMA (Touvron et al., 2023) is collection of foundation language models ranging from 7B to 65B parameters. It is trained on trillions of tokens from publicly available datasets and is demonstrated outperform larger-size LLMs such as GPT-3 (175B) across multi- tude of benchmarks. We use the official code from LLaMA 4
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions.
Hongxu Yin, Pavio Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj Jha, and Jan Kautz. 2020. Dreaming to distill: Data- free knowledge transfer via deepinversion. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715-8724.
Alpaca (Taori et al., 2023) is a project initi- ated by Stanford University with the objec- tive of developing and disseminating open- source model that adeptly follows instructions. It is based on LLaMA and fine-tuned on 52K instruction-follooiing examples generated by
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A humam-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364

---

querying OpenAl's text-davinci-003 model. On the self-instruct evaluation set, Alpaca mirrors text-davinci-003, but is notably more compact and cost-effective to reproduce. We use the official code from Alpaca S.
Hyperparameter
Lion-7B
Lion-13B
Batch size
128
128
Learning
2e-5
2e-5
Epoches
3
3
Max length
1024
1024
Optimizer
AdamW
AdamW
Scheduler
onsine
onsine
WizardLM (Xu et al., 2023) employs LLMs instead of humans to automatically mass- produce open-domain instructions of various difficulty levels, improve the performance of L.MMs. uses an Evol-Instruct method to bootstrap the 52k instruction instruction-folloinngg ex- amples of Alapca into a larger set of 250k more intricate instructions. Out of this larger set, 70k examples were selected to fine-tune LLaMA. We use WizardL.--B-V1.0 from the official code
Weight decay
0
Warmup ratio
0.03
0.03
Table Training hyperparameters.
Querying the gpt-3.5-turbo API We use dif- ferent sets of hyperparameters when querying the gpt-3.5-turbo API for different roles (Teacher, Ref- eree, Generator). These hyperparameters are found to work well and we listed them in Table
temperature
top-P
tokens
Vicuna (Chiang et al., 2023), superior open- source chatbot, excels in generating fluid and captivating responses to user queries. It based on LLaMA and fine-tuned on 70K user-shared conversations collected from ShareGPT, platform designed for sharing interactions with ChatGPT. Its impressive pabilities make it one of the leading open instruction-follooingg models today. Vicuna achieves competitive performance against pro- prietary models such as ChatGPT and Bard (Google, 2023). We use (ocum--B-V1.J and Micuna-1B-VIL. from FastChat
1024
Table 9: Hyperparameters for querying OpenAl gpt- 3.5-turbo API under different roles.
D Prompt Templates for Our Adversarial Distillation Framework
Fine-tuning LLM (i.e. ChatGPT) is costly and intricate, human-tailored prompt templates are uti- lized to solve various tasks (Wei et al., 2022d; Chan et al., 2023b,c; Jiang et al., 2022: Jiayang et al., 2023; Chan and Chan, 2023). The prompt template of the Teacher for generating responses is shown in Table 10. The prompt template of the Referee for comparing the quality of two responses gener- ated by two AI assistants is shown in Table 11. The prompt templates of the Generator for generating new hard instructions and new easy instructions are shown in Table 12 and Table 13, respectively.
ChatGPT (OpenAI, 2022), product of Ope- nAI, is an advanced AI chatbot renowned for its ability to interact with users in an authenti- cally human and engaging manner. The chat- bot is built on powerful LLMs such as GPT- 3.5 and GPT-4, which are trained on a vast corpus of internet text data. ChatGPT un- dergoes fine-tuning via both supervised and reinforcement learning techniques, with the human trainers providing necessary feedback and direction.
E Case Studies
Here we show 3 cases in Table 14, 15, and 16 to clearly compare the open-ended generation perfor- mance among various models including our Lion- 13B, LLaMA-13B, Alpaca- Vicuna-1 13B, and ChatGPT.
C Implementation Details
Training Hyperparameters The training pro- cess is conducted on A 1100 GPUs. During each iteration of adversarial knowledge distillation, the hyperparameters for training are shown in Table 8.
Besides, we show cases in Table 17, 18, 19, 20, 21, and 22 to clearly compare the reasoning capabil- ity among various models including our Lion- 13B, Vicuna-133 and ChatGPT. We utilize / and X to denote whether the response is correct or incorrect. respectively.
'https://githes htttt:///ggihuub..cm/tttss-rlld htt..///taattu-laa/satafttaooeee
hhttps://github wttpp///.wwo//pp.ccc///////aaa/a
/tttps://github...

---

system content
You are helpful assistant that generates response to given task instruction.
#hh Instruction: (instruction)
content
##и Response:
Table 10: Prompt template of gpt-3.5-turbo for generating responses. Note that the original instruction in Alpaca is composed an instruction prompt and an instance input. For example, instruction prompt is "write an abstract about the following method", and the instance input is "knowledge distillation", For better adaption real world scenarios, we concatenate the instruction prompt and the instruction prompt into one instruction using a line break.
system
You helpful precise assistant checking the quality of the [Instruction] (instruction)
[The Start of Assistant 1's Answer] (answer_1)
[The End of Assistant I's Answer]
[The Start of Assistant 2's Answer]
(answer_2) _2)
[The End of Assistant 2's Answer]
[System]
We would like to request your feedback on the performance of two AI assistants in response to the user instruction and input displayed above.
user content
Please rate the helpfulness, relevance, accuracy, and level of detail of their responses. Each assistant receives an overall score scale of 1 to 10, where higher score indicates better overall performance.
Please first provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgrent. Then, output two lines indicating the scores for Assistant and 2, respectively.
Output with the following format: Evaluation evidence: syour evaluation explanation here> Score of the Assistant <score>
Score Assistant <score>
Table 11: Prompt template gpt-3.5-turbo for comparing the quality of two responses generated by two AI assistants,
system content
You are helpful assistant.
I want you to act as an Instruction Creator.
Your goal is to draw inspiration from the #Given Instructiont to create a brand new instruction.
This new instruction should belong to the same domain and the same task type as the #Given Instruction#.
The LENGTH and difficulty level of the #Created Instructione should be similar to that of the #Given Instructions.
The #Created Instruction: must be reasonable and must be understood and responded to by humans.
user content
'nGiven Instructiont', '#Created Instructiont', 'given instruction' and 'created instruction' are not allowed to appear in #Created Instructions.
#Given Instruction#:
(instruction)
#Created Instruction#:
Table 12: Prompt template of gpt-3.5-turbo for generating new hard instructions.

---

system content
You helpful assistant.
want act as an Instruction Creator.
Your goal is to draw inspiration from the #Given Instructiont to create brand new instruction.
This instruction should belong the same domain the #Given Instructiont rare.
but be The LENGTH and difficulty level of the #Created Instructions should be similar that of the #Given Instructions.
be reasonable and be understood and responded The #Created Instructiont by humans..
content
'#Given Instructiont', '#Created Instructiont', 'given instruction' and 'created instruction' are not allowed to appear #Created Instructionf.
#Given Instruction::
(instruction)
#Created Instruction#:
Table 13: Prompt template of gpt-3.5-turbo for generating new easy instructions.

---

Instruction: If the endpoints of line segment ure (2, 2) and (10, 4). what is the length of the segment?
To find the length of the line segment with endpoints (2, 22) and (10, 4), formula. The distance formula derived from the Pythagorean theorem and is given as:
the distance
v(22 x1 2 + (02 y1)2)
where (xl,yl) and (x2, are the coordinates of the endpoints of line segment.
Substituting the given values, we get:
v (10 2)
2)
Lion
Simplifying this expression, we get:
((8)) + 69)
v(64 + 36)
(100)
= 10
Therefore, the length of the line segment with endpoints (2) and (10, 4) is 10 units.
ILaMA
The length of the line segment units.
Alpaca
The length of the line segment is units. X
The length of the line segment the distance between the two points. To find the length, you use the Pythagorean theorem, which states that the sum of the squares of the distances from point to the endpoints is equal to the square of the distance between the two endpoints.
In this case, the distance from the point (2,- 22) to the point (10, is given by the formula:
v ((22 =1)² (y2 =
Vicuna
V(110- - -4-
V(18² +
((224 16)
(340)
17.973
So the length of the line segment is 17.973. X
To find the length line segment two endpoints (æ1, yl) and (x2, y2), we the distance formula:
distence
V (a2 x1)a + (y2 y1)a
Plugging in coordinates the given endpoints, we get:
distance V((10- 2) (4 (-2))²)
ChatGPT
distance V(8 + 62)
distance = /(4 + 36)
distancx = / /1100)
distance = 10
Therefore, the length of the line segment is 10 units.
Table 14: A math case from Iicuut-nnstructions.

---

Instruction: What if Isaac Newton had focused on biology instead of physics?
Isaac Newton is renowned for his contributions to physics, particularly his laws of motion and universal gravitation. However, what if Newton had instead chosen to focus on biology?
Newton's work in physics was heavily influenced by his fascination with the natural world and his desire to understand how it worked. Similarly, had he chosen to study biology. he may have made significant contributions to the field. Newton's mathematical abilities and his interest in the natural world would have made him formidable biologist.
One area where Newton may have made a significant impact in biology is in the field of tax- onomy. Newton was keen ohserver of nature and had particular interest in plants. He spent much of his time studying and cataloguing the various species of plants around him. He even developed system for classitying plants hased on their physical characteristics.
Lion
In addition to his work in taxonomy, Newton may have also made important contributions the field of physiology. He interested in the workings of the human body and may have studied the mechanics of muscles and the circulatory system.
Overall, while Isaac Newton most well-known for his work in physics, his interests and abilities would have made him formidable biologist well. Who knows what discoverics he might have made had he chosen to focus on the natural world in different way.
LLaMA
I'm not sure. he did so, he wouldn't have discovered the law of gravity.
If Isaac Newton had focused biology instead of physics, he might have made important contributions the field of biology, such as developing a scientific theory of evolution explaining the processes of cell growth and development. He might also have studied and discovered new principles genetics and developed methods study and treat discases.
Alpaca
Isaac Newton widely regarded as one of the most influential scientists in history for his work physics, particularly for development the laws of motion and universal gravitation. However, is worth noting that Newton also made significant contributions to other fields, including mathematics, optics, and natural philosophy.
If Newton had decided to focus on biology instead of physics, it is possible that he would have made significant contributions to the field. However, it is difficult to say exactly what those contributions would have been without knowing more about his specific interests and areas of focus.
Vicuna
Newton was a prolific scientist and wrote extensively on a wide range of topics, including nat- ural philosophy. mathematics, and theology. He also conducted extensive research in optics, including the study of light and color, and made important contrihutions to the field of mathematics, including the development of calculus.
It is possible that if Newton had focused on biology, he would have made significant contribu- tions to the understanding of living organisms and their behavior. However, it is also possible that his work in physics and mathematics would have had a greater impact on the world.
If Isaac Newton had focused on biology instead of physics, the scientific world may he vastly different today. His advancements in mathematics, such as calculus, would have certainly been useful in the field of biology. Newton's curiosity, analytical mind, and dedication scientific inquiry would have been equally effective in biology as they were physies. He may have made significant contributions the field of genetics or evolutionary theory.
ChatGPT
On the other hand, his contributions physics were so groundbreaking that the subject may have been hindered without his contributions. It's difficult say exactly what would have happened, but it's likely that the course of scientific history would have been different.
Table 15: A counterfactual case from Vicuna-Instractions..

---

Instruction: How can develop my critical thinking skills?
Developing critical thinking skills important aspoct of persoeal growth and professional development. Critical thinking enables us analyze and evaliure information, ideas, and arguments objectively, make informed decisions, and solve complex problems. Here are some stralegies can help you develop your critical thinking skills:
1. Read widely and deeply: Reading widely exposes you different perspeetives and ideas, while reading deoply helps you to understand and analyze complex concepts. Choose books that are challenging and outside your cocnfort zone, and take notes as you read.
2. Ask questions: Asking questions is an essential part of critical thinking. Start by asking whyand howquestions to gain desper understanding a lopsc. Then,
ifiuessiios explore possible outeomes consequences.
3. Pmactice analysis and evaluation: Critical thinking involves analyzing and evalunting information and arguments. Start
Lion
by breaking down complex ideas into smaller parts, and then evaluate each part based on evidence, reasoning, and credibilitt.
4. Seek diverse perspectives: Surround yourself lives. Listen to their ideus and opinioes, and challenge your own ussumptions and biases.
people who have different hackgronds, experiences, and perspec-
5. Practice decision making: Critical thinking is essential in decision making. Practice making decisions by weighing the pros and cons, considering potential outcomes, and seeking advice from others.
6. Take courses and workshopp: Enrolling in courses and workshops that focms on critical thinking, problem-solvingg and decision making can provide you with the tools and techniques to develop your critical thinking skills.
implementing these strategies, you develop your oritical thinking skills and hecome better equipped analyze. evaluate, and make informed decision in your personal and professional life.
ways you developing your oritical 1. Analyze and interpeet: When you'se re presented with information, to analyze and interpret your terms. When you
Critical thinking vital skill have. hot he difticult develop. Here
thinking skills:
to analyze and interpret information on your own, will become easier to do so over time.
2. Assess your information: When you receive new information, to astess whether it's valid or reliable. This impoctant for
LLaMA
evalusuing and forming your own opinions.
3. Use your intuition: Use your intuition to complement your critical thinking skills, When you pay attention to your gut feelings. they'll Il usually tell you when you need to double check your information.
4. Leam from your mistakes: When you make mistake, try not to heat yourself up about it. Instead, learn from your mistakess and use them to your advaniage.
Developing critical thinking skills involves actively questioning assumptioon, weighing evidence, and tinking analytically. It also involves being open minded and considering various perspectives, well thinking logically and systematically. Practicing problem-solivngg and creative thinking. working through socsarios and dilemmas, and eogaging in thoughtful convensation are all ways to improve critical thinking. Addirionally, reading. reflecting, and writing can help increase your capacity for critical thinking.
Alpaca
Developing your critical thinking skills can help you better analyze information, make more informed decisions, and solve complex problems. Here
tips for improving your critical thinking skills:
I. Practice active listoning: Pay attention to what others are saying and try understand their perspoctive.
2. Ask questions: Be curious and ask questions to clarify information and better understand a situation.
3. Analyze information:
time carcfully analyae information
consider different viewpoints.
4. Seek our diverse viewpoints: Expose yourself a variety of viewpoints and idens to challenge your own assumptions and biases.
5. Practice logic and reasoning: Use logic and reasoning evaluate anguments and make decisions.
Vicuna
6. Lear from your mistakes: Be open to learning from your mistakes and using them as opportunities for growth.
7. Read and watch variety of sources: Read and watch variety of sources expand your knowledge and expose yourself to different perspectives.
8. Engage in debates and discussions: Engage in debutes and discussions with others to practice critical thinking and develop your skills.
9. Take breaks and reflect: Take breaks and reflect on your thinking process to improve your critical thinking skills.
10. Practice mindialness: Mindilluess can help be more aware of your thoughts and bisses, which can help you think more critically.
Sure. here a few practical tips to develop critical thinking skills:
I. Ask questions: Don't it simply accept information at face value. Ask questions that allow you to evalunte the reason, perspcetvve and logic behind the information.
2. Analyze: Take the time to analyae and evaluate the information that you receive. This will help you to ideniffy as-
sumptions, biases, strengths and waaknesses argument.
3. Consider different perspectives: When analyzing information, try to consider different perspectives gain wider derstanding of any topic.
4. Research and read extensively: Read wide range sources to gel betier understanding of different viewpoints and opinions.
ChatGPT
5. Practice logic and reasoning: Practice thinking thoogh problcmn logically by breaking them down into smaller parts.
6. Esrablish a point view: Consider developing an opinion thesis statement, and then colloct evidence to support that position.
7. Reflect on your own thinking: Consider how you arrived at conclusion, and whether you have any hiases or assump- tioms that may have influenced your thinking.
With time, and practicing these skills consistently, you become a better critical thinker.
Table 16: A generic case from Vicuna- Instructions.

---

Instruction: Q: If 120 reduced 96, what is the reduction percent? Answer Choices: (A)30% (B)40% (C)20% (D)10% (E)5% A: Among A through E, the answer is
To calculate the reduction percent, we need to divide the difference between the original number and the reduced number by the original number and then multiply by 100.
Here, the original number is 120 and the reduced number is 96. So, the difference between the two numbers is 120 - 96 24.
To calculate the percentage reduction, divide the difference by the original number and then multiply by 100:
Lion
(24 / 120) = 20%
Therefore, the correct answer is option (C) 20%.
Option (A) 30%, option (R) 40%, option (D) 10%, and option (F) are incorrect.
Vicuna
A. 30% X
To calculate the reduction percentage, we need to find the difference between the original value and the reduced value, and then divide it by the original value.
The difference between 120 and 96 120 - 96 24.
ChatGPT
Now, divide the difference by the original value and multiply by 100 to get the percentage re- duction:
(24 / 120) 100 = 20%
Therefore, the
is 20%. /
Table 17: An AQuA-RAT from AGIEval.
Instruction: Q: not ( True and ( True is? Answer Choices: (A) True (B) False A: Among A through B, the
asswer
The expression "not (True) and (True)" can be simplified "(not True) and (True)" "False and True". The "and" operator has higher procedence than the "not" operator, the first expression is evaluated "False and True". which is false.
Lion
Therefore, the correct answer is option (B) False.
(B) False
### Explanation:
This statement is false.
Vicuna
the given statement is "not (True)" and "True", it would be equivalent to "not (True) and True", which would he true.
(This is because negation of "True" is "False", and "True" and "False" evaluate to "True".)
Therefore, the correct answer is (A) True. X
ChatGPT
(A) True. X
Table 18: A boolean expressions case from BIGBench Hard.
Q: If you follow these instructions, do you return to the starting point? Take 8 steps. Turn around. Take 8 steps. Answer Choices: (A) Yes (B) No A: Among through B, the answer is
Instruction:
According the instructions given, if you follow them, you will return the starting point. Therefore, the correct answer Option "Yes". By taking 8 steps and turning around, you will up back where you started. It important follow instructions carefully achieve the desired outcome. In this case, following these instructions will that you return the starting point.
Lion
Vicuna
(A) Yes
ChatGPT
(B) No X
Table 19: navigate case from BIGBench Hard.

---

Instruction: Q: Today is Christmas Eve of 1937. What is the date tomorrow in MM/DD/YYYY? Answer Choices: (A) 12/11/1937 (B) 12/25/1937 (C) 01/04/1938 (D) 12/04/1937 (E) 12/25/2006 (F) 07/25/1937 A: Among A through F, the answer is
The correct answer is (B) 12/25/1937.
To arrive at this answer, we need to consider the fact that Christmas Eve is the evening before Christmas Day. Therefore, the date of Christmas Day is one day after Christmas Eve. this case, the date of Christmas Eve is 12/24/1937, and the date of Christmas Day is 12/25/1937. Therefore, the date lomorrow in MM/DD/YYYY format is 12/25/1937.
Option (A) is incorrect because it gives the date as 12/11/1937, which is incorrect.
Lion
Option (C) is incorrect because it gives the date as 01/04/1938, which is incorrect.
Option (D) is incorrect because it gives the date as 12/M/1937, which is incorrect.
Option (E) is incorrect because it gives the date as 12/25/2006, which is incorrect.
Option incorrect because gives the date as 07/25/1937, which incorrect.
(A) 12/11/1937
Vicuna
### Explanation:
The date tomorrow, on Christmas Eve of 1937, would be 1/1937. X
ChatGPT (C) 01/04/1938. X
Table 20: A data understanding from BIGBench Hard.
Instruction: Q: Which statement sarcastic? Answer Choices: (A) Hey just be happy then you won't it be depressed anymore (B) Hey just be happy that you won't be depressed anymore A: Among A through B. the answer is
The correct answer is (B). The statement "Hey just be happy that you won't be depressed anymore" is sarcastic because it implies that being happy is an casy and straightforward solution to depression, which is complex mental health issue that requires professional help and support. The use of the word "just" also adds dismissive tone to the statement, suggesting that the speaker doesn't 't take depression seriously, It's important to avoid using sarcasm when talking about mental health issues, it can be hurtful and invalidating to those who are struggling. Instead, it's important to offer compussion and support and to encourage individuals to seck professional help if needed. X
Lion
Vicuna
(B) Hey just be happy that you won't 'tn't be depressed anymore X
ChatGPT
sarcastic statement (A) "Hey just happy then you won't 't depressed anymore."
Table 21: A snarks case from BIGBench Hard.
Instruction: Q: the following sentences, explain antecedent of pronoun (which thing pronoun refers to), or state that is ambiguous. Sentence: The scientist collahorated with the artist, he shared story. Answer
Choices: (A) The scientist shared story (B) artist shared story (C) Ambiguous A: Among through the answer
the given sentence, the antecedent of the pronoun "he" ambiguous. "He" could refer cither the scientist or the artist, as both them shared a story. Therefore, option (C) is the correct answer.
Lion
(C) Ambiguous
### Explanation:
Vicuna
The pronoun "he" is ambiguous, as could refer to either the scientist or the artist. Without more context, it is impossible to determine which person is being refered to. Therefore, the answer is
"C- Ambiguous."
(C) Ambiguous.
ChatGPT
In the given sentenee, the antecedent of the pronoun "he" is unclear. It could refer to either the scientist or the artist because there is no clear indication of which person shared the story. Therefore, the sentence ambiguous.
Table 22: A disambiguation QA case from BIGBench Hard.


Reviews:
[{"review_id": "11q17Xk4qb", "reviewer": "Reviewer_THki", "summary": "", "questions": "A) In terms of language modeling capabilities (expressability, creativity), how does Lion differ from ChatGPT or LLaMA? \nB) Are the authors recommending that Lion can be used as a substitute for ChatGPT or LLaMA? If not, what are the authors recommending in terms of the use of Lion?\nC) Are there potential risks of deploying Lion as a ChatGPT alternative (e.g., more instances of text degeneration phenomena like hallucination, length bias, repetition, copying, etc.) - asking as far as potential for these drawbacks (particularly so these could be mentioned in the paper); a thorough evaluation is not necessary.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We’re glad that you found our paper novel, convincing, and exciting.\n\n> Q1: In terms of language modeling capabilities (expressability, creativity), how does Lion differ from ChatGPT or LLaMA? \n\nAssessing the intricate aspects of expressability and creativity in language models remains challenging, often confined to limited text-based evaluations [1].\nBased on Margaret Boden's definition of creativity—“the ability to come up with ideas or artifacts that are *new*, *surprising* and *valuable*” [2], recent analysis has explored the creative potential of Large Language Models (LLMs), suggesting that while they achieve value and a modest level of novelty and surprise, their autoregressive nature hinders transformative creativity [3].\n\nGiven that Lion, ChatGPT, and LLaMA share the autoregressive Transformer architecture, inherent limitations exist in their expressivity and creativity. Nonetheless, our empirical observations, exemplified by the counterfactual case in Table 13, indicate that Lion and ChatGPT may exhibit higher “expressivity” and “creativity” than LLaMA. This divergence could be attributed to further fine-tuning with instructional guidance.\n\n\n> Q2: Are the authors recommending that Lion can be used as a substitute for ChatGPT or LLaMA? If not, what are the authors recommending in terms of the use of Lion? \n\nBased on our extensive experiments (Table 1, 2, 3, and Figure 4), we advocate Lion as a viable alternative to ChatGPT in scenarios not requiring intricate reasoning, such as tasks involving general explanations (e.g., \"Explain quantum computing basics\") or practical operations (e.g., \"Design a recipe to bake a chocolate cake\"). Note that the Lion's lightweight nature and the ability to deploy locally presents several *practical* advantages for many applications scenarios, due to its resource efficiency, rapid inference speed, and far less concern in data security and privacy, compared with ChatGPT. However, we would like to highlight that evaluating the performance of large language models across various tasks is still an open problem (esp. in term of comprehensiveness and fairness); our experimental findings may serve as beneficial results towards this goal. \n\n\n> Q3: Are there potential risks of deploying Lion as a ChatGPT alternative? \n\n- **Security Issue:** As we mentioned (Lines 647-661), Lion does not utilize technics such as Reinforcement Learning from Human Feedback (RLHF) to reinforce the model to align with common values. Thus, it may exhibit unsafe output, and may be more vulnerable to certain attacks (e.g., prompt injection attack). \n- **Hallucination:** Lion may exhibit hallucination inherited from the larger teacher LLM (LLaMA in our paper). To investigate, our supplemental evaluation on TruthfulQA [4] reveals that while Lion-13B and LLaMA-13B exhibit comparable accuracy (46.5% vs. 47.0%), both fall behind ChatGPT (62.0%). Therefore, caution is advised in relying on Lion for critical decisions due to possible content fabrication. \n- **Length Bias:** Notably, Lion tends to produce lengthier responses than its base model LLaMA, as observed in Appendix E, Tables 12-14. This inclination might arise from Lion's training with hard instructions demanding increased response length.\n\nIn the final version, we will include the above insights and additional case studies for a more comprehensive assessment.\n\n\n> Presentation Improvements\n\nThanks for your valuable suggestions. We will remedy all the points mentioned according to your advice.\n\n\nReferences:\n\n[1] Putting GPT-3's Creativity to the (Alternative Uses) Test. Stevenson et al. ICCC 2022.\n\n[2] The Creative Mind: Myths and Mechanisms. M. A. Boden. Routledge 2003.\n\n[3] On the Creativity of Large Language Models. Franceschelli et al. arXiv 2023.\n\n[4] TruthfulQA: Measuring How Models Mimic Human Falsehoods. Lin et al. ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a new method (inspired by but distinct from previous work) to distill knowledge from ChatGPT or LLaMA to a smaller model requiring far less data and no extra human annotation.  The details of the method are presented in such a way that the paper is reproducible and the pitch is convincing.  The content and promise of the paper is exciting and likely of wide interest to the NLP community.   The experiments and evaluation is strong and covers a range of tasks and settings.", "reasons_to_reject": "- The final two paragraphs of the introduction (Line 063-104) do not seem as carefully written as the rest of the paper. Particularly, there are quite a few instances in which the writing is vague; because of this, the text does not successfully serve the purpose it is meant to. I believe this could be remedied by revising the writing. Examples include: “forcing it to learn” (Line 065), “unparalleled role adaptability” (Line 074); generally, the overview of the method has vague phrases that make it difficult to understand.  \n\n- As the authors quote in the limitations section, “model imitation is a false promise” (Gudibande et al., 2023); however, the authors do not mention this in the main text of the paper.  Though the evaluation section is strong, beyond these settings, it is unclear from the paper itself what the potential drawbacks of this method are — for example, from a language modeling perspective, how does Lion differ from ChatGPT? Are the authors recommending that it can be used as a substitute, and if so, what are the potential drawbacks? A careful description of the pros and cons of the method and the difference in the end result from ChatGPT/LLaMA will clarify to future readers how this approach is best used and what contexts would be a inappropriate. ", "questions_to_authors": "A) In terms of language modeling capabilities (expressability, creativity), how does Lion differ from ChatGPT or LLaMA? \nB) Are the authors recommending that Lion can be used as a substitute for ChatGPT or LLaMA? If not, what are the authors recommending in terms of the use of Lion?\nC) Are there potential risks of deploying Lion as a ChatGPT alternative (e.g., more instances of text degeneration phenomena like hallucination, length bias, repetition, copying, etc.) - asking as far as potential for these drawbacks (particularly so these could be mentioned in the paper); a thorough evaluation is not necessary.", "paper_topic_and_main_contributions": "This paper addresses the problem of the inaccessibility of proprietary large language models and presents a method (inspired by but distinct from previous work) to distill knowledge from a large language model like ChatGPT or LLaMA to a smaller model requiring far less data and no extra human annotation.  This paper provides contributions in the following areas: NLP engineering experiments, approach for data- and compute efficiency, and open source pre-trained models.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Figure 1 does not successfully clarify the difference between this approach and previous approaches.  \n- Figure 2: The caption or in-figure labels should be a little more verbose; for example, symbols/variables should be defined here as well.\n- The method seems different from adversarial knowledge distillation; in fact, the use of “adversarial” throws me off a little bit in terms of my first impression vs. the content of the paper, so I would ask the authors to consider if they think it is valuable to emphasize the AKD method less and to make it clearer how their method either is different or how it fits with the AKD paradigm."}}, {"review_id": "03HxQovJzD", "reviewer": "Reviewer_nhXy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We’re glad that you found our paper innovative, convincing, and adaptive.\n\n> Q1: The selection of some parameters, such as the threshold τ for discriminating difficult and easy commands and the ratio of difficult and easy commands generated, is not discussed and analyzed in detail.\n\nIn response, we have conducted a series of ablation studies based on Lion-7B, meticulously investigating the impact of various hyperparameters.\n\n- **The threshold $\\tau$ for distinguishing between hard and easy instructions:** We systematically explored $\\tau$ ranging from 0.0 to 2.0 and documented its influence on average performance across three datasets. Our findings reveal an optimal range of $\\tau$ between 1.0 and 1.5 for all datasets. Notably, elevating $\\tau$ from 0.0 to 1.0 consistently enhances performance across all datasets, indicating effective differentiation between hard and easy instructions. However, a continuous increase from 1.0 to 2.0 gradually degrades performance due to decreased diversity in hard instructions. The ablation results demonstrate that our method is not quite sensitive to a large value of $\\tau$.\n\n    | $\\tau$ | Vicuna-Instructions (Avg.)| AGIEval (Avg.) | BBH (Avg.) |\n    |:--------:|:-----:|:----------:|:---------:|\n    | 0.0 | 89.58 | 22.4 | 26.5 |\n    | 0.5 | 92.16 | 23.5 | 29.8 |\n    | 1.0 | 93.81 | **26.3** | **32.0** |\n    | 1.5 | **94.09** | 25.7 | 31.6 |\n    | 2.0 | 92.23 | 24.6 | 31.3 |\n\n- **The ratio of generated hard and easy instructions:** We change the ratio of generated hard instructions to generated easy instructions from 1:0 (all hard) to 0:1 (all easy) and investigate its impact on average performance across three datasets. It can be seen from the following table that higher ratios of hard to easy instructions generally lead to improved performance, with a balanced ratio of 1:1 yielding the highest average scores.\n\n    | Ratio of hard to easy | Vicuna-Instructions (Avg.)| AGIEval (Avg.) | BBH (Avg.) |\n    |:--------:|:-----:|:----------:|:---------:|\n    | 1:0 | 89.60 | 24.3 | 30.8 |\n    | 2:1 | 92.95 | 25.7 | **33.1** |\n    | 1:1 | **93.81** | **26.3** | 32.0 |\n    | 1:2 | 91.77 | 23.9 | 29.6 |\n    | 0:1 | 90.02 | 22.1 | 24.3 | \n    \n\n> Q2: The discussion of the model's capability, training process and evaluation metrics is rather brief and lacks in-depth analysis and discussion.\n\nWe have provided a comprehensive discussion in the Limitation Section (Lines 565-629):\n- **Model Capability:** We have detailed Lion's limitations in handling reasoning, multi-turn conversations, and processing lengthy documents, along with corresponding explanatory insights.\n- **Training Process:** We demonstrate the cost-effectiveness of Lion's training through a comparative analysis with WizardLM. We acknowledge the trade-off of longer training times due to iterative parametric updates.\n- **Evaluation Metrics:** We emphasize concerns surrounding GPT-4's utilization in automated assessments and advocate for a robust, standardized evaluation framework for chatbots.\n\nOverall, we believe the ablation study and the discussion of limitation are minor points that do not dismiss the main claims supported in the paper. We are grateful for the opportunity to address these concerns and sincerely hope the reviewer will reevaluate the potential contribution of our work in light of the clarification provided.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Presents an innovative adversarial distillation framework that enables efficient knowledge transfer by modeling model differences between teacher models and student models.\n(2) Extensive experiments on multiple open generative and reasoning benchmarks demonstrate the superior performance of the Lion model.\n(3) It has a wide range of applications, not only for ChatGPT, but also provides a convenient knowledge transfer method for other proprietary LLMs.", "reasons_to_reject": "(1) The selection of some parameters, such as the threshold τ for discriminating difficult and easy commands and the ratio of difficult and easy commands generated, is not discussed and analyzed in detail.\n(2) The discussion of the model's capability, training process and evaluation metrics is rather brief and lacks in-depth analysis and discussion.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel adversarial distillation framework for transferring a complex private large-scale language model (LLM) to a compact open-source LLM.By exploiting the versatile role adaptations of LLMs, the framework is able to motivate the teacher model to recognize \"difficult\" instructions and generate new \"difficult\" instructions for the student model, thus creating a three-stage adversarial loop. By utilizing the LLMs' versatile role adaptations, the framework is able to prompt the teacher model to identify \"difficult\" instructions and generate new \"difficult\" instructions for the student model, creating a three-stage adversarial cycle. By applying this adversarial framework, the authors successfully transferred knowledge from ChatGPT to a student model (named Lion) using only 70k of training data. Experimental results show that Lion-13B not only achieves a level comparable to ChatGPT in terms of open-ended generative power, but also outperforms traditional state-of-the-art models (SOTA) such as Vicuna-13B in challenging zero-sample reasoning benchmarks such as BIG-Bench Hard and AGIEval.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wmlqjTDvUi", "reviewer": "Reviewer_XLBf", "summary": "", "questions": "1. This is not strictly knowledge distillation. Would it not be better to term it as adversarial data augmentation ?\n\n2. The statement \"our Lion-13B approximates ChatGPT’s open-ended generation capabilities\" is a big claim and needs evidence. How do you support your statement ?\n\n3. On Table 1, the Lion-13B on setting 2 shows a relative response quality of 100.18. Is this because GPT4 scores some Lion responses to be better than ChatGPT ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and detailed reviews, we hope the following comments could address your concerns.\n\n> Q1: Lack of an ablation study which can demonstrate sensitivity of hyperparameters as well as the value of augmentation vs feedback (discrimination). \n\nWe have conducted a series of ablation studies based on Lion-7B, meticulously investigating the impact of various hyperparameters.\n\n- **The threshold $\\tau$ for distinguishing between hard and easy instructions:** We systematically explored $\\tau$ ranging from 0.0 to 2.0 and documented its influence on average performance across three datasets. Our findings reveal an optimal range of $\\tau$ between 1.0 and 1.5 for all datasets. Notably, elevating $\\tau$ from 0.0 to 1.0 consistently enhances performance across all datasets, indicating effective differentiation between hard and easy instructions. However, a continuous increase from 1.0 to 2.0 gradually degrades performance due to decreased diversity in hard instructions. The ablation results demonstrate that our method is not quite sensitive to a large value of $\\tau$.\n\n    | $\\tau$ | Vicuna-Instructions (Avg.)| AGIEval (Avg.) | BBH (Avg.) |\n    |:--------:|:-----:|:----------:|:---------:|\n    | 0.0 | 89.58 | 22.4 | 26.5 |\n    | 0.5 | 92.16 | 23.5 | 29.8 |\n    | 1.0 | 93.81 | **26.3** | **32.0** |\n    | 1.5 | **94.09** | 25.7 | 31.6 |\n    | 2.0 | 92.23 | 24.6 | 31.3 |\n\n- **The ratio of generated hard and easy instructions:** We change the ratio of generated hard instructions to generated easy instructions from 1:0 (all hard) to 0:1 (all easy) and investigate its impact on average performance across three datasets. It can be seen from the following table that higher ratios of hard to easy instructions generally lead to improved performance, with a balanced ratio of 1:1 yielding the highest average scores.\n\n    | Ratio of hard to easy | Vicuna-Instructions (Avg.)| AGIEval (Avg.) | BBH (Avg.) |\n    |:--------:|:-----:|:----------:|:---------:|\n    | 1:0 | 89.60 | 24.3 | 30.8 |\n    | 2:1 | 92.95 | 25.7 | **33.1** |\n    | 1:1 | **93.81** | **26.3** | 32.0 |\n    | 1:2 | 91.77 | 23.9 | 29.6 |\n    | 0:1 | 90.02 | 22.1 | 24.3 |\n\n\n> Q2: This is not strictly knowledge distillation. Would it not be better to term it as adversarial data augmentation? \n\nWe understand your concern since our approach involves transferring knowledge from larger models to smaller ones through adversarially generated instructions, resembling data augmentation. Yet, it's important to consider that within the realm of proprietary Large Language Models (LLMs), the lack of accessible model parameters has led to data generation (augmentation) becoming an inherent aspect of knowledge distillation [1,2]. In this light, we propose that our method aligns with the concept of sequence-level knowledge distillation [3], where the content of distillation is in the form of an instruction task.\n\n\n> Q3: The statement \"our Lion-13B approximates ChatGPT’s open-ended generation capabilities\" is a big claim and needs evidence. How do you support your statement? \n\nThe statement is supported by our experimental results. We evaluate Lion-13B on the widely adopted Vicuna-Instructions dataset (Lines 381-403), covering 9 varied task categories. Table 1 reveals Lion-13B's response quality at 96.57% and 100.18% relative to ChatGPT in distinct setups.\n\nHowever, we would like to highlight that evaluating the performance of large language models across various tasks is still an open problem (esp. in term of comprehensiveness and fairness). While our experimental findings contribute to this endeavor, we will further clarify this statement's scope in the final version.\n\n\n> Q4: On Table 1, the Lion-13B on setting 2 shows a relative response quality of 100.18. Is this because GPT4 scores some Lion responses to be better than ChatGPT? \n\nYes, within setting 2, GPT-4 assigns superior scores to Lion-13B over ChatGPT in 52.5% of Vicuna-Instruction samples. Please refer to Appendix E for compelling case studies that further confirm Lion-13B's potential for generating superior responses in comparison to ChatGPT.\n\n\n> Typos and References\n\nThank you for your detailed review. We will fix the typos and include all references you mentioned in our final version.\n\n\nReferences:\n\n[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions. Wang et al. ACL 2023.\n\n[2] Large Language Models Can Self-Improve. Huang et al. arXiv 2022.\n\n[3] LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions. Wu et al. arXiv 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple approach inspired from Adversarial KD which combines data augmentation and feedback from the LLM to improve on \"harder\" instructions.\n\n2. Performance upgrade over similarly sized models.\n\n", "reasons_to_reject": "1. Lack of an ablation study which can demonstrate sensitivity of hyperparameters as well as the value of augmentation vs feedback (discrimination)\n", "questions_to_authors": "1. This is not strictly knowledge distillation. Would it not be better to term it as adversarial data augmentation ?\n\n2. The statement \"our Lion-13B approximates ChatGPT’s open-ended generation capabilities\" is a big claim and needs evidence. How do you support your statement ?\n\n3. On Table 1, the Lion-13B on setting 2 shows a relative response quality of 100.18. Is this because GPT4 scores some Lion responses to be better than ChatGPT ?", "paper_topic_and_main_contributions": "This paper presents a method for knowledge transfer from a large proprietary language model such as ChatGPT to a smaller open source model using model imitation. Motivated by Adversarial Knowledge Distillation, they discriminate between easy and hard instructions, sample from the hard ones and generate new instructions based on them and train on these in the next iterations. They also sample, generate and train from the easy ones to prevent catastrophic forgetting.\n\nThey demonstrate better performance compared to Vicuna, Alpaca and Llama model on different tasks and datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. https://aclanthology.org/2021.emnlp-main.526.pdf\n2. https://aclanthology.org/2021.acl-long.86/", "Typos_Grammar_Style_and_Presentation_Improvements": "The word \"quantificationally\" on line 289-290 should be quantitatively."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of the inaccessibility of proprietary large language models and presents a method (inspired by but distinct from previous work) to distill knowledge from a large language model like ChatGPT or LLaMA to a smaller model requiring far less data and no extra human annotation. 

Pros:

Presents an innovative adversarial distillation framework that enables efficient knowledge transfer by modeling model differences between teacher models and student models. It combines data augmentation and feedback from the LLM to improve on "harder" instructions.

Extensive experiments on multiple open generative and reasoning benchmarks demonstrate the superior performance of the Lion model.

Cons:

More ablation studies (some of them have been added by the authors during discussion/rebuttal)

Mor in-depth analysis/discussion on the model's training process and evaluation metrics

The original work  will also benefit from incorporating the additional discussion points/experiments conducted during rebuttal.
Decision: Accept-Main