Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
Authors: Hang Zhou
Yehui Tang
Haochen Qin
Yujie Yang
Renren Jin
Deyi Xiong
Kai Han
Yunhe Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language models
Data
Instruction-Tuning
Primary area: natural_language_processing
TL;DR: We propose an automated framework for dataset optimization, which, upon employing the refined datasets for instruction-tuning of LLMs, has demonstrated a performance enhancement of approximately 12% on evaluation sets such as MT-bench.

Abstract: The efficacy of large language models (LLMs) on downstream tasks usually hinges on instruction tuning, which relies critically on the quality of training data. Unfortunately, collecting high-quality and diverse data is both expensive and time-consuming. To mitigate this issue, we propose  a novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. The framework adopts a three-pronged strategy. It  initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. Subsequently, the generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality. Finaly, the above process evolves in a dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework. Optimized datasets have achieved substantial improvements, with an average increase of 12\% and notable gains in specific metrics, such as a 40\% improvement in Fermi, as evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset. Codes will be released soon.

Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
Hang Zhou 1.2, Yehui Tang', Haochen Qin', Yujie Yang', Renren Jin', Deyi Xiong Kai Han?", Yunhe Wang"*
'College of Intelligence and Computing, Tianjin University, Tianjin, China. Huawei Noah's Ark Lab.
(zhouhang25, yehui.tang. qinhaochenl, yangyujie26) @ @huawei.com, (mjin, dyxiong @ @tju.edu.cn, (kai kai.han, yunhe.wang @ @huawei.com.
Abstract
The efficacy of large language models (LLMs) on downstream tasks usually hinges on instruction tuning, which relies critically on the quality of training data. Un- fortunately, collecting high-quality and diverse data is both expensive and time- consuming. To mitigate this issue, we propose novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi- agent collaboration and assessment. The framework adopts a three- three-pronged strategy. It initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. Subsequently, the generated data undergo a rigorous evaluation using dual-model method that assesses both difficulty and quality. Finaly, the above process evolves in dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework. Optimized datasets have achieved substantial improvements, with an average increase of 12% and notable gains in specific metries, such 40% improvement in Fermi, evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset. Codes will be released soor
Introduction
The research and development of natural language understanding and generation have been dramat- ically accelerated with the emergence and prevalence of LLMs [39 31 30). These models have been extensively applied in wide range of scenarios, c.g., question answering and text generation, significantly enhancing downstream task performance due to their exceptional ability to follow instructions 3
10 28) Such instruction- followiin capability is primarily acquired through a process known as instruction tuning 40 23 5), where LLMs are fine-tuned on instruction data.
53
49
hence widely acknowledged that the quality of instructions plays pivotal role [5] 20 48 29)
Historically, the creation of instruction data for training LLMs has heavily relied on the expertise of human annotators, as evidenced by substantial scholarly contributions [14 50 41 38
27 21
While expert-driven data generation assures the production of high-quality instructions, the enormous volume of data necessary for effective training renders this method economically untenable. In response, recent efforts have shifted towards the utilization of LLMs to automatically generate instructions, thereby mitigating the reliance on costly human annotation 37
32
44
18]. Concurrently,
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

there is growing emphasis on the generation and selection of challenging examples, grounded in the belief that more complex and difficult instructions can substantially elevate model capabilities 22 17
Despite the clear advantages of using L.MMs for data generation, several challenges persist in this strategy. Primarily, previous efforts often depend on a single LLM, resulting in data that may lack stylistic variety 44] and encompass a limited range of difficulty levels which may not be ideal for all models, Additionally, there is a trend towards the creation of exceedingly complex instructions [99 18), which may surpass the operational capabilities of models with small parameter scale, thereby hindering their ability to fully capitalize on the data's 's potential for performance enhancement.
To address the aforementioned challenges, propose the Star-Agents framework, advanced automatic data optimization system specifically designed to learn and refine instruction samples with suitable complexity and diversity for target L.MMs. The framework consists of three main components. First, to increase the diversity of generated data, an instruction data rewriting process involving multiple advanced LLM agents is proposed. This process samples different LLM agents for rewriting instructions and responses separately (referred to as agent-pairs). Next, to select high-quality samples, the generated data undergo a dual-model evaluation function with appropriate complexity as the selection metric. Finally, to balance data diversity and quality, the sampling probability of agent-pairs adjusted and evolved based on the composite scores of the selected data, identifying agent-pairs that generate high- hugh-tuulity data.
Extensive experiments are conducted evaluate instraction-folllwing capabilities of LLMs variety of benchmark datasets, including MT-bench (54), Vicuna-bench [54]], and the WizardIM testset 44 Instruction tuning experiments with LLMs such Pythia and LLaMA, demonstrate the effectiveness of the Star-Agents framework. LLMs trained on data generated by Star-Agent outperform those (the same LLMs) trained on the Evol-Instruct dataset [44) or data selected according to the Instruction-Followin Difficulty (IFD) metric (20). Significantly, the optimized datasets have resulted in an average performance improvement of 12%, with some metries such as Fermi demonstrating gains of over 40%.
2 Related Work
Our work is related to both instruction data generation and selection. We briefly review these topics within the constraint of space.
Instruction Data Generation Datasets like Dolly 7 and OpenAssistant [5] are built from human- generated instruction data. The ShareGPT dataset, built from conversations between humans and ChatGPT, has been effectively used to improve the instruction- forllooin performance of fine-tuned models (6) Both Self-Instruct [36] and Alpaca [33 leverage the generation capabílities of GPT-3 to expand seed instructions. The generated instructions undergo filtering to eliminate low-quality instructions while the kept instructions
used to fine-tune the model to enhance the model's 's ability to respond to instructions. Baize 45 proposes a self-dialogue framework, using questions from popular Q&& websites as starting topics, then having L.LMs converse with themselves. CAMEL [66 introduces a role-playing framework where LLMs discuss a given topic when playing a role as either "user" "assistant". UltraChat [8] uses real-world named entities combined with various text-writing tasks to generate diverse and high-quality multi-turn dialogues for LLMs. Lion [13 introduces the concept of adversarial distillation, using the stages to iteratively generate data, refine existing instructions, and produces more complex and diverse instructions expand the capabilities of the student model. Proo--Inttucct 44 uses five manually designed prompts to explicitly guide LLM in rewriting existing simple instructions into more complex ones. The WizardLM model. trained with
ranks highly on MT-Bench 54 highlighting the importance of data quality in training effective L.LMs.
Instruction Data Selection With the aforementioned methods, it is not difficult to use LLMs to generate large instruction tuning datasets at low cost. However, for instruction-tunat tuned language models, data quality is more crucial than quantity. In this aspect, ALPAGASUS 5 evaluates the effectiveness of instruction data by leveraging ChaLGPT. INSTAG 24 automatically generates tags for instruction samples with ChatGPT and keeps diversity by selecting subsets with more tags. Cherry LLM [ pioneers the self-guided approach, using the IFD metric to measure the difficulty for an LLM to learn

---

data, Seed Dala
Instracton: kean dasstiicatio ideatily suttable learning sppbed
Star-Agents
Ageat: Par Besponge The suttalle mathine leaming model for hunyy dascDration tades indlude: LI Logistic regresien: his statistral methot that wes logatic fancium model bisary dependers varibll.....
Instruction Memory Bank
Retrivvl
A Besponse can bundle begy dotanets high dimensionsliy ..............................
leaming
ettectve handing catepecical
Seed Data
Pair Reqgonae: For Foribtoryy dissiiication tults, sivani modiise loaming models suisable, dependlng the size of the dataset, complexity of the probien, and the availability feonures.
Step 1
Generating
Diserso Data
Diverse Data
Update Star-Agents
Step 2 Exahusting Taflardd Dual-mode Stralggs
Star-Agents
Small Medil
Large Model
11M
Step Exalating Star-Agents
Ap Pac Roguss Tix sulible
+0.01%
Agans Deppnne satalle
Seed Data
Agent Pai
Aget-Par3 Reguarss For hineyryy
Instruction M Memory Bank
Dual-model Evaluation
LLM Evaluation
Tailored Data
Figure 1: The diagram of the Star-Agents Framework. Step is designed to gather diverse instructions and responses as shown in Appendix A.3] Step 2 focuses selecting high-quality, tailored data from the data collected in Step 1. Finally, Step 3 aims to enhance the effectiveness and efficiency of the data generation process by evolving the Star-Agents framework.
an instruction sample. This allows to select instruction samples that significantly enhance training efficiency without resorting to external model. DEITA 22 first uses ChatGPT evaluate the complexity and quality of samples, then assesses the diversity of samples based on the distance between model embeddings, thereby guaranteeing complexity, quality, and diversity in the subset. LIFT [46] first guides GPT-4 to generate challenging instructions to expand the data distribution and then dimensionality reduction and row variance analysis to select representative high-quality data, where GPT-4 generates a quality score for each instruction. LESS
4 first stores the gradient features of samples in the dataset, then calculates the similarity between small number of samples from the target task and the training data samples. Based on the calculated similarity scores, it selects the training samples whose gradient features are most similar those of the target task samples as the fine-tuning instances. Data selection not only improves training efficiency but also prevents low-quality or poison data from undermining model performance by filtering them 47
3 Star-Agents
The aim of our research is to construct high-quality dataset T of tailored complexity for the target LLM through the enhancement of an initial seed dataset S (I,, R, M consisting of instruction- response pairs (I. R).
To this end, we introduce the Star-Agents Framework, depicted in Figure which segmented into three steps. The first step leverages spectrum of advanced LLMs, each trained independently. These models are engaged in a dynamic interaction to generate diverse data candidate set (SS)) by sampling agent-pair derived from S; as detailed in Section 3. Following this, apply dual-

---

model evaluation strategy T--) to meticulously extract the most suitable data from D(S,), aiming to substantially elevate the target model's performance. This process elaborated Section 3.2 To enhance the effectiveness and efficiency of the Star-Agents framework in generating tailored data, we
have developed an evolutionary strategy for the Star-Agents, as discussed in Section 3.3 After these
three steps, tailored high-quality dataset 7 is obtained from the seed dataset, which is formulated as:
T = {arg max क (D(S.) | 1, dED(S.)
(1)
3.1 Generating Diverse Data
To improve the instruction-tuned model, is crucial to assemble a high-quality and diverse instruction dataset [22]. Traditional methods often single LLM, such as ChatGPT, for data enrichment. In contrast, our approach employs multiple LLMs to avoid monotonous data distribution. This multifaceted strategy also addresses the limitations and risks of quality degradation domain- specific tasks associated with using single model. To counter these challenges, we propose to use an Agent-Pair Pair strategy.
Agent-Pair. Utilizing a spectrum of LLMs, each trained with discrepant setting, facilitates the generation of varied responses given instructions. This diversity is crucial for synthesizing dataset characterized by high richness [24].
The Star-Agents framework strategically pairs different LLMs to rewrite the instructions in the seed dataset and generate new responses to increase the diversity. With agent-pair (A/, A new instruction data can be generated as follows:
f (4, R2) = (A)())),
where A' and A represent the agents that rewrites the instruction and response to the instruction, respectively.
Given the high cost of deploying all agent-pairs, feasible solution balance cost and agent diversity is to sample subset of agent-pairs from the Star-Agents for data generation. Equation formulates this process, where D is collected dataset generated by performing / over all sampled pairs (A,, A of instruction agents A and response agent A with sampling probabilities PςRi
D (S;) = (fir (Sa).... ... Sas (S:) (Jm, kim)) ~ Pÿk: m = 1,2, M).
(3)
M is number of agent-pairs sampled for single seed sample. The sampling probability Pyk is initialized as a uniform distribution and will be updated using the method described in Subsection (3.3 during data generation. Meanwhile, Instruction Memory Bank that stores high-quality instructions will be updated. To ensure the lower bound of data quality, each iteration will consistently call a fixed of agent-pairs, referred base agent-pairs.
3.2 Evaluating Tailored Data via a Dual-model Strategy
Identifying and selecting tailored data from a diverse dataset is crucial for enhancing model per- formance, especially since the presence of low-quality data can impede model functionality. is acknowledged that data samples that are lengthy, complex, and challenging significantly benefit the instruction tuning process 22
Nevertheless, too complex instruction data may be not necessarily benefit model performance. We have observed that for models with 14M and 70M parameters as illustrated in Figure 2 the Evol- Instruct dataset, though more challenging than the Alpaca dataset, results in diminished model performance. This suggests that intricate examples may surpass the capabilities of small models and be harmful for model performance, despite the advantages of using complex data for large models.

---

IFD
2.2
Large model
Performence
2
1.8
1.6
Datasets
Small model
1.4
Alpaca
Evol-Instruct
50
100
150
Model Size (M)
Complexity
Figure Performance comparison of varied- scale models on the Alpaca and Evol-Instruct datasets. The tasks from the Evol-Instruc dataset are more complex than those from Alpaca.
Figure 3: Illustration of dual-model evaluation. Data with a significant gap between the IFD scores of the small and large models will be pri- oritised.
larger model to evaluate the difficulty of data instances together with the evaluation from a smaller model (target LLM), hence termed dual-model evaluation. Inspired by Cherry LLM (00), we employ the IFD metric to measure the degree of difficulty data sample presents to the target model, which is calcuated
Dual-model Evaluation. To address the issue mentioned above, we propose
exp (-
[R, EWER, log P(w) T.)
IF, R&)
(4)
exp
[.. EwER, log P(w)
We assume that for the same sample, stronger model yields smaller IFD score. When the IFD scores of the two models are close to each other, it indicates that the sample is either too simple or too complex, which is not contributive effective learning. However, when their IFD scores differ significantly, it indicates that the data is sufficiently complex for the smaller model but still within the capability range of the stronger model. This is tailored complexity for facilitating learning. The above data assessment method is illustrated at Figure Bl and formulated as
IFD (I,, Ra) IID (Ii, Ra) (IFD (I; R.) IF Diarge [i, R,)) Isism
dual
max
Noising data can be endowed with high score since the dual-model metric considers only the relative complexity with the neglect of generation quality. To address this issue, we utilize LLM referee for data sample scoring. This involves comparing each data sample in the same batch of diverse data samples generated by selected agent-pairs against base data sample generated by base agent-pairs. There are three potential outcomes: the base data sample is better, the diverse data sample is better, tic, shown in Appendix A.I These outcomes quantitatively assigned quality scores, thereby avoiding collecting noising instruction samples:
0, the base data sample is better,
1, if the generated data sample is better,
(6)
0.5,
if tie.
Finally, the evaluation scores from both the LLM and the dual-model evaluation are combined to compute a final composite score:
π llm
Tdual-
This score determines the overall quality and suitability of data for enhancing the model's capabilities. The highest scoring data sample is then selected into dataset T and Instruction Memory Bank as detailed Section 3.3 ensuring that the chosen dataset maximizes potential improvements model performance.
5

---

3.3 Evolving Star Agents
As mentioned in Section 3.1 we use the joint probability of instruction agents and response agents to regulate the invocation of each agent-pair. Considering the abilities and specialities of each LLM vary, however, sampling each agent-pair with the same probability is not optimal. We hence use the score from Section 3.22oon dynamically evolve the sampling probability. Additionally, since the generation performance of agent-pairs is task-dependent, we also propose an Instruction Memory Bank select the most suitable agent-pair for particular tasks.
has introduced the score n, which effectively esti- mates the quality of generated samples. During each iteration, if the generated samples are of high quality, we will increase the sampling probability the selected agent-pain which is updated as follows:
Agent-Pair Sampling Evolution. Section 3.2
Pyk: = Pyk + B B--TTII Rd),
Pik
(8)
Pys -
Psk
The updated sampling probability for the agent-pair of the j-th instruction agent and k-th response agent that successfully process the i-th data sample will be used in the next iteration, where / denotes the evolution rate. This formula adjusts the sampling probabilities based on the effectiiveness demonstrated by agent-pairs in generating relevant data. Iterative updates ensure that as the synthesis process advances, the probability of selecting more effective agent-pairs increases, while less effective pairs are gradually phased out.
Instruction Memory Bank Evolution. We establish an Instruction Memory Bank storing high- quality instructions aiming to accelerate sampling and relate the evolution with task data. When processing data sample (I, R:), we perform a query in the Instruction Memory Bank for I, retrieving the top closest matches according to embedding similarity. The associated agent-pairs, identified as highly proficient for tasks similar to I, are then sampled. We sample / agent-pairs from this pool using normalized probabilities to generate diverse data. Moreover, foster the creation of a diverse dataset, additional M - agent-pairs are sampled from the remaining pool using their respective probabilities to assist data synthesis. As result, M new samples are generated and then feed for data assessment. Subsequently, the Instruction Memory Bank will continuously evolve by incorporating tailored high-quality data, which get high socres as introduced in Section 3.2
4 Experiments
We conducted extensive experiments evaluate the proposed Star-Agents framework. wide range of LLMs, benchmark datasets were used in our experiments to guarantee the robustness of our evaluation.
4.1 Setups
Datasets. In alignment with the WizardLM [44], we adopted the Supervised Fine-Tuning (SFT) dataset, designated as the Enol-Instruct dataset, which consists of 70,000 instruction-response pairs. The instructions in this dataset were refined using "In- -n-Dehth Evolving" and "In- -reeBrradth Evolving" methods, which were tailored enhance the base instructions by adding intricate details or expanding the overall scope, respectively. To guarantee the fidelity of the data, ChatGPT was also integrated as generator into the refinement process. The quality of the instruction data from the Evol-Instruct hence, our research continues to leverage these refined instructions. Employing the Star Star-Agents framework, our study invokes multiple LLMS to generate diverse and high-quality responses for these instructions. For further enriching our comparative analysis, we employed the Alpaca dataset [32), comprising 52,000 instruction-follon following samples. This dataset, developed under the self-instruct paradigm, utilizes the ChatGPT instead of text-davinci-003 for a fair comparison
dataset has been validated as superior [44 25
/httss//chatppt.com/

---

Table 1: Typical LLMs utilized in Star-Agents.
Model Famliy
Model Size
Data Size
Method
Source
Phi 11
2.7B
1.4T
Pretrain
Microsoft
ChatGLM [51
6B
IT+
SFT & RLHF
Zhipu AI
Gemma 6
7B
6T
SFT & RLHF
Google
Mistral 22
7B
SFT
Mistral
Qwen
14B
SFT & RLHF
Alibaba
ChatGPT
SFT & RLHF
OpenAl
Models. In response to the growing need for cost-effective inference of LLMs at the edge, our study explores the capabilities of target models scaled at IB and 7B parameters. The 1B models, specifically the Pythia-IB [2], were trained on roughly 300 billion tokens derived from the Pile dataset. The 7B models, represented by the Llama-2-7B 35 were trained on an extensive corpus of 2 trillion tokens.
During our experiments, we integrated generator a diverse array of LLMs, as detailed in Table I Our hypothesis posits that models from different development teams possess unique capabilities, yielding rich responses to identical prompts due to the diversity their training data and strategies. For instance, the Phi2 [1] employed 1.4T tokens of meticulously curated texthook-like data without undergoing Reinforcement Learning with Human Feedback (RLHF) while the Gemma [3 was trained on 6T tokens primarily sourced from English web documents, mathematical content, and code, with subsequent fine-tuning through SFT and RLHE To ensure the diversity and quality of generated data, we assembled LLMs trained by different teams, widely regarded for their exceptional performance. pursuit of fostering the generation of data across varying levels of difficulty, the utilized LLMs range from 2.7B 14B parameters, including even larger models via API access. For a fair comparison with the Evol- Instruntt dataset, the most capable model employed was the ChatGPT, which was also used for generating responses within the Instruttt dataset. Notably, the ChatGPT was also served as evaluator to compute the comparison score Tilm-
Benchmarks. To rigorously evaluate the instruction-followee capabilities of Al models, we uti- lized three widely used benchmarks: MT-bench, Vicuna-bench, and the WizardLM testset. Specifi- cally, MT-bench and Vicuna- bench are designed to test the models' competencies in various complex cognitive tasks, including mathematics, reasoning, complex format handling, and writing through both multi-turn and single-turn dialogues. The WizardLM testset, conversely, extends the evaluation to encompass diverse fields such as technology, biology, and law. also features varied difficulty nuanced comparison of models' performance disparities. Following estab-
levels to facilitate lished protocols, we employed the Fast-Chat (54] to assess model performances, with GPT-4 acting the judge model.
Baselines. For bascline comparisons, we employed the Pythia- IB and Llama-2- 2-7B, both trained using the Evol-Instruct datasets. The Alpaca datasets were also referenced for comparative analysis, alongside IFD 20 and Random select as an additional comparsion for data selection methods.
Implementation Details. We fine-tuned our models (Pythia-1B 1B and Llama-2-7B) over three epochs using the Adam optimizer, with an initial learning rate of 2 x 10-5, maximum token count 2048, and batch size of 64. For the Star-Agents 10 agent-pant were employed.
4.2 Main Results
GPT-4 Automatic Evaluation Based on the findings summarized in Table 2 comprehensive training sessions were conducted for the Pythia- 1B and Llama-2-7B models utilizing three distinct datasets: Alpaca, Evol-Instruct, and the optimally refined Star Instruct datasets. The latter was developed through the application of Star-Agents, which are derivatives of the Evol-Instruct datasets. Through comparative analyses with other contemporary state-of oate-of-the-art models, we observe that the SFT-aligned models employing the Star Insiruct datasets consistently outperform nearly all aligned counterparts, across all evaluated model families.

---

Table 2: Results of different models Vicuna-bench, WizardLM testset and MT-Benchh
Model
Vicuna-Bench
WizardLM testset
MTT-eenhh
Average
1B Models
Pythia- 2
1.68
1.34
1.17
1.40
OPT-1.3B 52
2.49
1.64
1.12
1.75
42
2.73
1.86
1.39
2,06
Pythia- BB-alaca
4.14
2.97
2.20
3.10
Pythia-IB-evo instruct
5.07
3.55
2.56
3.73
Pythia-IB-IFD 200
4.60
3.21
1.98
3,26
Wytha--IB-Randee
5.13
3.39
2.35
3.62
Pythia- 1B-aarrrrttrr instruct
5.93
3.90
2.69
4.17
7B Models
Llama-2-7B 35
3.95
zephyr-beta-sft [22
5.32
mpt-7B-chat [22
5.45
KGen-7B-8k-Ima 26
5.55
3Recycled-Wi-77B
17
5.56
-lamaa--7B-alppca
6.33
5.08
3.63
7.27
6.57
5.21
6.35
Baamma---7B-str__nnstteue
8.24
6.87
5.74
6.95
generic
writing
writing
knowledge
humanities
roleplay
math
roleplay
stem
reasoning
0
2
10
0
2
8 10
coding
common-sense
extraction
math
counterfactual
fermi
coding
model
-lama-2-7B-star
instssutt_instructt instruct
model
1lama-2-78-stee instruct
Jlama-2-78-ev0- Instruct
Llama-2- 2-7---voll instruct
(a) Vicuna-bench
(b) MT-bench
Figure 4: Radar plot of detailed scores for Llama-2-7B-ste instreut against the major baseline on different subtasks of (a) Vicuna-Bess Bench and (b) MT-Bench.
Notably, at the 1B scale, models trained with the Star Instruct dataset demonstrate significant superiority, surpassing baselines across diverse evaluation datasets. Remarkably, in comparison to models trained with the Evol-Instruct dataset, those utilizing Star Instruci achieve average absolute improvement of approximately 0.45, which is corresponding to a performance enhancement of about 12%. Additionally, when compared models trained with the Alpaca dataset, our framework achieves an absolute improvement of point, thereby affirming that the Star Instruct dataset is particularly well- suited for the Pythia-IB model, significantly boosting its operational efficacy. Additionally, within the 7b model category, the Llama-2-785 2-tart instruct outperforms the sRecycled Wiz-7B-vv 7B-22 [17), which is trained the Alama-2-7B-stee instruct's s performance enhancements across nine metrics, with notable substantial improvements math, coding fermi problem-solvine where improvements surge up 40%. similar phenomenon can be observed in Figure 4b) Additionally, comparative examples of single-turn and multi-turn dialogues are provided Appendix A.2 and the performance on the Open L.LM Leaderboards of LLMs can be found in Appendix A.4
dataset enhanced by Selective Reflection- -unning.nTTTinn..... Figure EigureAajilla illlstrrtss the

---

Table Impact of different components.
Table 4: Imapet of the selection method.
Components
Average Score
Model
Average Score
Diversity
Data selection
Evolutiuon
Pythia- IB-evol instruct
3.73
4.17
Pythia-IBB IB-IFD (00
3.26
X
3.97
Pythia- IB-Random
3.62
X
3.62
X
X 3.73
Pythia- 1ythia-IB-star IB-star instruct
4.17
0.4
Agent-Pairs Pairs
Sampling Probability
Mistral-ChatGPT
ChatSPT-0613
PhiZ-ChatGPT
0.2
20k
30k
40k
50k
60k
Iteration
Figure 5: Evolution of the typical Agent- Pairs.
4.3
Ablation Study
conducted ablation experiments on the three principal components within the Star-Agents framework. Results indicate that models using solely diversified datasets with random sampling yield bit lower performance than the baseline. This occurs because the baseline employs data generated by ChatGPT, which is of high quality. In contrast, the diversified datasets draw from variety of sources, making challenging to ensure uniformly high quality. Thus, random sampling may introduce low-quality data, leading to diminished model performance. inclusion of a data selection module subsequently leads to a recovery in model performance, suggesting that this module effectively selects high-quality data suitable for the model. Integration of the evolution strategy also provides a significant improvement, demonstrating that the evolution module can effectively select the most appropriate data generation eggent-pair" from complex array of candidate agent-paing
Main Components. As illustrated in Table
Selection Method. As demonstrated in Table 4 evaluated range of conventional selection
methods, including both random selection and strategies informed by the IFD [20]. Our dual-model selection strategy significantly outperforms these approaches. Compared to random selection, method achieves a significant improvement, registering an improvement exceeding 0.5 points on average across variety of test sets. When compared with the IFD approach, our enhancement approaches a 0.9 point. These findings robustly validate the effectiveness of our dual-model selection strategy, illustrating its superior performance in refining model selection precision using diverse evaluation metrics.
Evolution. As depicted in Figure 5 we analyzed the sampling probability curves of typical agent- pairs throughout an iterative evolutionary process. Initially, each agent-pair began with a sampling probability of approximately 10%. Due its robust performance, the Mistral-ChatGPT receives consistent rewards, which leads to gradual increase in its sampling probability. By the completion of about 70,000 iterations, this probability has escalated 30%. In stark contrast, the Mi2TCChtGGPT undergoes steady decline over the same period, with its sampling probability ultimately plummeting to near zero as it is progressively phased out. Concurrently, the ChaGGLM3-ChatGPT exhibits relatively stable trajectory, albeit with a slight downward trend. Evolutionary trajectories present significant discrepancy indicating different generation suitability different generators on different tasks, where all the differences are captured by evolution mechanism.

---

5 Conclusion
In this paper, we have presented the Star-Agents framework, an automated system for optimizing data LLMs. to be optimally challenging for LLMS. This framework has been applied the open-source SFT datasets, and we conduct training sessions on a variety of model families, adjusting the data to enhance its efficacy. Our empirical investigations include a series of instruction tuning experiments that utilize both multiple baselines and specially optimized datasets on well-known models such as Pythia and L.aMA. Extensive experiments confirm the substantial impact of method: the optimized tailored datasets result in an average performance enhancement of approximately 12%. with certain metrics, especially those involved in Fermi problem tasks exhibiting increases exceeding 40%, as substantiated by results on benchmarks such as MT-bench, Vicuna bench, and the WizardLM testset. These findings underscore the premise that strategically diverse and tailored data can profoundly improve model alignment and performance. In conclusion, our research details highly effective automated framework that significantly augments dataset functionality, thus fostering more efficient model alignment.
Limitations. Our approach achieves remarkable performance improvements on single-turn instruc- tion datasets. However, it has not yet been evaluated on multi-turn conversations. We hence leave the evaluation multi-turn instruction datasets and validation on datasets with domain-specific instructions to our future work.
Acknowledgements
The present research was partially supported by the National Key Research and Development Program of China (Grant No. 2023YFE01 16400). We would like to thank the anonymous reviewers for their insightful comments.
References
[1] Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint 1.XX::2339.166099) 2023.
[2] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff. et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2433 PMLR, 2023.
[3] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. survey on evaluation of large language models, 2023.
[4] Justin Chih-Ye Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin and Mohit Bansal. Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. arXiv preprint arXiv:2402.0162 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratma, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou. Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data, arXiv preprint arXiv:2307.08700 2023.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open open-source chatbot impressing gpt-4 with 90% chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[7] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tune llm. Company Blog of Databricks, 2023.
[8] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-qualis instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
10

---

[9] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 320-335. Dublin, Ireland, May 2022. Association for Computational Linguistics.
[10] Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong. Deyi Xiong, et al. Evaluating large language models: A comprehensive survey. arXiv prepriat 11Xiv:2300.19736 2023.
[11] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog. 2023.
[12] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplor, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
[13] Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of large language model. arXiv preprint M3Xi::2305,12870 2023.
closed source
[14] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 1896-1907, Online, November 2020.
Association for Computational Linguistics.
[15] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanicy, Richárd Nagyfi, et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
[16] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. 2023.
[17] Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for Ilm instruction-tuning. arXiv preprint Xiv: 101440001111000 10, 2024.
[18] Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, and Tianyi Zhou. Reflection- tuning: Data recycling improves Ilm instruction-tuning. ArXiv, ahs/2310. 11716, 2023.
[19] Ming Li, Yong Zhang, Shwai Hc, Zhitao Li, Hongyu Zhao, Jianzong Wang. Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to- strong data filtering for fast instruction cuaructtoontuning.... 2024.
[20] Ming Li, Yong Zhang, 7hitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.122 12032, 2023.
[21] Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinmeng Ji, Tingting Cui, Tao Liu, et al. Openeval: Benchmarking chinese llms across capability, alignment and safety. arXiv preprint 223.:24433.121166 2024.
[22] Wei Liu, Weihao Zeng, Keqing Hc, Yong Jiang, and Junxian He, What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312 1566655... 2023.
[23] S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph. Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuming. ArXiv,
2023.
[24] Keming La, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. instag: Instruction tagging for analyzing supervised fine-tuning large language models. In The Twelfth International Conference on Learning Representations, 2023.
[25] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saced Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307 2if:22307.06435, 2023.
[26] Erik Nijkamp. Tian Xie, Hiroaki Hayashi, Bo Pang. Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, et al. 7b technical report. arXiv preprint (3Xi:22309004500) 2023.
11

---

[27] Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo. Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey, arXiv preprint arXiv:2309,15025 2023.
[28] Tianhao Shen, Sun Li, Quan Tu, and Deyi Xiong, Roleeval: A bilingual role evaluation benchmark for large language models. arXiv preprint arXiv:2312.16132, 2023.
[29] Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the exploitability of instruction tuning. 2023.
[30] Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, Menglong Cui, Jiangeun Dui, Yikun Lei, Lei Yang. Ling Shi, Juesi Xiao, et Fuxitranyu: multilingual large language model trained with balanced data. arXiv preprint arXiv:2408.06273 2024.
[31] Yehui Tang, Fangcheng Liu, Yunsheng Ni, Yuchuan Tian, Zheyuan Bai, Yi-Qi Hu, Sichao Liu, Shangling Jui, Kai Han, and Yunhe Wang. Rethinking optimization and architecture for tiny language models. arXiv preprint 2402 249.09910 2024.
[32] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Stanford alpaca: An inslooiiion-ollloiing llaaa model. https://githubb com com/tatsu-2a saabsttnfors ord__ alpaca 2023.
[33] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An fnstriction-folloiing Ilama model, 2023.
[34] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829, 2024.
[35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXivv2307.0922 09288, 2023.
[36] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint ar arXiv:2212. 10560, 2022.
[37] Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self- generated instructions. In Proceedings of the 61st Annual Mceting of Association for Computational Linguistics (Volume I: Long Papers), pages 13484-13508, Toronto, Canada, July 2023, Association for Computational Linguistics..
[38]
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi. Yeganeh Kordi. Amirreza Mirzaei. Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. Super- NaturalInstructions:: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5105 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[39] Yunhe Wang. Hanting Chen, Yehui Tang, Tianyu Guo, Kai Han, Ying Nie, Xutao Wang, Hailin Hu, Zheyuan Bai, Yun Wang, al. Pangu-pi: Enhancing language model architectures via nonlinearity compensation. arXiv preprint arXiv:2312.17276 2023.
[40] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[41] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. 2021.
[42] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.0669) 2023.
[43] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint 04333,, 2024.
12

---

[44] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 122444, 2023.
[45] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An spen-s chat model with parameter-efficiet tuning on self-chat data, arXiv preprint arXiv:230..0119, 2023.
[46] Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundaresan. Rethinking the instruction quality: Lift what you need, 2023.
[47] Jun Yan, Vikas Yadav, Shiyang Li. Lichang Chen, Zheng Tang, Hai Wang. Vijay Srinivasan, Xiang Ren, and Hongxia Backdooring instruction-tun large language models with virtual prompt injection. In NeurlPS 2023 Workshop on Backdoors in Deep Learning- The Good, the Bad, and the Ugly, 2023.
[48] Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang. Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Virtual prompt injection for instruction-tunce sutiiido--teddddd large language models, 2023.
[49] Jingfeng Yang. Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: A survey on chatgpt and beyond, 2023.
[50] Qinyuan Ye, Bill Yuchen Lin, Xiang Ren. CrossFit: few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7165 7163-7189, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[51] Aohan Zeng, Xiao Liu, Zhengxiao Du. Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
[52] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205 2000002200.00068. 2022.
[53] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjic Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nic, and Ji-Rong Wen. A survey of large language models, 2023.
[54] Lianmin Zheng, Wn Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Fric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chathot arena. Advances in Neural Information Processing Systems, 36, 2024.
13

---

A Appendix
A.1 Prompt Examples
Following the Fast-Chat [54], the prompts used in the data selection process are as listed in Table 5
Table 5: Prompts of data selection for LLMs.
System Prompt: Please act as an impartial judge and evaluate the quality of the responses provided by three AI assistants the user question displayed below. You should choose the assistant that follows the user's s instructions and answers the user's 's question best. Your evaluation should consider factors such the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant the bset, "[[B]]" if assistant the bset, and "[[C]]" for tie.
Prompt Template:
[User Question]
<question>
[The Start of Assistant A'S Answer]
<answer_ai a>
[The End of Assistant A's Answer]
(The Start of Assistant B's Answer]
<answer_h b>
[The End of Assistant B's Answer]
[Final Verdict]:
A.2 Case Study
Examples of the single-turn dialogue and multi-turn dialogue are presented in Table 6 and Table
Table 6: Single-turn dialogue example.
Instruction: tech startup invests S8000 software development in the first year, and then invests half of that amount in software development in the second year. What's the total amount the startup invested in software development over the two years?
Llama-2-7B-eval instruct: The total amount the startup invested in software development over the two years is $12,000.
Llama-2-7B-stan instruct: To calculate the total amount the startup invested in software de- velopment over the two years, we need to find the total investment in software development for each year and then add them together. In the first year, the startup invested $8000 software development. In the second year, the startup invested half of that amount, which is $4000. So, the total investment in software develop- ment for the two years is: $8000 (first year) + $4000 (second year) S12000 Therefore, the total amount the startup invested in software development over the two years is $12000.
14

---

Table 7: Multi-turn dialogue examples.
Instruction: Act as a math teacher. I will provide some mathematical equations concepts, and it will be your job to explain them in sasy-to-understue terms. This could include providing step-by-step instructions for solving problem, demonstrating various techniques with examples in everyday life suggesting online resources for further study, My first request is "I need help understanding how probability works."
-amma-2-B--evol iistructt: iinstruuttt Sure, I'd be happy to help! Probability is the likelihood of something happening, and it's fundamental concept in math and science. To explain it simply, probability is the chance of something happening out of all possible outcomes. For example, if you flip coin, the proha- bility of getting heads is out of 2. or 50%. Proba- bility can be used to make predictions and calculate risks, and it's important in fields like finance, statis- tics, and data analysis.
Llama-2-7B-stte iisttstttt Sure, I'd be happy to help explain probability. Probability is a measure of how likely an event is to occur. It's often expressed as number between 0 and 1, where 0 represents a very unlikely event and represents very likely event. To calculate probability, you need to know the total number of possible outcomes and the number of favorable outcomes for the event you'l 're interested in. For example, if you're re flipping fair coin, there are two possible outcomes (heads or tails), and the probability of getting heads 1/2 or 0.5, because there's 's only one way to get heads out of two possible outcomes. In everyday life, probability can be used calculate the likelihood of certain events, such as the chance of rain on a particular day or the proba- bility of getting a certain grade on a test. It's also used in decision-making situations, like calculating the risk of particular investment or the likelihood of success in a business venture. There are many resources available online to help you learn more about probability, such as video tutorials, practice problems, and interactive simulations. Some popular websites include Khan Academy, Coursera, and edX. which offer free courses and resources on probability and other mathematical concepts.
instruct:
To calculate probability, you can use different formu- las depending on the type of problem you're re solving. One common method is to use formula called the probability mass function, which tells you the likeli- hood of getting a certain outcome. Another method is to use the probability density function, which tells you the likelihood of getting certain range of out- comes.
There are many resources available online to help you learn more about probability, including video tutorials, interactive simulations, and practice prob- lems. Some popular websites for learning probability include Khan Academy, Mathway, and Coursera. I hope this helps!
Instruction: What are the differences between Riemannian geometry and euclidean geometry?
geometry and Euclidean geometry are two different types of geometries. In Euclidean geometry, the distance be- tween two points is always straight line, and the angles of a triangle always add up to 180 degrees. Riemannian geometry, the distance between two points is not always a straight line, and the angles of a triangle can add up to more than 180 degrees.
-lama-2-7B-eve - mnstruct:Riemannin
Great question! Ric- mannian geometry and Euclidean geometry are two different types of geometry that are used to describe and analyze the properties of spaces. The main differ- ences between the are as follows: Metric: In Euclidean geometry, the distance between two points is defined by the Pythagorean theorem, which states that the sum of the squares of the two sides of a right triangle is equal to the square of the length of the hypotenuse. In Riemannian geometry, the distance between two points is defined by a metric, which is a function that describes how much the shortest path between two points is curved. 2. Curvature: Euclidean geometry dcals with flat spaces where the curvature of the Earth's surface is negligible. In contrast, Riemannian geometry deals with curved spaces, such as the surface of the Earth, where the curvature plays significant role in determining the properties of the space. 3. Geodesics: In Euclidean geometry, the shortest path between two points is a straight line. In Riemannian geometry, the shortest path between two points is called geodesic, which be curved and depends on the metric. 4. Tensor Fields: Riemanniam geometry involves the use of tensor fields, which are mathematical objects that describe the geometry of space. These fields are used to calculate the curvature of the space and the behavior of objects moving through it. Overall, Rie- mannian geometry and Euclidean geometry are two different approaches to understanding the propertics of spaces, with Riemannian geometry being more applicable to curved spaces and Euclidean geometry being more applicable to flat spaces.
Llama-2-7B-star, iinttrruttttttt:::t
Another key difference between the two geometries the way they handle curved surfaces. In Fuclidean geometry, a flat surface is rectangle with parallel sides, while in Riemannian geometry, a surface can be curved in any way. This means that Riemannian geometry is better suited for describing the geometry of the universe, as it can handle the curvature of space-time.
Overall, Riemannian geometry is a more complex and advanced form of geometry, and it has many practical applications in fields such physics and engineering.
15

---

A.3 Data Examples
Data examples are shown in Table g
Table 8: Diverse Data Examples.
Instruction: Given some binary classification data, idcutity a suitable machine learning model that can be applisd to this task.
The suitable machine leamingg models for binary classification tasks include: 1) Logistic regression: is a statistical method that uses logistic function to model a bisary dependent variable. a relatively simple and interpeetable model that works well with linearly separable datasets.
Evol Instruct:
2) Support Voctor Machines (SVMs) It is popular classifier that is used for binary classification tasks, SVMs can handic high- dimensional data and work well with datasets that not linearly sepazable.
3) Random Forests: is an ensemble learning method that works by constructing multitude of decision trees at training time and outputs the class
the mode
classes output by the individual trees.
4) Neural networks: It is type of machine leamnin model that is widely used for binary classification tasks. Neural networks can handle non non-Linear relationships between variables and are capable of learning complex patterns in the data.
The choice of the model depends on the particular characteristics the data and the spocific requirements project."
Agent-Pair Pair There several machine learning models that can be applied binary classification tacks, depending on the nature of the data and the problem at hand. Here are few popular ones:
I. Logistic Regression: This is a simple and effective model that can be used for binary classification problems. works by estimating the probubility of the positive class and using
make predictions.
2. Decision Trees: Decision Trees are popular machine learning model that can handle both categorical and numerical features. They work by recursively spliting the data into subscts based the values of the input features.
3. Random Forest. Random Forest is an casemble learning methed that combines multiple docision trees to improve the accuracy of the model. It works by creating set of decision trees then averaging their predictions.
4. Support Vector Machines (SVM): SVM is powerful model that can handle high-dinenssooal data. It works by finding the hyperplane that maximally separates the classes the feature space.
5. Neural Networks: Neural Networks
set of algorithms that can learn and classify from data. They handle complex and nonlineur relationships between the input features and the output lahels.
The choice of the model depends various factors such as the size and complexity of the datasct, the type features, the computing resources, and the desired level of accuracy.
Agent-Pail Pair B: For binary classification tasks, several machine learning models are suitable, depending the size of the datasct, complex- ity of the problem, and the availahility of features. Here are some popular models that can be applied: Logistic Regression: simple and interpretable model that works well for small to medium-sized datasets. It's linear and can handle both categorical and continuous features.
Decision Trees and Random Forest: These are easy to interpret and can handle both numerical and calegorical data. Random Forest an ensemble method that combines multiple decision trees, reducing overfitting
Support Vector Machines (SVM): Effective for dimeeninnnii data and can handle non-linear relationships through kemel functions. SVMs are robust to outlicrs and can be used for both lincar and non-lincar classification.
4. Naive Bayes: A prohahilistic model that assumes features are independent. It's fast, easy to implement, and works well for text classifi- cation when features not highly correlated.
5. K-Nearest Neighbors (KNN): A simple model that classities based on the majority class of its nearest neighbors. It's non-parametric
and can he useful for small datasets.
6. Neural Networks (e.g. Feedforward, Convolational, or Recurrent Neural Networks): Deep learning models that can capture complex
patterss in large datasets. They are powerful roquire more data and computational resourees.
7. Gradicat Boosting Machines (GBM) XGBoost: Ensemblc methods that lteratively build weak learners (e.g.. decision trees) create
a strong classifier. They are often used for structured data and can handle high dimensionality.
AdaBoost: Another boesting algorithm that combines weak leamerr create strong classifier. It's particularty useful when dealing with imbolanced datasets.
Neural Network Ensembles (e.g.. stacking, blending): Combining predictions from multiple models can improve performance and robustness.
To choose the best model, it's essential to perform exploratory data analysis, split the data into training and testing sets, and ovaluate the models using uppropriate metrics (e.g. accuracy. precision, recall, Fl Fl-score AUC-ROC for imbalanced datasets). Cross-validation can help in selecting the optimal model and hyperparameters
Agent-Pait Pair C: A suitable machine learning model for this task would be Random Forest Classifier, This model is effective in handling both entegorical and numerical data, and can handie large datasets with high dimeasionalitt..
Agent-Pair D: One suitable machine leaming madel for binary classification tasks the Logistic Regression model. Logistic Regression
commonly used for binary classification it provides the probability an instance belonging particular class. It simple and efficient algorithm that works well with linearly separable data.
Other pupular machine learning models for binary classification tasks include Suppori Vector Muchines (SVM), Decision Trees, Random Forest, Gradient Borsting Machines (GBM), and Neural Networks. The choice of the model depends on the complexity of the data, size the dataset, inteprecabillity roquirements, and computational resources available.
16

---

A.4 Performance on Open LLM Leaderboards
The performance on Open LLM Leaderboards is shown in Table 9
Table Performance on Open LLM Leaderboards.
Model
ARC
HellaSwag
MMLU
TruthfulQA
Average
WizardIm
51.60
77.74
42.74
45.75
54.18
Llama-2-7B-evol insstruttt
51.88
76.70
45.76
46.10
55.11
1aima-2-7--ttarinsttti
54.44
77.64
46.94
46.13
56.29
A.5
Computational Cost.
The computational overhead of our proposed method primarily depends on the inference computa- tional load of the various LLMs used:
Qwen-14th During inference with sequence length of 256 tokens, the computational load is approximately 4 x 10 Multiply-Add cumulations (MACs).
Phi-2-2.7B: For the same sequence length, the inference computational load is around 7 x 10 MACs.
ChatGPT: Given that ChatGPT is proprietary model, don't have details on its computa- tional requirements.
Nonetheless, for estimation purpose, we can approximate the overall computational cost. Assuming an iterative process involving multiple LLMs (e.g., 10 LLMs) and large dataset (e.g.. 70,000 samples), the total computation without using our framework can be roughly estimated as:
4 10 FLOPs (Qwen-14B) 10 LLMs 70,000 samples 2.8 10 MACs
While, when the Agent-Pairs Sampling and Instruction Memory Bank are employed, 5 of 10 LLMs used to generate data therefore, total computation be significantly reduced and roughly estimated as:
10 FLOPs (Qwen- 14B) LLMs 70,000 samples 1.4 10 MACs
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction reflect the paper's contributions and scope accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Section Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: YYs]
Justification: All the information needed to reproduce the main experimental results are provided in the Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of LLM), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification:
We will release codes soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ic for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
httppp/liiiigiiieeeeeeeedeaaaaaa CoodeSubeeiieeeeeeeeee ssi asiinnolle onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details are provided in the Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The error bar is not provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details provided in the Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https::/ //tprrip///erupss.... http://......bbi..ttttcssooo....
Answer: [Yes]
Justification: is conformed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is risk in the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Included all citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppawpptheeee whrodee.wtecee..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing research with human subjects.
Guidelines: The paper does not involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "uzPiQnMQAB", "reviewer": "Reviewer_piXU", "summary": "The paper presents a framework for enhancing the quality of instruction datasets used for tuning large language models (LLMs). The proposed framework, Star-Agents, leverages multi-agent collaboration to generate, evaluate, and refine instruction data automatically. The approach comprises three main components:\n\n1. **Diverse Data Generation**: The framework employs multiple LLM agents to generate varied instruction data. Each agent pair, consisting of distinct instruction and response agents, creates diverse data samples to enrich the dataset.\n2. **Dual-model Evaluation**: This component introduces a two-tiered evaluation mechanism using both small and large models. The evaluation metric considers the difficulty and quality of the generated data, ensuring it is challenging yet manageable for the target model.\n3. **Dynamic Refinement**: The framework dynamically adjusts the sampling probabilities of agent pairs based on their performance, promoting the selection of the most effective agents for data generation.\n\nEmpirical studies demonstrate the optimized datasets generated by Star-Agents led to performance improvements, with an average increase of 12% and up to 40% in specific benchmarks such as MT-bench, Vicuna-bench, and the WizardLM testset.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------\nThank you for your reply and I have updated my score accordingly.", "questions": "What will be the results if base models are larger, e.g. llama 13b or llama 70b?", "limitations": "See more details in Weaknesses section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality**\nThe paper introduces a framework, Star-Agents, for automatic data optimization in instruction tuning for large language models (LLMs). This originality arises from:\n\n- **Multi-Agent Collaboration**: Utilizing multiple LLM agents to generate diverse and high-quality instruction data. This approach addresses the common limitations of single-model data generation methods, ensuring a richer and more varied dataset.\n- **Dual-Model Evaluation**: Implementing a dual-model evaluation strategy that assesses both the difficulty and quality of the generated data. This innovative metric ensures the data is challenging yet manageable, enhancing the instruction tuning process.\n- **Dynamic Refinement**: The dynamic adjustment of sampling probabilities for agent pairs based on performance is a creative mechanism that optimizes the data generation process over time.\n\n**Quality**\nThe paper is supported by empirical studies. The framework is tested using instruction-tuning experiments with two models, including Pythia and LLaMA. The results consistently show performance improvements, validating the quality of the proposed method.\n\n**Clarity**\nThe paper is clearly written and well-structured, facilitating easy understanding of the proposed framework.", "weaknesses": "**Limited Dataset Evaluation**\nThe paper evaluates the Star-Agents framework on a limited set of datasets, which may not fully capture the framework's robustness and generalizability. Specifically:\n\n- **Small Evaluation Datasets**: The datasets used for evaluations are relatively small. Evaluating the framework on larger, more diverse datasets would provide a better understanding of its effectiveness across different data scales and domains.\n\n**Complexity of the Method**\nThe Star-Agents framework involves multiple stages and the use of several teacher models, which introduces complexity:\n\n- **Multiple Teacher Models**: The requirement for multiple LLM agents increases the computational cost and complexity of the implementation. Simplifying the framework or exploring methods to reduce the number of required teacher models could enhance its practical applicability.\n- **Three-Stage Process**: The three-pronged approach, while thorough, may be overly complex for some applications. Streamlining the process without sacrificing performance gains could make the framework more accessible and easier to implement.\n\n**Lack of Human Evaluations**\nThe paper does not include human evaluations of test data, which is crucial for assessing the practical quality and usability of the model.\n\n\"Absence of Standard Benchmark Results\"\nThe framework's performance is not evaluated on widely recognized benchmarks, limiting the comparability of its results:\n\n- **Standard Benchmarks**: Including evaluations on standard benchmarks such as GSM8K, HumanEval, or other well-known datasets would provide a clearer comparison with existing methods. This would help situate the Star-Agents framework within the broader context of instruction tuning and LLM performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments.\n\n**Q1**: Small Evaluation Datasets\n\n**A1**: We followed several seminal works[r1,r2], and used widely accepted datasets such as Mt-bench, Vicuna-bench, and the WizardLM testset for our evaluations. These datasets are commonly utilized to assess the effectiveness of LLM's instruction tuning capabilities.\nTo further substantiate the robustness and generalizability of our framework, we also conducted evaluations using datasets from the standard benchmark Open LLM Leaderboard. As shown in Table 1, despite the primary focus of our datasets on instruction tuning capabilities, our method demonstrated  improvements across multiple test sets. This indicates that the Star-Agents framework not only excels in instruction tuning tasks but also performs well on a broader range of datasets.\n\nTable 1. Performance on Open LLM Leaderboard \n\n|Model|ARC|HellaSwag|MMLU|TruthfulQA|Average|\n|-|-|-|-|-|-|\n|Wizardlm-7B|51.60|77.7|42.7|44.7|54.18|\n|Llama-2-7B-evol_instruct|51.88|76.70|45.76|46.10|55.11|\n|Llama-2-7B-star_instruct|54.44|77.64|46.94|46.13|56.29|\n\n**Q2**: Multiple Teacher Models\n\n**A2**:  Compared to a single LLM, using multiple LLMs may increase the time complexity. However, we employed a model parallelism approach during the data generation process, which does not significantly increase the latency of our method while effectively improving the results. Additionally, we have proposed a selection strategy to reduce complexity.\nWe introduced a dynamic selection strategy for LLMs, where during each data generation process, we sample based on the past performance of the LLMs. The sampled LLMs are responsible for generating diverse data. By limiting the number of activated LLMs each time, we can ensure the quality of data generation while significantly reducing the resources required for LLMs inference. In the future, we will continue to research simplified methods to enhance the efficiency of our approach.\n\n**Q3**: Three-Stage Process\n\n**A3**: Thank you for your comments.\nFirstly, Our three-stage method ensures diverse, high-quality data by leveraging multiple LLMs for varied data generation, using dual-model evaluation to identify high-quality, tailored data, and employing an evolution strategy to adjust sampling probabilities for efficient, task-specific data generation. Each stage serves a distinct purpose, and has a clear and justified function, and their integration ensures a streamlined process in practice. \nWe acknowledge the need to simplify the process. Our future work will focus on optimizing data generation, reducing computational costs, and exploring automated tools to enhance usability. These improvements aim to make the Star-Agents framework easier to implement without compromising performance.\n\n**Q4**: Lack of Human Evaluations\n\n**A4**: In our paper, we conducted evaluations on three different test sets using the LLM-as-a-judge method based on GPT-4, as outlined in [r1]. According to [r1], the LLM-as-a-judge approach based on GPT-4 achieves an agreement rate of 85% with human evaluations, surpassing the inter-rater agreement rate among humans, which is approximately 80%. Moreover, numerous seminal works, such as [r3], and [r4], have successfully employed GPT-4 in LLM-as-a-judge to replace human evaluations.\nAdditionally, we conducted a manual inspection of the evaluation results and found that the trends observed with GPT-4 evaluations align with those of human evaluations. Therefore, we adopted the LLM-as-a-judge method based on GPT-4 for our evaluations in this paper.\n\n**Q5**：Standard Benchmarks:\n\n**A5**：In response to this suggestion, we have conducted additional evaluations of the Star-Agents framework using the datasets available on the standard benchmark Open LLM Leaderboard. As shown in Table 1, our framework demonstrated improvements across various test sets, even though our chosen datasets were focused on dialogue performance. We believe that these additional benchmark results will provide a clearer and more robust comparison, demonstrating the effectiveness and versatility of the Star-Agents framework in various scenarios.\n\n**Q6**：What will be the results if base models are larger, e.g. llama 13b or llama 70b?\n\n**A6**：Due to time and computational constraints, we conducted preliminary experiments on the LLaMA2-13B model,  as presented in Table 2 . Our method shows a significant improvement compared to the baseline. Additionally, in the MT-bench First-turn evaluation, Llama-2-13B-star_instruct achieved a score of 6.98, which is close to the Llama2-70B-Chat score of 6.99, a much larger model that has undergone SFT and DPO. While, the teacher models used in our experiments were mostly smaller than 13B. For larger models, it is essential to incorporate more powerful teacher models to further enhance their performance effectively. \nTable2. Performance of Llama2-13B\n|Model|MT-bench|\n|-|-|\n|Llama-2-13B-evol_instruct|5.88|\n|Llama-2-13B-star_instruct|6.28|\n\nReferences:\n\n[r1] Judging llm-as-a-judge with mt-bench and chatbot arena[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[r2] WizardLM: Empowering large pre-trained language models to follow complex instructions[C]//The Twelfth International Conference on Learning Representations. 2024.\n\n[r3] Mixtral of experts[J]. arXiv preprint arXiv:2401.04088, 2024.\n\n[r4] Minicpm: Unveiling the potential of small language models with scalable training strategies[J]. arXiv preprint arXiv:2404.06395, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QQnIbljfsc", "reviewer": "Reviewer_kHqZ", "summary": "The Star-Agents framework presents an advanced approach for enhancing data quality in instruction-tuning of large language models (LLMs) through multi-agent collaboration and automated assessment. By generating diverse instruction data using various LLM agents and evaluating it with a Dual-model metric, this framework ensures both diversity and quality. It dynamically refines its processes to prioritize more effective agents, leading to significant performance improvements. Experimental results show an average increase of 12% in model performance, with some metrics seeing increases up to 40%. This method addresses common issues in data generation such as lack of stylistic variety and overly complex datasets, making it a robust solution for optimizing LLM training datasets.", "questions": "What is the size and composition of the datasets used in your experiments?\nHave you compared your framework's performance using standard benchmarks like MMLU or those from Hugging Face?\nWhat would be the impact of using only Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data?", "limitations": "The authors acknowledge a limitation in their framework, noting that it is currently designed for optimizing single-turn interactions. They suggest that extending this approach to multi-turn scenarios could further enhance its applicability and effectiveness, addressing more complex dialogue systems where contextual continuity and conversational depth are critical.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper stands out for its innovative dual-model scoring system, which refines data quality assessment for large language models (LLMs) and is a key contribution to the field. It demonstrates state-of-the-art results, with the framework achieving an average performance increase of 12% and up to 40% in specific metrics, highlighting its effectiveness. Additionally, the paper is clearly written and well-structured, making complex concepts accessible and easy to follow, which enhances its academic impact and usability.", "weaknesses": "I could not find any weakness apart from mentioned in the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments.\n\n**Q1**：What is the size and composition of the datasets used in your experiments?\n\n**A1**：The datasets used in our experiments consist of 70,000 samples based alpaca evol instruct datasets [r1]. Each sample is paired with an instruction and a corresponding response.  Here is an example: \n\n- Instruction:  Can you suggest a highly specialized conference dedicated to exploring advanced machine learning algorithms and their practical applications in the vast field of artificial intelligence? We are looking for keynote speakers who are prominent figures in the field of AI research, and the event should provide attendees with a chance to participate in immersive, hands-on workshops, as well as poster sessions highlighting groundbreaking research in this field. Additionally, the conference should offer insights into the latest trends and innovations in the world of machine learning.\n\n- Response:  I recommend checking out the Conference on Neural Information Processing Systems (NeurIPS). It is one of the most prestigious conferences in the field of machine learning and artificial intelligence. NeurIPS features keynote speakers from top universities and research institutions, as well as hands-on workshops and poster sessions showcasing cutting-edge research in the field. The conference also offers a chance to learn about the latest trends and innovations in machine learning.\n\n**Q2**: Have you compared your framework's performance using standard benchmarks like MMLU or those from Hugging Face?\n\n**A2**: We evaluated our framework on the standard benchmarks from the Open LLM Leaderboard (including MMLU). As shown in Table 1, although the training datasets we used in our paper focus more on conversational performance, our framework still achieved improvements across multiple standard test sets.\n\nTable 1. Performance on Open LLM Leaderboard \n\n| Model                    | ARC   | HellaSwag | MMLU  | TruthfulQA | Average |\n| ------------------------ | ----- | --------- | ----- | ---------- | ------- |\n| Wizardlm                 | 51.60 | 77.7      | 42.7  | 44.7       | 54.18   |\n| Llama-2-7B-evol_instruct | 51.88 | 76.70     | 45.76 | 46.10      | 55.11   |\n| Llama-2-7B-star_instruct | 54.44 | 77.64     | 46.94 | 46.13      | 56.29 |\n\n**Q3**: What would be the impact of using only Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data?\n\n**A3**: Thank you for your insightful question regarding the impact of using Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data. We have conducted a detailed analysis to address your concerns.\n\nFirstly, we generated 70,000 data samples exclusively using the agent-pair Mistral-ChatGPT. In terms of diversity, we observed a decline in the variation of responses for the same instructions. Regarding quality, some of the data generated was of lower quality compared to using the all of agent-pairs, but it was still superior to the baseline data.\n\nTo further assess the implications, we trained the model using this generated dataset. The performance metrics are presented in the Table 2. The results demonstrate that relying on Mistral-ChatGPT for data generation negatively impacts the model's overall capability compared to using the full set of agent-pairs, but it is still superior to the baseline. This confirms the effectiveness of our approach.\n\nTable 2. Performance of Mistral-ChatGPT agent pair\n| Model                   | Vicuna-bench | Wizardlm-testset | MT-bench | Average |\n| ----------------------- | ------------ | ---------------- | -------- | ------- |\n| Pythia-1B-evol_instruct | 5.07         | 3.55           | 2.56     | 3.73    |\n| Pythia-1B-star_mistral  | 5.23         | 3.88             | 2.69     | 3.93    |\n| Pythia-1B-star_instruct | 5.93         | 3.90             | 2.69     | 4.17    |\n\n\nReferences:\n\n[r1] WizardLM: Empowering large pre-trained language models to follow complex instructions[C]//The Twelfth International Conference on Learning Representations. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9HJP64W5zY", "reviewer": "Reviewer_b1DV", "summary": "The paper introduces the \"Star-Agents\" framework, designed to optimize data for instruction tuning in large language models (LLMs). This system automates the process of enhancing dataset quality by employing a multi-agent approach. Empirical studies demonstrate that the optimized datasets lead to significant performance improvements in models such as Pythia and LLaMA.", "questions": "Please refer to the above.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear and it is easy to follow their method.\n2. The framework's effectiveness is validated through extensive experiments, showing significant performance improvements in various benchmarks.", "weaknesses": "1. What is the overhead of this proposed method, like wall-clock time?\n2. stability: can this method be scalable to large scale dataset optimization, like web-crawl data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments.\n\n**Q1**：What is the overhead of this proposed method, like wall-clock time?\n\n**A1**：Thank you for your insightful question regarding the overhead of the proposed method.\n\nThe computational overhead of our proposed method primarily depends on the inference computational load of the various  Large Language Models (LLMs) used. To provide a clearer understanding, let us consider specific LLMs use in this paper:\n\n - LLM Qwen-14B: During inference with a sequence length of 256 tokens, the computational load is approximately 4×10^12 floating point operations (FLOPs).\n - LLM Phi-2-2.7B: For the same sequence length, the inference computational load is around 7×10^11 FLOPs.\n - LLM ChatGPT: Given that ChatGPT is a proprietary model, we lack precise data on its computational requirements. \n\nNonetheless, for estimation purposes, we can approximate the overall computational cost. Assuming an iterative process involving multiple LLMs (e.g.,  10 LLMs) and a large dataset (e.g., 70,000 samples), the total computation without our methods can be roughly estimated as:\n\n4×10^12 FLOPs (Qwen-14B) × 10  LLMs  × 70,000 samples = **2.8×10^18 FLOPs**\n\nWhile, when the Agent-Pairs Sampling and Instruction Memory Bank were employed,  5 of 10 LLMs are employed to generate data ,  therefore, total computation can be significant reduced and roughly estimated as:\n\n4×10^12 FLOPs (Qwen-14B) × **5**  LLMs  × 70,000 samples = **1.4×10^18 FLOPs**\n\nThis estimation highlights the significant computational requirements, largely dominated by the inference processes of the LLMs. Other components of our method contribute negligible computational overhead compared to the inference load of the LLMs.\n\n**Q2**：stability: can this method be scalable to large scale dataset optimization, like web-crawl data?\n\n**A2**：We appreciate your inquiry about the scalability of our method. Our approach focuses on enhancing the quality and diversity of instruction datasets, and it is inherently scalable to larger datasets. The methodology we proposed is not constrained by dataset size, thus it can be effectively applied to extensive datasets.\n\nSpecifically, there are existing studies that leverage large models to synthesize and process pre-training data [r1]. Our method can be integrated with these existing approaches to optimize and handle large-scale pre-training datasets such as web-crawl data. This integration is a promising area for our future research, and we are committed to exploring this direction further.\n\nReferences:\n\n[r1] : Nemotron-4 340B Technical Report[J]. arXiv preprint arXiv:2406.11704, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PfGK2eiqQf", "reviewer": "Reviewer_4rks", "summary": "The paper introduces the Star-Agents framework, an innovative system designed to enhance the quality of datasets used for instruction-tuning of large language models (LLMs). The framework addresses the challenges of collecting high-quality and diverse data by automating the process through multi-agent collaboration and assessment. The optimized datasets resulted in significant performance improvements. The research contributes to an advanced automated data optimization system that refines instruction samples with suitable complexity and diversity for LLMs, leading to more efficient model alignment and improved performance.", "questions": "1. Could you provide some specific cases? For example, demonstrate how the outputs from different agent pairs differ. \n\n2. What impact would removing some or adding additional agent pairs have on the results?", "limitations": "The authors have mentioned the limitations of this work in the paper and have also suggested potential research directions for addressing these limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel approach to enhancing the quality of datasets for instruction-tuning of LLMs. The concept of using multiple agents to generate diverse instruction data and the dual-model evaluation strategy for data selection are innovative. \n\n2. The paper includes comprehensive experiments with different models, showcasing the effectiveness of the Star-Agents framework. \n\n3. The reported performance improvements of up to 12% on average, and over 40% in specific metrics, highlight the practical significance of the research.", "weaknesses": "1. This paper utilizes multi-agent collaboration, but it seems that the paper does not explore the impact of different numbers of agents or agent pairs on the results. The authors should also specify the exact number of agents used in the experiments in the \"Implementation Details\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments.\n\n**Q1**: Different numbers of agents or agent pairs on the results.\n\n**A1**:  We have conducted experiments to explore the impact of varying the number of agent pairs on the results. As shown in Table 1, we observed that as the number of agent pairs decreases, the model's performance exhibits a corresponding decline. This finding highlights the importance of the number of agents in achieving optimal results. We will include these experimental results and a discussion of their implications in the revised manuscript.\n\nTable 1. Performance using different agent pair\n\n| model                          | agent-pair | Vicuna-bench | Wizardlm-bench | MT-bench | Average |\n| ------------------------------ | ---------- | ------------ | -------------- | -------- | ------- |\n| Pythia-1B-evol_instruct        | 1          | 5.07         | 3.55           | 2.56     | 3.73    |\n| Pythia-1B-star_instruct_2_pair | 2          | 5.23         | 3.88           | 2.69     | 3.93    |\n| Pythia-1B-star_instruct_5_pair | 5          | 5.29         | 3.87           | 2.78     | 3.98    |\n| Pythia-1B-star_instruct_10_pair | 10         | 5.93         | 3.90           | 2.69     | 4.17    |\n\n**Q2**: The authors should also specify the exact number of agents used in the experiments in the \"Implementation Details\" section.\n\n**A2**: Thank you for your insightful suggestion. In our experiments, we utilized a total of 10 agent-pairs. We will include this specific information in the \"Implementation Details\" section of the revised manuscript to ensure clarity and completeness.\n\n**Q3**: Could you provide some specific cases? For example, demonstrate how the outputs from different agent pairs differ.\n\n**A3**: we have included a specific case in the table below to demonstrate how the outputs from different agent pairs differ briefly. Due to the character limitations of this rebuttal, more details are provided in the supplementary PDF attached to the global rebuttal. We kindly ask you to review the detailed examples there.\n\n- Instruction: Can you suggest a highly specialized conference dedicated to exploring advanced machine learning algorithms and their practical applications in the vast field of artificial intelligence? We are looking for keynote speakers who are prominent figures in the field of AI research, and the event should provide attendees with a chance to participate in immersive, hands-on workshops, as well as poster sessions highlighting groundbreaking research in this field. Additionally, the conference should offer insights into the latest trends and innovations in the world of machine learning.\n- Agent-Pair A: I recommend checking out the Conference on Neural Information Processing Systems (NeurIPS). It is one of the most prestigious conferences in the field of machine learning and artificial intelligence. NeurIPS features keynote speakers from top universities and research institutions, as well as hands-on workshops and poster sessions showcasing cutting-edge research in the field. The conference also offers a chance to learn about the latest trends and innovations in machine learning.\n- Agent-Pair B: The conference you are looking for is the International Conference on Machine Learning (ICML). It is a highly specialized conference that focuses on exploring advanced machine learning algorithms and their practical applications in the field of artificial intelligence. The conference is organized by the International Machine Learning Society (IMLS) .......\n- Agent-Pair C: I would highly recommend the NeurIPS (Conference on Neural Information Processing Systems) conference for your needs. NeurIPS is one of the largest and most prestigious annual conferences in machine learning and artificial intelligence. It attracts leading researchers, engineers, and students in the field, providing an ideal setting for sharing new research, ideas, and innovations..........\n- Agent-Pair D: Certainly! I recommend the \\\"Machine Learning and Artificial Intelligence Conference\\\" which will be held in San Francisco, California. The conference will feature keynote speakers such as Andrew Ng, Yoshua Bengio, and Geoffrey Hinton, who are all renowned experts in the field of AI research. Attendees will have the opportunity to participate in immersive, hands-on workshops, as well as poster sessions highlighting groundbreaking research in machine learning. The conference will also offer insights into the latest trends and innovations in the world of machine learning.\n- Agent-Pair E: One highly specialized conference that meets your criteria is the \\\"International Conference on Machine Learning\\\" (ICML). ICML is one of the premier events in the field of machine learning and attracts top researchers, practitioners, and industry leaders. The conference features keynote speeches by renowned experts, in-depth workshops, and numerous poster sessions showcasing cutting-edge research.\\n\\nHere are some key details about ICML:........Another conference that you might consider is the \\\"NeurIPS\\\" (Conference on Neural Information Processing Systems), which is another major event in the AI and machine learning community. NeurIPS has a similar format and focus on advanced research and practical applications.\n- Agent-Pair F: ......... \n\n**Q4**: What impact would removing some or adding additional agent pairs have on the results?\n\n**A4**: Thank you very much for your insightful suggestion. We have conducted additional experiments to investigate the impact of varying the number of agent pairs on our model's performance. As illustrated in Table 1, we observed that reducing the number of agent pairs tends to result in a gradual decline in the model's effectiveness. This trend indicates that the performance of our model is positively correlated with the number of agent pairs. We hope these findings address your concerns satisfactorily.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for the overall positive reviews and helpful feedback, which we have incorporated to improve our work. If any remaining doubts exist, we encourage the reviewers to engage in the discussion so we can clarify them. If all concerns have been resolved, we kindly ask the reviewers to consider raising their scores.

Best Regards,

Submission 15632 Authors

Meta-review: The paper introduces Star-Agents, an innovative framework designed to optimize datasets for instruction-tuning large language models. The approach involves a multi-LLM process to generate diverse instruction data, a dual-model evaluation mechanism to assess difficulty and quality, and a refinement process which prioritizes more effective LLMs. The reviewers generally agree on the strengths of the paper, particularly the IFD-based data selection mechanism, with its impact confirmed in ablations. The approach shows substantial improvements consistently across multiple benchmarks. Despite minor concerns regarding computational cost and the complexity of the framework, this looks like a compelling approach to dataset optimization for LLM instruction-tuning.
Decision: Accept (poster)