Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning

Authors: hang  zhou
yehui  tang
haochen  qin
yujie  yang
renren  jin
deyi  xiong
kai  han
yunhe  wang
seed  data
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:36:47

Abstract: The efficacy of large language models (LLMs) on downstream tasks usually hinges on instruction tuning, which relies critically on the quality of training data. Unfortunately, collecting high-quality and diverse data is both expensive and time-consuming. To mitigate this issue, we propose  a novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. The framework adopts a three-pronged strategy. It  initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. Subsequently, the generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality. Finaly, the above process evolves in a dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework. Optimized datasets have achieved substantial improvements, with an average increase of 12\% and notable gains in specific metrics, such as a 40\% improvement in Fermi, as evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset. Codes will be released soon.

Main paper: 
 introduction: 
 The research and development of natural language understanding and generation have been dramatically accelerated with the emergence and prevalence of LLMs [39, 31, 30] . These models have been extensively applied in a wide range of scenarios, e.g., question answering and text generation, significantly enhancing downstream task performance due to their exceptional ability to follow instructions [3, 53, 49, 10, 28] . Such an instruction-following capability is primarily acquired through a process known as instruction tuning [40, 23, 5] , where LLMs are fine-tuned on instruction data. It is hence widely acknowledged that the quality of instructions plays a pivotal role [5, 20, 48, 29] .Historically, the creation of instruction data for training LLMs has heavily relied on the expertise of human annotators, as evidenced by substantial scholarly contributions [14, 50, 41, 38, 9, 27, 21] . While expert-driven data generation assures the production of high-quality instructions, the enormous volume of data necessary for effective training renders this method economically untenable. In response, recent efforts have shifted towards the utilization of LLMs to automatically generate instructions, thereby mitigating the reliance on costly human annotation [37, 32, 44, 18] . Concurrently, there is a growing emphasis on the generation and selection of challenging examples, grounded in the belief that more complex and difficult instructions can substantially elevate model capabilities [22, 17] .Despite the clear advantages of using LLMs for data generation, several challenges persist in this strategy. Primarily, previous efforts often depend on a single LLM, resulting in data that may lack stylistic variety [4] and encompass a limited range of difficulty levels , which may not be ideal for all models. Additionally, there is a trend towards the creation of exceedingly complex instructions [19, 44, 18] , which may surpass the operational capabilities of models with small parameter scale, thereby hindering their ability to fully capitalize on the data's potential for performance enhancement.To address the aforementioned challenges, we propose the Star-Agents framework, an advanced automatic data optimization system specifically designed to learn and refine instruction samples with suitable complexity and diversity for target LLMs. The framework consists of three main components. First, to increase the diversity of generated data, an instruction data rewriting process involving multiple advanced LLM agents is proposed. This process samples different LLM agents for rewriting instructions and responses separately (referred to as agent-pairs). Next, to select high-quality samples, the generated data undergo a dual-model evaluation function with appropriate complexity as the selection metric. Finally, to balance data diversity and quality, the sampling probability of agent-pairs is adjusted and evolved based on the composite scores of the selected data, identifying agent-pairs that generate high-quality data.Extensive experiments are conducted to evaluate instruction-following capabilities of LLMs on a variety of benchmark datasets, including MT-bench [54] , Vicuna-bench [54] , and the WizardLM testset [44] . Instruction tuning experiments with LLMs such as Pythia and LLaMA, demonstrate the effectiveness of the Star-Agents framework. LLMs trained on data generated by Star-Agent outperform those (the same LLMs) trained on the Evol-Instruct dataset [44] or data selected according to the Instruction-Following Difficulty (IFD) metric [20] . Significantly, the optimized datasets have resulted in an average performance improvement of 12%, with some metrics such as Fermi demonstrating gains of over 40%. 
related work: 
 Our work is related to both instruction data generation and selection. We briefly review these topics within the constraint of space.Instruction Data Generation Datasets like Dolly [7] and OpenAssistant [15] are built from humangenerated instruction data. The ShareGPT dataset, built from conversations between humans and ChatGPT, has been effectively used to improve the instruction-following performance of fine-tuned models [6] . Both Self-Instruct [36] and Alpaca [33] leverage the generation capabilities of GPT-3 to expand seed instructions. The generated instructions undergo filtering to eliminate low-quality instructions while the kept instructions are used to fine-tune the model to enhance the model's ability to respond to instructions. Baize [45] proposes a self-dialogue framework, using questions from popular Q&A websites as starting topics, then having LLMs converse with themselves. CAMEL [16] introduces a role-playing framework where LLMs discuss a given topic when playing a role as either "user" or "assistant". UltraChat [8] uses real-world named entities combined with various text-writing tasks to generate diverse and high-quality multi-turn dialogues for LLMs. Lion [13] introduces the concept of adversarial distillation, using the Imitation-Discrimination-Generation stages to iteratively generate data, refine existing instructions, and produces more complex and diverse instructions to expand the capabilities of the student model. Evol-Instruct [44] uses five manually designed prompts to explicitly guide LLM in rewriting existing simple instructions into more complex ones. The WizardLM model, trained with Evol-Instuct, ranks highly on MT-Bench [54] , highlighting the importance of data quality in training effective LLMs.With the aforementioned methods, it is not difficult to use LLMs to generate large instruction tuning datasets at low cost. However, for instruction-tuned language models, data quality is more crucial than quantity. In this aspect, ALPAGASUS [5] evaluates the effectiveness of instruction data by leveraging ChatGPT. INSTAG [24] automatically generates tags for instruction samples with ChatGPT and keeps diversity by selecting subsets with more tags. Cherry LLM [20] pioneers the self-guided approach, using the IFD metric to measure the difficulty for an LLM to learn an instruction sample. This allows to select instruction samples that significantly enhance training efficiency without resorting to an external model. DEITA [22] first uses ChatGPT to evaluate the complexity and quality of samples, then assesses the diversity of samples based on the distance between model embeddings, thereby guaranteeing complexity, quality, and diversity in the subset. LIFT [46] first guides GPT-4 to generate challenging instructions to expand the data distribution and then uses dimensionality reduction and row variance analysis to select representative high-quality data, where GPT-4 generates a quality score for each instruction. LESS [43] first stores the gradient features of samples in the dataset, then calculates the similarity between a small number of samples from the target task and the training data samples. Based on the calculated similarity scores, it selects the training samples whose gradient features are most similar to those of the target task samples as the fine-tuning instances. Data selection not only improves training efficiency but also prevents low-quality or poison data from undermining model performance by filtering them out [47] . 
star-agents: 
 The aim of our research is to construct a high-quality dataset T of tailored complexity for the target LLM through the enhancement of an initial seed dataset S = (I i , R i ) N i=1 , consisting of instructionresponse pairs (I, R). To this end, we introduce the Star-Agents Framework, depicted in Figure 1 , which is segmented into three steps. The first step leverages a spectrum of advanced LLMs, each trained independently. These models are engaged in a dynamic interaction to generate a diverse data candidate set D(S i ) by sampling agent-pair derived from S i as detailed in Section 3.1. Following this, we apply a dual-model evaluation strategy π(•) to meticulously extract the most suitable data from D(S i ), aiming to substantially elevate the target model's performance. This process is elaborated in Section 3.2. To enhance the effectiveness and efficiency of the Star-Agents framework in generating tailored data, we have developed an evolutionary strategy for the Star-Agents, as discussed in Section 3.3. After these three steps, a tailored high-quality dataset T is obtained from the seed dataset, which is formulated as:T = {arg max d∈D(Si) π (D(S i )) | i = 1, 2, • • • , N }.(1)To improve the instruction-tuned model, it is crucial to assemble a high-quality and diverse instruction dataset [22] . Traditional methods often use a single LLM, such as ChatGPT, for data enrichment.In contrast, our approach employs multiple LLMs to avoid monotonous data distribution. This multifaceted strategy also addresses the limitations and risks of quality degradation on domainspecific tasks associated with using a single model. To counter these challenges, we propose to use an Agent-Pair strategy.Agent-Pair. Utilizing a spectrum of LLMs, each trained with discrepant setting, facilitates the generation of varied responses to given instructions. This diversity is crucial for synthesizing a dataset characterized by high richness [24] .The Star-Agents framework strategically pairs different LLMs to rewrite the instructions in the seed dataset and generate new responses to increase the diversity. With agent-pair (A I j , A R k ), a new instruction data can be generated as follows:EQUATIONwhere A I and A R represent the agents that rewrites the instruction and response to the instruction, respectively.Given the high cost of deploying all agent-pairs, a feasible solution to balance cost and agent diversity is to sample a subset of agent-pairs from the Star-Agents for data generation. Equation 3 formulates this process, where D is collected dataset generated by performing f over all sampled pairs (A I j , A R k ) of instruction agents A I j and response agent A R k with sampling probabilities p jk :EQUATIONM is number of agent-pairs sampled for a single seed sample. The sampling probability p jk is initialized as a uniform distribution and will be updated using the method described in Subsection 3.3 during data generation. Meanwhile, an Instruction Memory Bank that stores high-quality instructions will be updated. To ensure the lower bound of data quality, each iteration will consistently call a fixed set of agent-pairs, referred to as base agent-pairs.Identifying and selecting tailored data from a diverse dataset is crucial for enhancing model performance, especially since the presence of low-quality data can impede model functionality. It is acknowledged that data samples that are lengthy, complex, and challenging significantly benefit the instruction tuning process [22] .Nevertheless, too complex instruction data may be not necessarily benefit model performance. We have observed that for models with 14M and 70M parameters as illustrated in Figure 2 , the Evol-Instruct dataset, though more challenging than the Alpaca dataset, results in diminished model performance. This suggests that intricate examples may surpass the capabilities of small models and be harmful for model performance, despite the advantages of using complex data for large models. Data with a significant gap between the IFD scores of the small and large models will be prioritised.Dual-model Evaluation. To address the issue mentioned above, we propose to use a larger model to evaluate the difficulty of data instances together with the evaluation from a smaller model (target LLM), hence termed as dual-model evaluation. Inspired by Cherry LLM [20] , we employ the IFD metric to measure the degree of difficulty a data sample presents to the target model, which is calcuated asEQUATIONWe assume that for the same sample, stronger model yields a smaller IFD score. When the IFD scores of the two models are close to each other, it indicates that the sample is either too simple or too complex, which is not contributive to effective learning. However, when their IFD scores differ significantly, it indicates that the data is sufficiently complex for the smaller model but still within the capability range of the stronger model. This is a tailored complexity for facilitating learning. The above data assessment method is illustrated at Figure 3 and formulated asEQUATIONNoising data can be endowed with high score since the dual-model metric considers only the relative complexity with the neglect of generation quality. To address this issue, we utilize an LLM as referee for data sample scoring. This involves comparing each data sample in the same batch of diverse data samples generated by selected agent-pairs against a base data sample generated by base agent-pairs. There are three potential outcomes: the base data sample is better, the diverse data sample is better, or a tie, as shown in Appendix A.1. These outcomes are quantitatively assigned as quality scores, thereby avoiding collecting noising instruction samples:π llm =    0, if the base data sample is better, 1,if the generated data sample is better, 0.5, if tie. (6) Finally, the evaluation scores from both the LLM and the dual-model evaluation are combined to compute a final composite score:EQUATIONThis score determines the overall quality and suitability of data for enhancing the model's capabilities. The highest scoring data sample is then selected into dataset T and Instruction Memory Bank as detailed in Section 3.3, ensuring that the chosen dataset maximizes potential improvements in model performance.As mentioned in Section 3.1, we use the joint probability of instruction agents and response agents to regulate the invocation of each agent-pair. Considering the abilities and specialities of each LLM vary, however, sampling each agent-pair with the same probability is not optimal. We hence use the score from Section 3.2 to dynamically evolve the sampling probability. Additionally, since the generation performance of agent-pairs is task-dependent, we also propose an Instruction Memory Bank to select the most suitable agent-pair for particular tasks.Agent-Pair Sampling Evolution. Section 3.2 has introduced the score π, which effectively estimates the quality of generated samples. During each iteration, if the generated samples are of high quality, we will increase the sampling probability of the selected agent-pair, which is updated as follows:EQUATION)The updated sampling probability for the agent-pair of the j-th instruction agent and k-th response agent that successfully process the i-th data sample will be used in the next iteration, where β denotes the evolution rate.This formula adjusts the sampling probabilities based on the effectiveness demonstrated by agent-pairs in generating relevant data. Iterative updates ensure that as the synthesis process advances, the probability of selecting more effective agent-pairs increases, while less effective pairs are gradually phased out.Instruction Memory Bank Evolution. We establish an Instruction Memory Bank storing highquality instructions aiming to accelerate sampling and relate the evolution with task data. When processing a data sample (I i , R i ), we perform a query in the Instruction Memory Bank for I i , retrieving the top n closest matches according to embedding similarity. The associated agent-pairs, identified as highly proficient for tasks similar to I i , are then sampled. We sample l agent-pairs from this pool using normalized probabilities to generate diverse data. Moreover, to foster the creation of a diverse dataset, additional M -l agent-pairs are sampled from the remaining pool using their respective probabilities to assist in data synthesis. As a result, M new samples are generated and then feed for data assessment. Subsequently, the Instruction Memory Bank will continuously evolve by incorporating tailored high-quality data, which get high socres as introduced in Section 3.2. 
experiments: 
 We conducted extensive experiments to evaluate the proposed Star-Agents framework. A wide range of LLMs, benchmark datasets were used in our experiments to guarantee the robustness of our evaluation.Datasets. In alignment with the WizardLM [44] , we adopted the Supervised Fine-Tuning (SFT) dataset, designated as the Evol-Instruct dataset, which consists of 70,000 instruction-response pairs. The instructions in this dataset were refined using "In-Depth Evolving" and "In-Breadth Evolving" methods, which were tailored to enhance the base instructions by adding intricate details or expanding the overall scope, respectively. To guarantee the fidelity of the data, ChatGPT was also integrated as generator into the refinement process. The quality of the instruction data from the Evol-Instruct dataset has been validated as superior [44, 25] ; hence, our research continues to leverage these refined instructions. Employing the Star-Agents framework, our study invokes multiple LLMs to generate diverse and high-quality responses for these instructions. For further enriching our comparative analysis, we employed the Alpaca dataset [32] , comprising 52,000 instruction-following samples. This dataset, developed under the self-instruct paradigm, utilizes the ChatGPT2 instead of text-davinci-003 for a fair comparison [44] . Models. In response to the growing need for cost-effective inference of LLMs at the edge, our study explores the capabilities of target models scaled at 1B and 7B parameters. The 1B models, specifically the Pythia-1B [2] , were trained on roughly 300 billion tokens derived from the Pile dataset. The 7B models, represented by the Llama-2-7B [35] , were trained on an extensive corpus of 2 trillion tokens.During our experiments, we integrated as generator a diverse array of LLMs, as detailed in Table 1 . Our hypothesis posits that models from different development teams possess unique capabilities, yielding rich responses to identical prompts due to the diversity in their training data and strategies.For instance, the Phi2 [11] employed 1.4T tokens of meticulously curated textbook-like data without undergoing Reinforcement Learning with Human Feedback (RLHF) while the Gemma [34] was trained on 6T tokens primarily sourced from English web documents, mathematical content, and code, with subsequent fine-tuning through SFT and RLHF. To ensure the diversity and quality of generated data, we assembled LLMs trained by different teams, widely regarded for their exceptional performance. In pursuit of fostering the generation of data across varying levels of difficulty, the utilized LLMs range from 2.7B to 14B parameters, including even larger models via API access. For a fair comparison with the Evol-Instruct dataset, the most capable model employed was the ChatGPT, which was also used for generating responses within the Evol-Instruct dataset. Notably, the ChatGPT was also served as evaluator to compute the comparison score π llm .Benchmarks. To rigorously evaluate the instruction-following capabilities of AI models, we utilized three widely used benchmarks: MT-bench, Vicuna-bench, and the WizardLM testset. Specifically, MT-bench and Vicuna-bench are designed to test the models' competencies in various complex cognitive tasks, including mathematics, reasoning, complex format handling, and writing through both multi-turn and single-turn dialogues. The WizardLM testset, conversely, extends the evaluation to encompass diverse fields such as technology, biology, and law. It also features varied difficulty levels to facilitate a more nuanced comparison of models' performance disparities. Following established protocols, we employed the Fast-Chat [54] to assess model performances, with GPT-4 acting as the judge model.Baselines. For baseline comparisons, we employed the Pythia-1B and Llama-2-7B, both trained using the Evol-Instruct datasets. The Alpaca datasets were also referenced for comparative analysis, alongside IFD [20] and Random select as an additional comparsion for data selection methods.Implementation Details. We fine-tuned our models (Pythia-1B and Llama-2-7B) over three epochs using the Adam optimizer, with an initial learning rate of 2 × 10 -5 , a maximum token count of 2048, and a batch size of 64. For the Star-Agents, 10 agent-pairs were employed.GPT-4 Automatic Evaluation Based on the findings summarized in Table 2 , comprehensive training sessions were conducted for the Pythia-1B and Llama-2-7B models utilizing three distinct datasets: Alpaca, Evol-Instruct, and the optimally refined Star Instruct datasets. The latter was developed through the application of Star-Agents, which are derivatives of the Evol-Instruct datasets. Through comparative analyses with other contemporary state-of-the-art models, we observe that the SFT-aligned models employing the Star Instruct datasets consistently outperform nearly all aligned counterparts, across all evaluated model families. Notably, at the 1B scale, models trained with the Star Instruct dataset demonstrate significant superiority, surpassing baselines across diverse evaluation datasets. Remarkably, in comparison to models trained with the Evol-Instruct dataset, those utilizing Star Instruct achieve an average absolute improvement of approximately 0.45, which is corresponding to a performance enhancement of about 12%. Additionally, when compared to models trained with the Alpaca dataset, our framework achieves an absolute improvement of 1 point, thereby affirming that the Star Instruct dataset is particularly wellsuited for the Pythia-1B model, significantly boosting its operational efficacy. Additionally, within the 7b model category, the Llama-2-7B-star_instruct outperforms the sRecycled-Wiz-7B-v2 [17] , which is trained on the Evol-Instruct dataset enhanced by Selective Reflection-Tuning. Figure 4a illustrates the Llama-2-7B-star_instruct's performance enhancements across nine metrics, with notable substantial improvements in math, coding and fermi problem-solving, where improvements surge up to 40%. A similar phenomenon can be observed in Figure 4b . Additionally, comparative examples of single-turn and multi-turn dialogues are provided in Appendix A.2, and the performance on the Open LLM Leaderboards of LLMs can be found in Appendix A.4. Average ScorePythia-1B-evol_instruct 3.73 Pythia-1B-IFD [20] 3.26 Pythia-1B-Random 3.62 Pythia-1B-star_instruct 4.17 Main Components. As illustrated in Table 3 , we conducted ablation experiments on the three principal components within the Star-Agents framework. Results indicate that models using solely diversified datasets with random sampling yield a bit lower performance than the baseline. This occurs because the baseline employs data generated by ChatGPT, which is of high quality. In contrast, the diversified datasets draw from a variety of sources, making it challenging to ensure uniformly high quality. Thus, random sampling may introduce low-quality data, leading to diminished model performance. The inclusion of a data selection module subsequently leads to a recovery in model performance, suggesting that this module effectively selects high-quality data suitable for the model. Integration of the evolution strategy also provides a significant improvement, demonstrating that the evolution module can effectively select the most appropriate data generation agent-pairs from a complex array of candidate agent-pairs. Selection Method. As demonstrated in Table 4 , we evaluated a range of conventional selection methods, including both random selection and strategies informed by the IFD [20] . Our dual-model selection strategy significantly outperforms these approaches. Compared to random selection, our method achieves a significant improvement, registering an improvement exceeding 0.5 points on average across a variety of test sets. When compared with the IFD approach, our enhancement approaches a 0.9 point. These findings robustly validate the effectiveness of our dual-model selection strategy, illustrating its superior performance in refining model selection precision using diverse evaluation metrics.Evolution. As depicted in Figure 5 , we analyzed the sampling probability curves of typical agentpairs throughout an iterative evolutionary process. Initially, each agent-pair began with a sampling probability of approximately 10%. Due to its robust performance, the Mistral-ChatGPT receives consistent rewards, which leads to a gradual increase in its sampling probability. By the completion of about 70,000 iterations, this probability has escalated to 30%. In stark contrast, the Phi2-ChatGPT undergoes a steady decline over the same period, with its sampling probability ultimately plummeting to near zero as it is progressively phased out. Concurrently, the ChatGLM3-ChatGPT exhibits a relatively stable trajectory, albeit with a slight downward trend. Evolutionary trajectories present significant discrepancy indicating different generation suitability of different generators on different tasks, where all the differences are captured by our evolution mechanism. 
conclusion: 
 In this paper, we have presented the Star-Agents framework, an automated system for optimizing data to be optimally challenging for target LLMs. This framework has been applied to the open-source SFT datasets, and we conduct training sessions on a variety of model families, adjusting the data to enhance its efficacy. Our empirical investigations include a series of instruction tuning experiments that utilize both multiple baselines and specially optimized datasets on well-known models such as Pythia and LLaMA. Extensive experiments confirm the substantial impact of our method: the optimized tailored datasets result in an average performance enhancement of approximately 12%, with certain metrics, especially those involved in Fermi problem tasks exhibiting increases exceeding 40%, as substantiated by results on benchmarks such as MT-bench, Vicuna bench, and the WizardLM testset. These findings underscore the premise that strategically diverse and tailored data can profoundly improve model alignment and performance. In conclusion, our research details a highly effective automated framework that significantly augments dataset functionality, thus fostering more efficient model alignment.Limitations. Our approach achieves remarkable performance improvements on single-turn instruction datasets. However, it has not yet been evaluated on multi-turn conversations. We hence leave the evaluation on multi-turn instruction datasets and validation on datasets with domain-specific instructions to our future work. 
 
 Appendix: Following the Fast-Chat [54] , the prompts used in the data selection process are as listed in Table 5 .Table 5 : Prompts of data selection for LLMs.System Prompt: Please act as an impartial judge and evaluate the quality of the responses provided by three AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question best. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: " [[A]]" if assistant A is the bset, "[[B]]" ifExamples of the single-turn dialogue and multi-turn dialogue are presented in Table 6 and Table 7 .Table 6 : Single-turn dialogue example.Instruction: A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year. What's the total amount the startup invested in software development over the two years?Llama-2-7B-evol_instruct: The total amount the startup invested in software development over the two years is $12,000.Llama-2-7B-star_instruct: To calculate the total amount the startup invested in software development over the two years, we need to find the total investment in software development for each year and then add them together. In the first year, the startup invested $8000 in software development. In the second year, the startup invested half of that amount, which is $4000. So, the total investment in software development for the two years is: $8000 (first year) + $4000 (second year) = $12000 Therefore, the total amount the startup invested in software development over the two years is $12000. Instruction: Act as a math teacher. I will provide some mathematical equations or concepts, and it will be your job to explain them in easy-to-understand terms. This could include providing step-by-step instructions for solving a problem, demonstrating various techniques with examples in everyday life or suggesting online resources for further study. My first request is "I need help understanding how probability works."Llama-2-7B-evol_instruct: Sure, I'd be happy to help! Probability is the likelihood of something happening, and it's a fundamental concept in math and science. To explain it simply, probability is the chance of something happening out of all possible outcomes. For example, if you flip a coin, the probability of getting heads is 1 out of 2, or 50%. Probability can be used to make predictions and calculate risks, and it's important in fields like finance, statistics, and data analysis. To calculate probability, you can use different formulas depending on the type of problem you're solving. One common method is to use a formula called the probability mass function, which tells you the likelihood of getting a certain outcome. Another method is to use the probability density function, which tells you the likelihood of getting a certain range of outcomes. There are many resources available online to help you learn more about probability, including video tutorials, interactive simulations, and practice problems. Some popular websites for learning probability include Khan Academy, Mathway, and Coursera. I hope this helps! Llama-2-7B-star_instruct: Sure, I'd be happy to help explain probability! Probability is a measure of how likely an event is to occur. It's often expressed as a number between 0 and 1, where 0 represents a very unlikely event and 1 represents a very likely event. To calculate probability, you need to know the total number of possible outcomes and the number of favorable outcomes for the event you're interested in. For example, if you're flipping a fair coin, there are two possible outcomes (heads or tails), and the probability of getting heads is 1/2 or 0.5, because there's only one way to get heads out of two possible outcomes. In everyday life, probability can be used to calculate the likelihood of certain events, such as the chance of rain on a particular day or the probability of getting a certain grade on a test. It's also used in decision-making situations, like calculating the risk of a particular investment or the likelihood of success in a business venture. There are many resources available online to help you learn more about probability, such as video tutorials, practice problems, and interactive simulations. Some popular websites include Khan Academy, Coursera, and edX, which offer free courses and resources on probability and other mathematical concepts.Instruction: What are the differences between Riemannian geometry and euclidean geometry?Llama-2-7B-evol_instruct:Riemannian geometry and Euclidean geometry are two different types of geometries. In Euclidean geometry, the distance between two points is always a straight line, and the angles of a triangle always add up to 180 degrees. In Riemannian geometry, the distance between two points is not always a straight line, and the angles of a triangle can add up to more than 180 degrees. Another key difference between the two geometries is the way they handle curved surfaces. In Euclidean geometry, a flat surface is a rectangle with parallel sides, while in Riemannian geometry, a surface can be curved in any way. This means that Riemannian geometry is better suited for describing the geometry of the universe, as it can handle the curvature of space-time. Overall, Riemannian geometry is a more complex and advanced form of geometry, and it has many practical applications in fields such as physics and engineering.Llama-2-7B-star_instruct: Great question! Riemannian geometry and Euclidean geometry are two different types of geometry that are used to describe and analyze the properties of spaces. The main differences between the two are as follows: 1. Metric: In Euclidean geometry, the distance between two points is defined by the Pythagorean theorem, which states that the sum of the squares of the two sides of a right triangle is equal to the square of the length of the hypotenuse. In Riemannian geometry, the distance between two points is defined by a metric, which is a function that describes how much the shortest path between two points is curved. 2. Curvature: Euclidean geometry deals with flat spaces where the curvature of the Earth's surface is negligible. In contrast, Riemannian geometry deals with curved spaces, such as the surface of the Earth, where the curvature plays a significant role in determining the properties of the space. 3. Geodesics: In Euclidean geometry, the shortest path between two points is a straight line. In Riemannian geometry, the shortest path between two points is called a geodesic, which can be curved and depends on the metric. 4. Tensor Fields: Riemannian geometry involves the use of tensor fields, which are mathematical objects that describe the geometry of a space. These fields are used to calculate the curvature of the space and the behavior of objects moving through it. Overall, Riemannian geometry and Euclidean geometry are two different approaches to understanding the properties of spaces, with Riemannian geometry being more applicable to curved spaces and Euclidean geometry being more applicable to flat spaces.Data examples are shown in Table 8 .Instruction: Given some binary classification data, identify a suitable machine learning model that can be applied to this task.Evol Instruct: The suitable machine learning models for binary classification tasks include: 1) Logistic regression: It is a statistical method that uses a logistic function to model a binary dependent variable. It is a relatively simple and interpretable model that works well with linearly separable datasets.2) Support Vector Machines (SVMs): It is a popular classifier that is used for binary classification tasks. SVMs can handle highdimensional data and work well with datasets that are not linearly separable.3) Random Forests: It is an ensemble learning method that works by constructing a multitude of decision trees at training time and outputs the class that is the mode of the classes output by the individual trees. 4) Neural networks: It is a type of machine learning model that is widely used for binary classification tasks. Neural networks can handle non-linear relationships between variables and are capable of learning complex patterns in the data. The choice of the model depends on the particular characteristics of the data and the specific requirements of the project."Agent-Pair A: There are several machine learning models that can be applied to binary classification tasks, depending on the nature of the data and the problem at hand. Here are a few popular ones: 1. Logistic Regression: This is a simple and effective model that can be used for binary classification problems. It works by estimating the probability of the positive class and using it to make predictions. 2. Decision Trees: Decision Trees are a popular machine learning model that can handle both categorical and numerical features. They work by recursively splitting the data into subsets based on the values of the input features. 3. Random Forest: Random Forest is an ensemble learning method that combines multiple decision trees to improve the accuracy of the model. It works by creating a set of decision trees and then averaging their predictions. 4. Support Vector Machines (SVM): SVM is a powerful model that can handle high-dimensional data. It works by finding the hyperplane that maximally separates the classes in the feature space. 5. Neural Networks: Neural Networks are a set of algorithms that can learn and classify from data. They can handle complex and nonlinear relationships between the input features and the output labels. 1The choice of the model depends on various factors such as the size and complexity of the dataset, the type of features, the computing resources, and the desired level of accuracy.Agent-Pair B: For binary classification tasks, several machine learning models are suitable, depending on the size of the dataset, complexity of the problem, and the availability of features. Here are some popular models that can be applied: 1. Logistic Regression: A simple and interpretable model that works well for small to medium-sized datasets. It's linear and can handle both categorical and continuous features. 2. Decision Trees and Random Forest: These are easy to interpret and can handle both numerical and categorical data. Random Forest is an ensemble method that combines multiple decision trees, reducing overfitting. 3. Support Vector Machines (SVM): Effective for high-dimensional data and can handle non-linear relationships through kernel functions. SVMs are robust to outliers and can be used for both linear and non-linear classification. 4. Naive Bayes: A probabilistic model that assumes features are independent. It's fast, easy to implement, and works well for text classification or when features are not highly correlated. 5. K-Nearest Neighbors (KNN): A simple model that classifies based on the majority class of its nearest neighbors. It's non-parametric and can be useful for small datasets. 6. Neural Networks (e.g., Feedforward, Convolutional, or Recurrent Neural Networks): Deep learning models that can capture complex patterns in large datasets. They are powerful but require more data and computational resources. 7. Gradient Boosting Machines (GBM) or XGBoost: Ensemble methods that iteratively build weak learners (e.g., decision trees) to create a strong classifier. They are often used for structured data and can handle high dimensionality. 8. AdaBoost: Another boosting algorithm that combines weak learners to create a strong classifier. It's particularly useful when dealing with imbalanced datasets. 9. Neural Network Ensembles (e.g., stacking, blending): Combining predictions from multiple models can improve performance and robustness.To choose the best model, it's essential to perform exploratory data analysis, split the data into training and testing sets, and evaluate the models using appropriate metrics (e.g., accuracy, precision, recall, F1-score, or AUC-ROC for imbalanced datasets). Cross-validation can help in selecting the optimal model and hyperparameters Agent-Pair C: A suitable machine learning model for this task would be a Random Forest Classifier. This model is effective in handling both categorical and numerical data, and can handle large datasets with high dimensionality.Agent-Pair D: One suitable machine learning model for binary classification tasks is the Logistic Regression model. Logistic Regression is commonly used for binary classification as it provides the probability of an instance belonging to a particular class. It is a simple and efficient algorithm that works well with linearly separable data. Other popular machine learning models for binary classification tasks include Support Vector Machines (SVM), Decision Trees, Random Forest, Gradient Boosting Machines (GBM), and Neural Networks. The choice of the model depends on the complexity of the data, size of the dataset, interpretability requirements, and computational resources available.Answer: [No] Justification: We will release codes soon. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://chatgpt.com/ 


Decision: The paper introduces Star-Agents, an innovative framework designed to optimize datasets for instruction-tuning large language models. The approach involves a multi-LLM process to generate diverse instruction data, a dual-model evaluation mechanism to assess difficulty and quality, and a refinement process which prioritizes more effective LLMs. The reviewers generally agree on the strengths of the paper, particularly the IFD-based data selection mechanism, with its impact confirmed in ablations. The approach shows substantial improvements consistently across multiple benchmarks. Despite minor concerns regarding computational cost and the complexity of the framework, this looks like a compelling approach to dataset optimization for LLM instruction-tuning.

Reviews: review: paper_summary: paper_summary: The paper presents a framework for enhancing the quality of instruction datasets used for tuning large language models (LLMs). The proposed framework, Star-Agents, leverages multi-agent collaboration to generate, evaluate, and refine instruction data automatically. The approach comprises three main components:

1. **Diverse Data Generation**: The framework employs multiple LLM agents to generate varied instruction data. Each agent pair, consisting of distinct instruction and response agents, creates diverse data samples to enrich the dataset.
2. **Dual-model Evaluation**: This component introduces a two-tiered evaluation mechanism using both small and large models. The evaluation metric considers the difficulty and quality of the generated data, ensuring it is challenging yet manageable for the target model.
3. **Dynamic Refinement**: The framework dynamically adjusts the sampling probabilities of agent pairs based on their performance, promoting the selection of the most effective agents for data generation.

Empirical studies demonstrate the optimized datasets generated by Star-Agents led to performance improvements, with an average increase of 12% and up to 40% in specific benchmarks such as MT-bench, Vicuna-bench, and the WizardLM testset.

------------------------------------------------------------------------------------------------------------------------------------------------------------
Thank you for your reply and I have updated my score accordingly.
strength_weakness: strength_weakness: **Limited Dataset Evaluation**
The paper evaluates the Star-Agents framework on a limited set of datasets, which may not fully capture the framework's robustness and generalizability. Specifically:

- **Small Evaluation Datasets**: The datasets used for evaluations are relatively small. Evaluating the framework on larger, more diverse datasets would provide a better understanding of its effectiveness across different data scales and domains.

**Complexity of the Method**
The Star-Agents framework involves multiple stages and the use of several teacher models, which introduces complexity:

- **Multiple Teacher Models**: The requirement for multiple LLM agents increases the computational cost and complexity of the implementation. Simplifying the framework or exploring methods to reduce the number of required teacher models could enhance its practical applicability.
- **Three-Stage Process**: The three-pronged approach, while thorough, may be overly complex for some applications. Streamlining the process without sacrificing performance gains could make the framework more accessible and easier to implement.

**Lack of Human Evaluations**
The paper does not include human evaluations of test data, which is crucial for assessing the practical quality and usability of the model.

"Absence of Standard Benchmark Results"
The framework's performance is not evaluated on widely recognized benchmarks, limiting the comparability of its results:

- **Standard Benchmarks**: Including evaluations on standard benchmarks such as GSM8K, HumanEval, or other well-known datasets would provide a clearer comparison with existing methods. This would help situate the Star-Agents framework within the broader context of instruction tuning and LLM performance.
questions: questions: What will be the results if base models are larger, e.g. llama 13b or llama 70b?
limitations: limitations: See more details in Weaknesses section.
paper_summary: paper_summary: The Star-Agents framework presents an advanced approach for enhancing data quality in instruction-tuning of large language models (LLMs) through multi-agent collaboration and automated assessment. By generating diverse instruction data using various LLM agents and evaluating it with a Dual-model metric, this framework ensures both diversity and quality. It dynamically refines its processes to prioritize more effective agents, leading to significant performance improvements. Experimental results show an average increase of 12% in model performance, with some metrics seeing increases up to 40%. This method addresses common issues in data generation such as lack of stylistic variety and overly complex datasets, making it a robust solution for optimizing LLM training datasets.
strength_weakness: strength_weakness: I could not find any weakness apart from mentioned in the questions below.
questions: questions: What is the size and composition of the datasets used in your experiments?
Have you compared your framework's performance using standard benchmarks like MMLU or those from Hugging Face?
What would be the impact of using only Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data?
limitations: limitations: The authors acknowledge a limitation in their framework, noting that it is currently designed for optimizing single-turn interactions. They suggest that extending this approach to multi-turn scenarios could further enhance its applicability and effectiveness, addressing more complex dialogue systems where contextual continuity and conversational depth are critical.
paper_summary: paper_summary: The paper introduces the "Star-Agents" framework, designed to optimize data for instruction tuning in large language models (LLMs). This system automates the process of enhancing dataset quality by employing a multi-agent approach. Empirical studies demonstrate that the optimized datasets lead to significant performance improvements in models such as Pythia and LLaMA.
strength_weakness: strength_weakness: 1. What is the overhead of this proposed method, like wall-clock time?
2. stability: can this method be scalable to large scale dataset optimization, like web-crawl data?
questions: questions: Please refer to the above.
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces the Star-Agents framework, an innovative system designed to enhance the quality of datasets used for instruction-tuning of large language models (LLMs). The framework addresses the challenges of collecting high-quality and diverse data by automating the process through multi-agent collaboration and assessment. The optimized datasets resulted in significant performance improvements. The research contributes to an advanced automated data optimization system that refines instruction samples with suitable complexity and diversity for LLMs, leading to more efficient model alignment and improved performance.
strength_weakness: strength_weakness: 1. This paper utilizes multi-agent collaboration, but it seems that the paper does not explore the impact of different numbers of agents or agent pairs on the results. The authors should also specify the exact number of agents used in the experiments in the "Implementation Details" section.
questions: questions: 1. Could you provide some specific cases? For example, demonstrate how the outputs from different agent pairs differ. 

2. What impact would removing some or adding additional agent pairs have on the results?
limitations: limitations: The authors have mentioned the limitations of this work in the paper and have also suggested potential research directions for addressing these limitations.

Comments: title: Thanks for your feedback!
Thanks for your feedback!
Response to the authors
Thanks for the comments.
Thanks for the comments.
Thanks for the comments.
Thanks for your feedback!
comment: Dear Reviewer b1DV,

Thanks for your feedback and valuable suggestions!  We will ensure that the recommended additions are included in the camera-ready version of the paper.

Regards
Thanks for your response and it would be great if we can add this in the camera ready version. I will keep my score.
Dear Reviewer 4rks,

Thank you for your valuable feedback. We understand your concern regarding the balance between performance and cost when increasing the number of agent pairs.

To address this, we conducted additional experiments. As shown in Table 2, we observed that the performance improvement starts to plateau when the number of agent pairs increases to 8. While the model's performance is highest with 10 agent pairs, the difference between 8 and 10 pairs is marginal. Considering the trade-off between performance and cost, we believe that using 8–10 agent pairs is a reasonable and effective choice. This range provides a good balance, ensuring high performance while keeping the cost manageable.

Table 2. Performance using different agent pairs
| model                           | agent-pair | Vicuna-bench | Wizardlm-bench | MT-bench | Average |
| ------------------------------- | ---------- | ------------ | -------------- | -------- | ------- |
| Pythia-1B-evol_instruct         | 1          | 5.07         | 3.55           | 2.56     | 3.73    |
| Pythia-1B-star_instruct_2_pair  | 2          | 5.23         | 3.88           | 2.69     | 3.93    |
| Pythia-1B-star_instruct_5_pair  | 5          | 5.29         | 3.87           | 2.78     | 3.98    |
| Pythia-1B-star_instruct_7_pair  | 7          | 5.48         | 3.92           | 2.79     | 4.06    |
| Pythia-1B-star_instruct_8_pair  | 8          | 5.69         | 3.92           | 2.78     | 4.13    |
| Pythia-1B-star_instruct_9_pair  | 9          | 5.80          | 3.91           | 2.77     | 4.16    |
| Pythia-1B-star_instruct_10_pair | 10         | 5.93         | 3.9            | 2.69     | 4.17    |
Thanks to the authors for resolving my concern. For the Q1, from the table, it seems that 10 agent pairs is not the number that achieve the top of performance. What still concerns me is the balence of performance and cost (if more agent pairs cost more).
Dear Reviewer 4rks,

We sincerely appreciate the time you’ve taken to provide valuable feedback on our paper. In our rebuttal, we have thoroughly addressed all of your initial concerns and included the requested experimental results. If you have any further questions or concerns, we would be happy to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!
Dear Reviewer b1DV,

We sincerely appreciate the time you took to provide valuable comments on our paper. In our rebuttal, we have addressed all of your initial concerns and included the requested experimental results. If you still have any question or concern, we would be glad to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Regards
Dear Reviewer piXU,

We sincerely appreciate the time and effort you invested in providing valuable feedback on our paper. In our rebuttal, we have thoroughly addressed all of your initial concerns and included the requested experimental results. If you have any further questions or concerns, we would be happy to discuss them with you. Additionally, we welcome any new suggestions or comments you might have!

Regards
Dear Reviewer kHqZ,

Thank you for your thoughtful feedback and valuable comments. We appreciate the opportunity to further clarify our methodology.

Regarding the metric used to calculate diversity, we employed cosine similarity as the primary metric. Specifically, we calculated the cosine similarity between the sentence embeddings of the datasets. The diversity metric is inversely related to the cosine similarity; hence, lower cosine similarity indicates higher diversity in the generated data.
In our analysis, we found that the Mistral-ChatGPT dataset exhibited a cosine similarity of approximately 20.8%, which indeed suggests a higher degree of similarity (and thus lower diversity) when compared to the dataset generated using all agent pairs, which had a cosine similarity of 6.9%. 

We hope this clarification adequately addresses your concerns, and we are grateful for your guidance in improving the clarity and rigor of our work.
Thank you for addressing most of my concerns. I have few comments.
"Thank you for your insightful question regarding the impact of using Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data. We have conducted a detailed analysis to address your concerns.

Firstly, we generated 70,000 data samples exclusively using the agent-pair Mistral-ChatGPT. In terms of diversity, we observed a decline in the variation of responses for the same instructions. Regarding quality, some of the data generated was of lower quality compared to using the all of agent-pairs, but it was still superior to the baseline data.

To further assess the implications, we trained the model using this generated dataset. The performance metrics are presented in the Table 2. The results demonstrate that relying on Mistral-ChatGPT for data generation negatively impacts the model's overall capability compared to using the full set of agent-pairs, but it is still superior to the baseline. This confirms the effectiveness of our approach."


Here what metric did you use to calculate diversity?