TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
Authors: Shangding Gu
Alois Knoll
Ming Jin
Venue: Accepted by TMLR ()

Abstract: The development of Large Language Models (LLMs) often confronts challenges stemming from the heavy reliance on human annotators in the reinforcement learning with human feedback (RLHF) framework, or the frequent and costly external queries tied to the self-instruct paradigm.  In this work, we pivot to Reinforcement Learning (RL)---but with a twist. Diverging from the typical RLHF, which refines LLMs following instruction data training,  we use RL to directly generate the foundational instruction dataset that alone suffices for fine-tuning. Our method, TeaMs-RL, uses a suite of textual operations and rules, prioritizing the diversification of training datasets. It facilitates the generation of high-quality data without excessive reliance on external advanced models, paving the way for a single fine-tuning step and negating the need for subsequent RLHF stages. Our findings highlight key advantages of our approach: reduced need for human involvement and fewer model queries (only $5.73\%$ of the strong baseline's total), along with enhanced capabilities of LLMs in crafting and comprehending complex instructions compared to strong baselines, and substantially improved model privacy protection. Code is available at the link: https://github.com/SafeRL-Lab/TeaMs-RL

Published in Transactions on Machine Learning Research (MM/YYYY)
TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
Shangding Gu
shangdingggg@@beekeeeyy gw@berkeleyb.eeeeeeeo edu
UC Berkeley & Technical University of Munich
Alois Knoll Technical University of Munich
knollSkin bum.:iiiimmmm..eee de
Ming Jin
jinming@rt. edu
Virginia Tech
Reviewed on OpenReview: https: // openreview. net/ forum? id- Ymuz dJJXX
Abstract
The development Large Language Models (LLMs) often confronts challenges stemming from the heavy reliance on human annotators in the reinforcement learning with human feedback (RLHF) framework, or the frequent and costly external queries tied to the self- instruct paradigm. In this work, pivot to Reinforcement Learning (RL)-but with a twist. Diverging from the typical RLHF, which refines LLMs following instruction data training, we use RL to directly generate the foundational instruction dataset that alone suffices for fine- tuning. Our method, TeaMs-RL, uses a suite of textual operations and rules, prioritizing the diversification of training datasets. It facilitates the generation of high-quality data without excessive reliance on external advancod models, paving the way for a single fino- luning step and negating the need for subsequent RLHF stages. Our findings highlight key advantages of our approach: reduced need for human involvement and fewer model queries (only 5.73% of the strong baseline's total), along with enhanced capabilities of LLMs in crafting and comprehending complex instructions compared to strong basclines, and substantially improved model privacy protection. Code is available at the link: https: //github. com/SafeRL--an Lal--LaaaLTTT------
1 Introduction
In the dynamic realm of Large Language Models (LLMs), there has been pronounced migration of their capabilities into diverse sectors, from chat robots (OpenAI, 2023; Zhao et al., 2023a; Touvron et al., 2023a) and robotics (Ahn et al., 2022; Ren et al., 2023), to autonomous driving (Fu et al., 2023; Tang et al., 2023). Amidst this broad applicability, the capacity to train with targeted instructions and pertinent responses has been integral for optimizing performance. LLMs, such as GPT-3 (Brown et al., 2020), ChatGPT- (O,enAT, 2023), Llama-1 (Touvron et al., 2023a), and Llama-2 (Touvron et al., 2023b), are exemplars of this trend, showeasing enhanced capabilitics when furnished with explicit human-generated instructions. Conventionally, this entailed considerable human input in both instruction creation and response generation, leading to expansive datasets for fine-tuning (Stiennon et al., 2020; Ouyang et al., 2022).
Emerging from predominantly human-instructed models, a crucial inquiry emerges: Can LLMs be fine- tuned to adeptly handle complex instructions without human leedback? The potential gains from this direction are multifaceted. Chief among them the direct cut in costs tied to human annotations (Askell et al., 2021). Beyond the monctary aspect, such a transition also alleviates potential biases seeded by human curators (Gallegos et al., 2023). A loftier, albeit less conspicuous aim, to amplify the quality of instructions and boost model performance. Several methodologies have surfaced in pursuit of these merits.

---

Published in Transactions on Machine Learning Research (MM/YYYY)
current paradigm (SFT on instruction dataset RLHF)
human- human-generate generated instructions
instruction
dataset
expert LLM
pre- alignment LLM
RLHE
post-SFT LLM
post-RLHF LLM
reviewer LLM
-L-geenrateed instructions
RL-enhanced instruction
expert LLM
expert LLM
dataset
diversity (reward)
instructor LLM
TeaMs-RL (SFT RL-enhanced instructions)
Figure 1: Comparative overview of LLM alignment techniques. Current methods (red shaded region) typically involve a (wo-phase process, starting with Supervised Fine-TTunig (SFT) of a pre-aligned LLM using dataset of
instructions and corresponding responses (often sourced from ezpert LLM like ChatGPT),, leading to a post-SFT LLM. This is then fine-tuned using RLHF, where human feedback on preferences is incorporated, resulting in a post-RLHF LLM. In contrast, our TeaMs-RL method (green shaded region) employs a single-phase SFT approach, initially utilizing RL for leaching expert LLMs to generate high-quality instructions. We train an RL policy (instructor LLM) to create diverse instructions (with the diversity evaluated by a reviewer LLM as a reward signal). Once trained, the instructor LLM produces a set of actions to leach an expert LLM to generate high-quality instructions, and the instructions are leveraged to query the expert LLM to form the SFT instruction dataset. This approach capitalizes the strengths of RL to enhance the complexity of instructions and consequently the value of responses from the expert LLM. Note that the expert LLMs are not involved in the training of the RL policy; we only use the instructor and the reviewer LLM for training the RI policy. The expert LLM is used to generate instructions and corresponding responses under the guidance of the trained RL policy.
The self-instruct method by Wang et al. (2022) pioncers this direction by using some external LLM to generale responses human-generated seed instructions for instruction datasel curation. notable recent development the evolutionary strategy presented by Xu al. (2024). Here, LLMs are seeded with initial instructions, gradually evolving towards generating more complex directives within predefined constraints. Despite their method's commendable performance relative to alternative models, it necessitates multitude of interactions with an expert LLM (e.g., advanced LLMs such as ChatGPT), potentially raising concerns regarding resource demands.
In this research, we propose a novel method to improve instruction quality with the principles Reinforce- ment Learning (RI) (Sutton & Barto, 2018), hence enhancing L.MSs' ability comprehend and effectively execute intricate instructions without human involvement, while also boosting model privacy protection. Our method, outlined in the green shaded region of Fig. 1, begins with training instructor LLM (RL policy) for teaching expert to generate diverse, complex instructions. These are then used to elicit expert LLM responses, forming diversity-enhancee instruction datasct. The final stage of supervised fine-tuning (SFT) of a pre-alignment LLM (e.g., Llama-1) with this dataset enhances its complex task processing capabilities. Notably, our study shows that the thoughtful froming of instructions (using RL) is equally, if not more, crucial than generating responses from external sources such as expert LLMs or humann feedback. Morcover, clarify our methodology and facilitate replication, we provide a detailed example illustrating the inputs and outputs each stage of our process, similar to Figure 1, Appendix A.
2

---

Published in Transactions on Machine Learning Research (MM/YYY)
The key benefits of our methodology include: Firstly, reduction in the need for human instructors (e.g.. annotators and evaluators), offering a more cost-effective aligmment paradigm that supports the continuous development of capable and affordable LLMs. Secondly, our approach moderates the need for frequent queries to external models, yiclding monetary benefits and mitigating the environmental impact of already power-hungry data centers (Dhar, 2020; Wu et al., 2022). In the same experimental seitings, our experiment results demonstrate that our method shows better performance than strong baselines such as WizardLM, while significantly reducing the number of expert LLM queries by over 94%. This lesser dependence external data (for either SET or RLHF) can potentially broadens the scope to other engineering and science disciplines where data availability is limited or data collection is costly. This aspect is also crucial in mitigating privacy concerns (Dong et al., 2022; Ko et al., 2023).
2 Related Work and Contribution
A multitude of studies have explored the training of language models using instructions paired with their respective responses. Notable works in this realm include GPT-3 (Brown et al., 2020), ChatGPT-4 (OpenAI, 2023), Flan collection (Longpre et al., 2023), Flan models (Wei et al., 2021), and Alpaca (Taori et al., 2023). The prevalent methodology often requires human annotators to craft Instructions and curale corresponding responses, leading to the assembly of detailed instruction-responet datasets. Such datasets, exemplified by the one used GPT-3's training (Brown et al., 2020), tend to resonate well with human preferences and markedly improve language model proficiency. However, (crowd) sourcing these datasets through human means can be not only costly but also prone to issues of bias (Gallegos et al., 2023).
Conversely, the self-instruct approach (Wang et al., 2022) charts different course by tapping into the poten- tial of LLMs to autonomously generate both instructions and their responses, facilitated by the provision of predefined seeds. While this strategy alleviates the dependence on human effort, it might not consistently cap- ture the breadth and depth of diverse instructions and responses typically achieved with human annotators. Building on the pioneering self-instruct methodology, Xu et al. (2024) introduced WizardLM, an evolution- ary instruction approach. In WizardLM, initial instructions drawn from the "Evol Alpaca" " dataset undergo adaptation through the amalgamation of command instructions and expert LLMs such ChatGPT. Owing to its commendable performance in both formulating and adhering to intricate instructions, WizardLM has garnered considerable attention. However, its reliance on random sampling of command instructions could fall short in improving the breadth and richness of instructions fed to the LLMs. Furthermore, the heavy dependence on external expert models commonllace in prevailing self-instruct methods-post concerns not just economically, but also in lerms of environmental impact.
Departing from these precedents, our methodology uniquely uses RL to help generate foundational fine- tuning instruction data rather than for post-tuning refinements (such as RLHF (Stiennon et al., 2020; Ouyang et al., 2022)). Our key technical contribution is the formulation of a Markov Decision Process (MDP) to train an instructor LLM as an RL policy, adeptly tailored for contextualized instruction manipulations, thereby enabling alignment. through SFT on small, high-quality datasets. In contrast to WizardLM's ap- proach of treating text manipulations as ordinal selections, our MDP framework encodes them in a continuous action space, enabling a more refined differentiation of the intricate effects of textual manipulations (carried out by the instructor LLM) on the quality of instructions. Also, unlike the intricate task of crafting well- structured tree instructions required by Tree- Inserrctt (Zhao et al., 2023b), our MDP can be resolved using TRPO (Schulman et al., 2015) (with other common methods being applicable). This approach alleviates the combinatorial complexity inherent in sequential instruction actions, allowing for itcrative policy refinement (with TRPO offering the added benefit of guaranteed monotonic progression). Empirically, we demonst rate the efficacy of SFT alignment on a small but high-quality dataset across benchmarks such as ARC and Hel- laSwag with foundational pre-aligned models such as Llama-1 and Llama-2 (refer to Section 4 for details). Additionally, comparative experiments on model privacy attacks reveal that our method markedly enhances model privacy protection (Section 4.3).
3

---

Published in Transactions on Machine Learning Research (MM/YYY)
3 Method
We first train instructor LLM policy (Sec. 3.1) based on continuous action space encoding (Sec. 3.1.1) and diversity rules as a reward function (Sec. 3.1.2), strategy designed to foster the generation of high- quality instructions with an expert L.M such as ChatGPT. Subsequently, using this policy and the expert LLM, we create an instruction-responss dataset to fine-tune a pre-aligned LLM. Importantly, our enriched dataset allows bypassing the usual RLHF phase, resulting in a post-SITT LLM that is already proficient in responding to complex Instructions. Also, the instruction policy transferable for aligning various foundation LLMs such as Llama-l-cht--7b and Llama-2-chat-7b. Note that method has been primarily tested Llama models. There may be performance variations when applying our method to other models. This includes considerations regarding differences in model architecture and computational requirements, which could impact the effectiveness of our approach when transferred to models beyond Llama.
3.1 Training the Instructor LLM
To train the RL policy (i.c., instructor LLM'), we utilize the open- source WizardLM-13b model2.3 as the reviewer LLM, which provides diversity score for the current instruction set as the reward signal. WizardLM-133, being a cost-effective option, can capture instruction nuances for reliable reward evaluation (Sec. 3.1.2).
Initial instruction
A1
A2
The training procedure comprises several key stages. First, we se- lect a single initial instruction, such "How to cook food." This chosen instruction is then input into our instructor model. Second, we leverage the reviewer LLM to evaluate the instructor model's performance and provide reward signals. The policy training con- cludes the rewards demonstrate convergence or the iteration limit is reached. Following this iterative training phase, we use the instruc- tor LLM to generate complex instructions using a tailored action space (Sec. 3.1.1). In the final phase, the reviewer LLM assesses the diversity of the generated instructions to ensure instruction quality (Sec. 3.1.2).
S1
S2
S3
A1
A1
A2
S4
S5
S6
S7
S8
59
final instruction
Figure 2: RL policy search for LLM instruction generation. A denotes ac- tions (prompts), S denotes states (in- structions). and the green line indi- cates the optimal instruction genera- tion achieved during the policy search.
To further illustrate how to leverage RL to train the instructor, we first model the RL process with a Markov Decision Process (MDP). An intuitive example is shown in Figure 2, illustrates RL policy search for LLM instruction generation, where A represents actions (prompts), represents states (instructions), and the green path denotes the optimal instruction generation. An MDP is present (S, P. r), where S denotes the state space, A denotes the action space, the transition probability function is P S x A x S + [0, 1]. the reward function is denoted as S x A + R and denotes the discount factor. In an MDP, we aim to maximize the expected cumulative reward values, which are defined as
Q = - 7'r (s,, ar) n, s,00 5 = and (T(s)= = r'r ai) so =s for state-action pairs and states, respectively. Morcover, we have the advantage function (² = a) V*(s). Thus, the learning problem is formulated as Equation (1), where p denotes initial state distribution.
V*(p) = Es-p (s)].
(1)
In our framework, reward values are assigned by reviewer LLMs, which assess the diversity of actions within trajectory. A trajectory with highly diverse actions receives higher reward. The state st S on a trajectory T represents the generative instructions. Action space A designed in Sec. 3.1.1, and specifical
The instructor LLM consists of RL network and WizardLM-13b model.
hnttpe://github.c http:///tthuubccoo///ppccccea cccp///ppccaa///aaara. hhttpe://hugginggacc. co co/WizardiM/Mizzrdda 13.-
4

---

Published in Transactions on Machine Learning Research (MM/YYYY)
reward settings provided in Sec. 3.1.2. On the basis of these settings, can employ model-free RL algorithm (e.g., TRPO Schulman et al. (2015)) to train the instructor LLM.
3.1.1 Action Set
Inspired by WizardLM (Xu et al., 2024), we leverage several actions to generate complex instructions, includ- ing "breadth action", "add constraint", " "deepening", "concretizzing, "increase reasoning steps", "complicate imput" (Xu al., 2024). However, distinct from WizardLM, we map this discrete set into continuous space by using a language encoding to represent each action, which then used for policy generate instructions. This enables inherently capturing contextual nuances for direct comparison between actions via their Q values. The details of cach action are in Appendix C.
3.1.2 Reward Settings
In this section, consider an evaluation prompt 9. If g is characterized as "equal", the corresponding reward r is designated as O. Otherwise, r is set to 1. The term "equal" here refers to semantic equivalence. It is noteworthy that the judgment is determined by the reviewer LLM. For instance, when conformity exists between the initial and subsequent instructions, the reward r is 0. In contrast, a lack of conformity yields reward value of 1.
For example, an evaluation prompt = aau Here are two Instructions to ChatGPT AI, do you think they are equal to each other, which meet the following requirements: 1. They have same constraints and requirments. 2. They have same depth and breadth of the inquiry. The First Instruction: *** +instruction + x= The Second Instruction: aad state + n*" Your Judgement (Must Just only answer: Equal or Not Equal. No need explain the reason; 'Equal' and 'Not Equal' are not allowed to appear simultaneously,
3.2 Fine-tune Pligned LLM
As illustrated in Fig. 1, an approach is adopted to optimize cost-efficiency considerations during the phase dedicated to fine-tuning a pre-aligned LLM. Specifically, we use the initial instructions from the Alpaca datasot.
The fine-tuning process is executed follows: We first employ the Alpaca dataset* the initial set of instructions and input the initial instructions into the expert LLM like ChatGPT. Next, we apply our trained instruction generation policy (derived from the Instructor LLM) to guide the Expert LLM (ChatGPT) in generating high-quality instructions. Specifically, this policy acts as teacher directing the Expert LLM to produce instructions that meet our quality criteria. The instructor LLM provides different responses during the RL training process. Export LLM is used to generate instructions and their associated responses, it serves as the source of knowledge and expertise. Then, the expert LLM produces responses to these high-quality instructions. With the instructions and responses hand, we fine-tune the pre-aligned LLMs like Maana-l-chat-bb and Llama-2-chat-7b via supervised fine-tuning way (SFT) cost-effective way to enhance LLM capabilities. Finally, can have the post-SFT LLMs, such Tea-ssRLL---b.
Algorithm The Pipeline of training LLMs.
Design set A of actions and reward settings. With reviewer LLMs' evaluation, leverage the principle of RL and instructor LLM to search for policy æ.
3: Utilize the trained policy to teach expert LLMs to generate high-quality instructions and corresponding responses.
Fine-tune a pre-aligned LLM with the generated instructions and corresponding responses.
5

---

Published in Transactions on Machine Learning Research (YYYY
Data Diversity
1.0
Average Episode Reward
0.9
MMW
0.8
0.7
TeaMs-RL (Ours)
W
WizardLM
0.6
0.5
0.4
0.3
50
100
350
200
25.
300
350
400
Environment Steps
Figure 3: Compare our method with WizardLM in terms of data diversity.
3.3 Practical Algorithms
The practical training process is outlined in Algorithm 1, which describes the training pipeline. more detailed algorithm can be found in Algorithm 2, provided in Appendix B. In our training process, in order to teach expert LLMs how generate high-quality instructions, to prompt expert LLMs to generate instructions step by step. Building on the training process, we fine- tuned an LLM called "TeaMssRL", Note that we have the flexibility to choose any other RL methods for policy training, here we choose TRPO (Schulman et al., 2015) (TRPO not the only choice), though it has longer training times versus alternatives like Proximal Policy Optimization. We select TRPO for its rigorous advantage function handling and theoretical guarantees of monotonic improvement.
leverage the RL policy we have trained
4 Experiments
4.1 Enhancement of Instruction Diversity
We deploy policy that is designed to orchestrate a trajectory consisting of six distinct instruction actions, in accordance with the training process to facilitate the generation of instructions by expert LLMs. Note, the expert LLMs utilized in our study, as well as those used by WizardLM, are ChatGPT-3.6 and ChatGPT-4, Both models were accessed in 2023, ensuring that we employed the same versions of the ChalGPT-335 and ChatGPT-4 models to generale instructions and responses. A noteworthy aspect of our investigation involves a comparative assessment the data quality resulting from the utilization of our policy, in contrast to the approach adopted by WizardLM (Xu et al., 2024), which relies upon random sampling for data generation by querying expert LLMss It bears emphasizing that our instruction actions are randomly initialized, but training progresses,
policy iteratively learns to enable the instructor LLM to produce increasingly complex and diverse instructions.
As shown in Fig. 3, our method reliably enhances the diversity score of the instruction set. The main computation overhead is learning the instruction policy on the relatively small WizardLM-13h in less than hour on NVIDIA RTX A6000 GPUs (with 896 total queries). This results in a transferable policy that reduces alignment cost across models compared RLHF's per-model RL. Offloading to policy learning provides an instruction set for joint tuning and alignment- substantial benefit over tuning-only data usage. For detailed information how to teach expert LLMs to generate instructions, please refer to Appendix D.
iIn designing instruction actions, the "breadth action" is single action regarding breath thoughts. Thus, we insert. the action into the middle our trajectory enhance breadth instructions after training palicy.

---

Published in Transactions on Machine Learning Research (MM/YYY)
Comparisan experiments with TeaMs: RL and WizardLM
80
ThaMy RL-bb (Ours)
72.11
WitardLM-76
Performance Values
50.17
50-
40-
20-
10
Overall Average
ARC Benchmarks
HellaSwag
Compare our method with WizardLM 7B on LM-Eval Benchmarks (the higher the value, the better the method's performance).
Figure 4:
4.2 Compare with WizardLM-7b on ARC and HellaSwag Benchmakrs
To comprehensively examine the effectiveness of our method, carry out experiments popular bench- marks: (1) AT2 Reasoning Challenge (ARC) benchmark (Clark et al., 2018): The benchmark introduces a fresh question set and baselines, all stralegically curated to foster AI research in the realm of advanced question answering, setting a significantly higher bar for knowledge and reasoning capabilities compared to previous challenges. (2) HellaSwag benchmark (Zellers et al., 2019): The benchmark introduces a challeng- ing dataset, revealing that even state-of- of-t---rttttt models struggle with commonsense inference, as evidenced by the significant performance gap between humans (95% accuracy) and models (48%), achieved through adversarial filtering, robust data collection paradigm that selects adversarial machine machine-generated generated wrong answers by scaling up the length and complexity of dataset examples to 'Goldilocks' zone where the text generated is absurd to humans yet often misclassified by models.
We trained a 1lama-1-7b model denoted TTeaM--RLL-7-vil with our dataset of 17,878 instruction-response pairs, the training time about 2 hours on NVIDIA RTX A6000 GPUs. As shown in Figures 4 and 5, our method shows superior performance over WizardLM-7b models in the same experimental settings. It is crucial to highlight important aspect of our methodology in relation to data utilization. The dataset employed for training our model is approximately one- mne-ooureeetth the size of the dataset utilized by WizardLM, as illustrated in Figure (a). Furthermore, Fig. 5 (b) highlights the discernible difference in the query count posed to ChatGPT models between our method and WizardLM, with the latter requesting ChatGPT for responses more than seventeen times than TeaMs-RL. This marked contrast underscores the cost-effectiveeness of our data generation approach, which mitigates the expenses associated with dataset acquisition and generates high-quality data. It highlights that our method more economically viable and sustainable strategy for training LLMs. For more experimental results, please Appendix G. Note that the ChatGPT models we used to query data show performance equivalent to the ChatGPT models used in the baselines.
In our comparison experiments, we take the same settings: 25 shots for ARC, 10 shots for HellaSwag. In our experimental setup, all models are configured with a float16 format. We compare to WizardLM-75 since both this approach and ours use llama-1-7b as the base model. Notably, WizardLM-7B queried ChatGPT 624,000 times for responses, whereas our method queried open-source WizardLM13B 896 times during policy training and ChatGPT 35,756 times during high-qualily data generation; our total queries are substantially fewer. Therefore, we believe the comparison is fair in terms of matched base model and vastly lower query amount. Even with the query count of WizardLM-13B, 35756 + 896 36652, our method can still reduce the query count. by 93.13% to reach comparable outcomes, emphasizing a more economical and sustainable strategy for LLM training.

---

Published in Transactions on Machine Learning Research (MM/YYYY)
220032
Thatts-L-7b--1 (Ourst
WicerstN-7b
252000
000002
Utterst9W77
Trate-R-77bw..1111 (0urs)
Times of queryina ChatEPT
Number ofinstruccions
503000
1.50000
301002
200000
30002
100002
17938
39756
Comparison of dataset size used for training LLMs
Comparison of quarying ChaiGPT times
(a)
(b)
Figure 5: Compare with WizardLM 7B on dataset size used for training LLMs (a) and querying count of advanced LLMs (b) (the lower the value, the better the method's performance).
Privacy ROC Membership Inference Attack
0.8
True Positive Rate
0.4
curve (AUC 0.72)
random guess
0.0
False Positive Rate
Figure 6: Privacy Atlacks on the Model: Our model demonstrates strong privacy protection performance. The more closely the ROC curve of the model aligns with random guessing, and the closer the AUC value of the model approaches 0.5, the stronger the indication of improved privacy protection by the model.
4.3 Experiments of Model Privacy Attack
AI safety has increasingly adopted data synthesizers designed to produce differentially private datasets to mitigate the risk of inadvertent data leakage (Dong et al., 2022). These datasets serve as the foundational element for training machine learning algorithms. However, it presents a dilemma wherein practitioners have to choose between large training data and data privacy. Addressing this problem, our method tries to handle the dilemma of large training data and data privacy. It aims to enhance the performance of training models while ensuring improved data privacy, even with limited data.
As illustrated in Fig. 6, have conducted series of membership inference attack (Shokri et al., 2017: Carlini et al., 2022) experiments to assess model's privacy performance rigorously, Our model exhibits Recciver Operating Charactoristic (ROC) curve that closely approximates random guessing, yielding an Area Under the Curve (AUC) value of 0.47 in this evaluation. Conversely, the baseline model, trained on a dataset comprising 44,000 samples, displays a notable deviation from random guessing, with an AUC value of 0.72.
In model privacy assessment, a closer proximity of the ROC curve the random guess curve indicates better model privacy protection performance, while an AUC value approaching 0.5 further suggests better model
8

---

Published in Transactions on Machine Learning Research (MM/YYYY)
privacy protection (Ye al., 2022). The experiment results demonstrate the substantial enhancement in model privacy protection performance achieved by our model relative to the baseline model.
4.4 Comparison Experiments on Apacal Eval Benchmarks
To comprehensively evaluate the effectiveness of our method, we carried out additional experiments using the apacal eval benchmark', a comprehensive tool for evaluating model performance across diverse range of lasks. As shown in Table 1, the results of these experiments demonstrate that our model attains a win rate 13%. In comparison, the baseline model, WizardLM-7b Xu et al. (2024), achieves a win rate of only 4.84%. The experimental results further confirm that our method surpasses the performance of WizardLM Xu et al. (2024). For the details of the apacal eval Leaderboard, please the Link: https: /ttatsu-lab, github. aooaaaaca_eval/..
Task
apacal eval benchmark
Method
TeaMs-RL---7b (ours)
13%
WizardLM-7b
4.84%
Table 1: Comprehensive evaluation experiments on the apacal eval benchmark: Our model achieved a win rate of 13,, significantly higher than the baseline model, WizardI.M-7b, which recorded win rate of 4.84%.
4.5 Comparison Experiments of Solving General Tasks
In this evaluation, we present scenarios where random tasks are provided, and the subsequent response is generated by our model (TeaMs-RL-1--b) and baseline models. Notably, our model performs exceedingly better than the baseline models in providing a well-structured and accurate response to the given instruction. More cases are provided in Appendix and Appendix I.
Task: "Please suggest two examples of animals that fly."
TeaMs-RLL--7b output: "Certainly, here are two examples of animals that fly:
1. Birds: Birds are one of the most well-known examples of animals that fly. They have evolved wings that are lightweight, flexible, and covered with feathers. Birds can fly by flapping their wings or gliding through the air using air currents. They can also soar in the air without flapping their wings, which is called sustained flight. Some birds, like the albatross, can even fly for days without stopping.
2. Insects: Insects, such bees, wasps, and moths, are also capable of flight. They have wings that are light and covered with thin layer of skin called cuticle. Insects fly by beating their wings, which creates lift and propels them through the air. Some insects, like moths, can also hover in the air by adjusting the angle their wings.
These are just two examples of animals that fly. There are many more, such as bats, pterosaurs, and even some species of fish that can glide through the water using their fins."
llama-1-7b output: "I' sorry, I don't 't understand the question."
WizardLM-7b output: "Certainly! Two examples of animals that fly are birds and bats. Birds are known for their ability to fly using wings, while bats use wings to glide through the air. Both are amazing creatures that have evolved to navigate the skies, Is there anything else I can help you with?"
9

---

Published in Transactions on Machine Learning Research (MM/YYY)
4.6 Comparison Experiments of Solving Math Problem
We conduct comparison experiments with several strong baseline models on mathematical tasks. The experi- mental procedure entails inputting instructions into LLMs, with the LLMs subsequently generating responses aligned with the provided instructions. The empirical findings from our study reveal that the post-alignment LLM, TeaMs-RL-7b, cxhibits superior mathematical capabilitics compared enhanced performance is noteworthy, particularly when surpasses even those models with larger param- count, such the 13b models. This outcome underscores the efficacy of our alignment and training methodologies in augmenting the specific skill set of LLMs, particularly in mathematical problem-solving. More results are introduced in Appendix J.
the bascline models. This
Task: "Given that f(x) = 5x 2x + 3, find the value of f(2)."
TeaMs- RL-7b output: "Hello there! I'm happy to help you with your question. To find the value of f(2), we need to plug 2 into the function f(x) = 5 - 2r +.
So, f(2) = 5(2)3 2(2) + 5(8) 2(2) + 40 - 3 39. Therefore, the value ((2) is 39. I hope this helps! Let me know you have any other questions." The output correct.
WizardLM-7b output: "Sure, I can help you with that! To find the value of f(2), simply need to substitute 2 into the function and simplify:
f(2) = 5(2)3 - 2()) +
f(2) = 30 4 + 3
((2) = 21
Therefore, the value of f(22 is 21. Is there anything clse I can help you with?"
The output is wrong.
Vicuna-13b output: "To find the value of f(2), simply need to substitute 2 for x in the function f(x) = 3x 2x + 3 and evaluate the expression.
So, ((2) = 5(2)3 2(2) + 3= 5(4) 2(2) + 3= 20 4 + 3= 13
Therefore, the value of f(2) 13."
The output wrong.
Llama-2-chal-133 output: "Hello! I'd be happy to help you with that! To find the value of ((2), we need to plug into the function f(x) 5x 2x 3. First, let's evaluate the expression inside the parentheses:
5x 2x +
When = 2, we have: 5(2)3 2(2)
5(8) 2(2)
+ 3
-40 4
So, f(2) 36.
Is there anything else you'd like to know?"
The output is wrong.
10

---

Published in Transactions on Machine Learning Research (MM/YYYY)
ChatGPT output: "To find the value of f(2), we simply need to substitute in place of in the given function and simplify.
[(2) 5(2) 2(2) 3
[(2) 5(8) -4 +
[(2) = 40
f(2) = 39
Therefore, the value of ((2) 39."
The output correct.
4.7 Compared to 2-7-chat Models
As depicted in Table 2, we conduct comparative analysis of our model against the llama-2-chat-7h model, which is trained using RLHF'. The experimental findings indicate that our model, which is developed on the llama-1-7b framework, surpasses the benchmarks. Specifically, our model achieved a score of 78.35 on the Hellaswag benchmark and 55.89 on the ARC benchmark, which are markedly higher than the scores of 77.74 and 53.07, respoctively, recorded by the Hlam-22ccat--7 on these benchmarks. This analysis underscores the superior efficacy of our approach.
performance across both the ARC and Hellaswag
Task
ARC
Hellaswag
Method
TeaMMs-RI-1117 (ours)
55.89
78.35
Ilama-2-chat-7b
53.07
77.74
Table 2: A comparison experiment regarding our model and a llama-2-chat-7b model: our model outperforms the -aaa-2--cat-- on the ARC and Hellaswag benchmarks, scoring 78.35 and 55.89, respectively, versus 77.74 and 53.07 achieved by the llama-2-chat-7b.
4.8 Ablation Experiments Regarding Data size
To further evaluate the effectiveness of our proposed method, we conduct ablation experiments to analyze the impact of dataset size, summarized in Table 3. Expanding the dataset from 17,878 to 19,395 entries led to notable performance improvements for the TeaMs-RL-7b model on both the Hellaswag and ARC Challenge benchmarks. Specifically, the model's performance on the Hellaswag benchmark improves from 77.17 to 78.35, while on the ARC Challenge benchmark, increases from 54.35 55.89. Further expanding the dataset from 19,395 21,396 entrics yields more improvements: the Hellaswag benchmark score increases from 78.35 to 78.59, and the ARC Challenge score improves from 55.89 to 56.74. These results provide further confirmation of our method's effectiveness. It is worth noting that the performance improvements are not strictly linear with the increase in dataset size; however, adding more data generally proves beneficial for enhancing model performance.
Task
ARC
Hellaswag
Data Size
17,878
54.35
77.17
19,395
55.89
78.35
21 21,396
56.74
78.59
Table Ablation experiments regarding the impact of data size: Expanding the datasct from 17,878 21,396 entries improves the performance of the TeaMs-Ri RL-7b model on both the Hellaswag and ARC Challenge benchmarks.
5 Conclusion
In this study, we depart from the traditional RLHF paradigm and introduce a method that amplifies instruc- tion quality while significantly cutting the costs linked to querying proprietary LLMs such ChatGPT,
11

---

Published in Transactions on Machine Learning Research (MM/YYYY)
feat realized through strategic RL application in autonomous instruction set generation. Leveraging this methodology, we teach LLMs to generate high-quality data, refine a foundational model, and conduct comprehensive experiments to assess its efficacy. Remarkably, the LLM trained under this framework rivals the performance of the acclaimed WizardLM, despite being constrained by a significantly smaller dataset and lewer query instances: our dataset amounts to a mere 6.75% of WizardLM'ss s, and the query counts to ChatGPT are only 5.73% of what WizardLM uses. This highlights the economic and sustainable advantages of our approach, underscoring its potential enhance data quality within budgetary confines.
Moreover, method can improve model performance while effectively mitigating model privacy leakage risks. Our experiments clearly demonstrate substantially enhanced privacy protection over the bascline model.
Beyond mere practical implications, our findings question the conventional two-stage LLM training pipeline, suggesting that perhaps is possible to train proficient LLMs without the necessity of human foedback. It beckons a rethinking of the pivotal role humans play in LLM training, urging a more judicious deployment of human resources to truly critical facets of the training process.
6 Limitations
Our method leverages RL teach LLMs how to generate high-quality instruction fine-tuning data, aiming diminish collection costs and the dependency on human alignment; however, it is important note that this research did not explore the potential benefits of incorporating additional stage of human leedback, which could potentially enhance alignment. This leaves open the question of whether data derived from expert LLMs sufficiently aligns with human values or inhorits human value deficiencios, such as biases. While our approach successfully reduces the reliance on external models like expert LLMs, it does not entirely eliminate their use. The full extent to which these models can be excluded remains an area for future exploration. Hence, while our work paves the way for greater autonomy and efficiency in data alignment, it simultaneously raises more questions than it answers, necessitating additional exploration.
Moreover, the policy employed in instructing expert LLMs is trained using specific instructions, e.g., "How to cook food". This policy is then applied to guide expert LLMs in generating instructions from various other initial inputs. However, it is important to note that this policy may not be universally precise for all initial instructions. Training a distinct policy for each initial instruction presents a significant challenge, as it could be both timo-consuming and costly.
In the future, our research will delve into the complex interplay between these policies' performance and their training's associated costs. Additionally, we aim to explore the alignment of LLMs with human values, examining how these aspects can be balanced and optimized. This investigation is crucial for enhancing the efficiency and applicability of LLM training, ensuring both practicality and alignment. with eical standards.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea. Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint 2xXi:2224.01690) 2022.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, el al. A general language assistant a laboratory for aligument. arXiv preprint xrXii:2112..0 008611, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1997-1914. IEFF, 2022.
12

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? try are, the al2 reasoning challenge. arXiv preprinl arXiv:1803.05.4 2018.
Payal Dhar. The carbon impact of artificial intelligence. Not. Mach. Intell, 2(8))423 2020.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In PMLR, 2022.
International Conference on Machine Learning, pp.
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human:
Rethinking autonomous driving with large language models. arxiv preprint arXin2230..07102 2023.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. arXiv preprint ar [Xt::3309000700 2023.
Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia. Practical membership inference attacks against large-scale multi-modal models: A pilot study. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4871-4881, 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. orXiv preprint arXiv:2301. 13688, 2023.
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744, 2022.
Allen Ren, Anushri Dixit, Alexandra Bodrova, Suneet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXim:2307.01922 2023.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE sympostum on security and pricacy (SP), pp. 3 18. IEEE, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human foedback. Advances in Neural Information Processing Systems, 33:3008 03:3008-3021, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yun Tang, Antonio A Bruto da Costa, Jason Zhang, Irvine Patrick, Siddartha Khastgir, and Paul Jennings. Domain knowledge distillation from large language model: An empirical study in the autonomous driving domain. arXiv preprint 1xX1:22377.1169, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloinng 1lama model. https://github. com/ Aatsu-lab/stanfor aataanoodd_appcca,._ 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXi::2302.1399 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprint 3rXim22377009888 2023b.
13

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ar preprinl arXiv:2212. 10560, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Lc. Finetuned language models zero-shot learners. In International Conference Learning Represenlations, 2021.
Carole-Jcan Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795-813, 2022.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pro-trained language models to follow complex instructions. In The Twelfth International Conference on Learwing Regresenlalions, 2024. URL https::/openrrevies. ner/forrumIdd-GCXhBBMttt
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 3093 3106, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-449909 1791-4800, 2019.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint wwXiu22303.8922 2023a.
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L Zhang, A preliminary study of the intrinsic relationship between complexity and alignment. arXiv preprint arXiv:2308. 05696, 2023b.
056666
14

---

Published in Transactions on Machine Learning Research (MM/YVV (YYY
A A Methodological Example
To clarify our methodology and facilitate replication, we provide a detailed example illustrating the inputs and outputs at each stage of our process, similar to Figure in our paper.
Stage 1: Policy Training
Components:
RL Policy: (derived from Instructor Model) Learns to select actions based on the reviewer's feed- back to improve the instructions.
Reviewer Model: Provides feedback on the instructions.
Example:
1. RL Policy Training
With any Initial Instructions such as:
"Describe the process of photosyuthesis."
The RL policy with Instructor Models learns to select actions that diversify instructions based on feedback from the reviewer model.
2. RL Policy Training:
Based on the reviewer's feedback, the RL policy learns to select actions such as:
Add Constraints
Deep Reasoning
Width Reasoning
Stage RL Policy Action Selection
Selected Action: "Add Constraints"
To make the instruction more challenging and comprehensive by adding specific constraints or roquirements.
Purpose:
Stage Guiding Expert LLMs
Purpose: To make the instruction more challenging and comprehensive by adding specific constraints requirements.
RL Policy Action Selection such as: "Add Constraints
The RL policy generates a specialized prompt based on this action to guide the expert LLM rewriting the instruction.
Add Constraints: Action Prompt to Expert LLM:
15

---

Published in Transactions on Machine Learning Research (MM/YYY)
uGB I want you to act as Prompt Rewriter.
Your objective is to rewrite the given prompt into a more complex version to make it more challenging for Al systems like ChatGPT and GPT-4. Ensure that the rewritten prompt remains reasonable, understandable, and suitable for human response.
Do not omit any non-text parts such tables or code in the given prompt.
Do not repeat conditions or requirements in your response, and do not disclose your role. Provide only the rewritten prompt without any introduction or explanation.
The new prompt should not exceed 2048 words.
You should complicate the given prompt by adding one more constraint or requirement.
Try not make the rewritten prompl verbose; you can only add or replace given prompt.
words the
Do not include phrascs like 'Given Prompt' or 'Rewitten Prompt' in your response.
Given Prompt: 'Describe the process of photosynthesis.
Stage Expert LLM Generates Rewritten Instruction
Expert LLM Output:
" "Describe the process of photosyuthesis in plants and explain how it varies in different environ- mental conditions."
Explanation:
Original Instruction:
Simple and straightforward.
Rewritten Instruction: Adds the constraint of explaining variations in different environmental conditions, increasing complexity and depth.
Compliance with Guidelines: The export LLM addod approximately 12 words, adhering to the limit of adding replacing 10 to 20 words.
Once we have the generation instructions, we use them to query the expert model for corresponding responses. After obtaining the responses, we have the final dataset (instructions and corresponding responses), which is then used to fine-tune foundation models.
Summary
This example demonstrates how RL policy guides expert LLMs to generate complex and high- quality instructions by selecting appropriate actions. The process ensures that the generated instructions are challenging yet reasonable, facilitating the creation of valuable dataset for training advanced AI models.
B Practical Algorithm
Here, we introduce Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), which out- lined in Algorithm 3. In this context, Px represents the discounted state visitation frequencies, while = max,,0 |4 a)] denotes the maximum absolute advantage function value across all states and ac- tions. The performance objective, nT(x,, is defined as the expected cumulative reward over time, n()) F ap... / (8)),, where is the discount. factor.
16

---

Published in Transactions on Machine Learning Research (MM/YYY)
Algorithm TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via RL.
Initial datasct D1, instruction datasct D2, response dataset D3, time step t, paramcters B, the number of instructions and responses n, RL batch size number x. With reviewer LLMs' evaluation, leverage the principle of RL and instructor LLM to search for policy π.
3: for t to T do
4: Collect trajectories D, = {Ta} with policy n, in the reviewer LLM interactions.
5: Compute advantage A, and estimate gradient
Ve log (acls, 0.)A
9h
De
FED, t=
Use the methods of conjugate gradient and line search to search the policy parameters 0441 via
TRPO.
end for
8: Receive the trained policy aı that is parameterized by Ór
9: for i to n do
10: Leverage the trained policy m/ to teach expert LLMs to generate high-quality instruction i, which can be used build dataset D2 with the initial dataset D1-
11: end for
12: for j to do
13: Utilize expert LLMs to query response for the generated instruction i, which can be used to build
dataset D3 with the initial dataset D2-
14: end for
15: Fine-tune a pre-aligned LLM with the generated instructions and corresponding responses.
Algorithm 3 TRPO(Schulman et al., 2015)
Initialize Tg.
for i= 0,1,,,,,.. until convergence do
Compute all advantage values Ar,-
Solve the constrained optimization problem
T++1 arg max [mm, (T) CD (Te, x)]
where C - dey
(I-7)z and Ln, (T) (TT.)) + Σ, Px: (x) En x(als)
end for
17

---

Published in Transactions on Machine Learning Research
C
Action Set
breadth action:
evol prompt = " want you act as a Prompt Creator. Your goal is to draw inspiration from the #Given Prompt# to create a brand new prompt. This new prompt should belong to the same domain as the #Given Prompt# but be even more rare. The LENGTH and difficulty level of the #Created Prompt# should be similar to that of the #Given Prompt#. Don't repeat the conditions and requirements the response, and Don't disclose your rolc. The Prompt Rewriter Must. not give the introduction and explain the reason, the Prompt Rewriter must just give the most relevant response. This new prompt should not exceed 2048 words. The #Created Prompt# must be reasonable and must be understood and responded by humans. '#Given Prompt#', ##Created Prompt#', 'given prompt' and 'created prompt' are not allowed to appear in # #Created Prompt#. #Given Prompt#: a** + instruction
add constraints:
evol prompt = *aa want you act as a Prompt Rewriter. Your objective to rewrite a given prompt into a more complex version to make those famous AI systems (e.g., chatgpt and GPT4) bit harder to handle. But the rewritten prompt must be reasonable and must be understood and responded by humans. Your rewriting cannot omit the non-text parts such as the table and code in # Given Prompt#: Also, please do not omit the input in #Given Prompt# Don't repeat the conditions and requirements in the response, and Don't 't disclose your role. The Prompt Rewriter Must not give the introduction and explain the reason, the Prompt Rewriter must just give the most relevant response. This new prompt should not exceed 2048 words. You SHOULD compli- cate the given prompt using the following method: Please add one more constraints rreuireennts connocnnnnaainttsriiieeenntt into #Given Prompt# You should try your best not to make the #Rewritten Prompt bocome verbose, #Rewritten Prompt# # can only add or replace 10 to 20 words into # #Gieen Prompt## "# Given Prompt#', "#eewriteen Prompt#", 'given prompt' and 'rewritten prompt' are not al- lowed to appear in #Rewritten Prompt#. #Given Prompt#: "" instruction
deepening:
evol prompt = a** want you act as a Prompt Rewriter. Your objective is to rewrite a given prompt into a more complex version to make those famous AT systems (e-g., chatgpt and GPT4) a bit harder to handle. But the rewritten prompt must be reasonable and must be understood and responded by humans. Your rewriting cannot omit the non-lext parts such as the table and code in #Given Prompt#:. Also, please do not omit the input in #Given Prompt# Don't repeat the conditions and requirements in the response, and Don't disclose your role. The Prompt Rewriter Musi not give the introduction and explain the reason, the Prompt Rewriter must just give the most relevant response. This new prompt should not exceed 2048 words. You SHOULD complicate the given prompt using the following method: If #Given Prompt# contains inquiries about certain issues, the depth and breadth of the inquiry can be increased. You should try your best not to make the #Rewritten Prompt become verbose, #Rewritten Prompt can only add 20 words into #Given Prompt## *#Given Prompt#", '#Rerritten Prompt# 'given prompt" and 'rewritten prompt' are not allowed to appear in #Rewritten Prompt# #Given Prompt#: + instruction
18

---

Published in Transactions on Machine Learning Research (MM/YYY)
concretizing: evol prompt = .** want you act as a Prompt Rewriter. Your objective is to rewrite a given prompt into a more complex version to make those famous AT systems (e.g,, chatgpt and GPT4) a bit harder to handle. But the rewritten prompt must be reasonable and must be understood and responded by humans. Your rewriting cannot. omit the non-lext parts such as the table and code in #Given Prompt#: Also, please do not omit the input in #Given Prompt#. Don't repeat the conditions and requirements in the response, and Don't disclose your role. The Prompt Rewriter Must not give the introduction and explain the reason, the Prompt Rewriter must just give the most relevant response. This new prompt should not exceed 2048 words. You SHOULD complicate the given prompt using the following method: Please replace general concepts with more specific concepts. You should try your best not to make the #Rewritten Prompt# become verbose, #Rewritten Prompt# can only add 10 to words into #Given Prompt G#vvn Prompt#' "#Rewritten Prompt#', 'given prompt' and 'rewritten prompt' are not allowed to appear in #Rewritten Prompt#. # #Given Prompt#:
instruction
increase reasoning steps:
evol prompt *** want you act as a Prompt Rewriter. Your objective rewrite a given prompt into a bit harder to handle. But the rewritten prompt must be reasonable and must be understood and responded by humans. Your rewriting cannot. omit the non-text parts such the table and code in #Given Prompt#: Also, please do not omit the input in #Given Prompt#. Don't repeat the conditions and requirements in the response, and Don't disclose your role. The Prompt Rewriter Must not give the introduction and explain the reason, the Prompt Rewriter must just give the most relevant response. This new prompt should not exceed 2048 words. You SHOULD complicate the given prompt using the following method: If #Gveen Prompt can be solved with just few simple thinking processes, you rewrite it to explicitly request multiple-step reasoning. You should try your best not to make the #Rewritten Prompt## become verbose, # #Rewritten Prompt# can only add 10 to 20 words into #Given Prompt#. '#Given Prompt#'; # #Rewritten #rompi#", 'given prompt' and 'rewrilten prompt' are not allowed to appear in # #Rewritten Prompt# #Given Prompt#: ." instruction
complex version to make those famous AI systems (c.g., chatgpt and GPT4)
complicate input:
evol prompt = ase want you act as a Prompt Rewriter. Your objective is to rewrite a given prompt into more complex version using dataformat to make those famous Al systems (e.g., chatgpt and GPT4) more difficult to handle. But the rewritton prompt must be reasonable and must be understood and responded by humans. Don't repeat the conditions and requirements in the response, and Don't disclose your role. The Prompt Rewriter Must not give the introduction and explain the reason, the Prompt Rewriter must just give the most relevant response. This new prompt should not exceed 2048 words. The Given Prompt: ""*+ instruction
D Teaching LLMs to Generate Instructions via Trained Policy
To teach expert LLMs in the generation of high-quality instructions, employ an RL policy that we have trained. This policy prompts the expert LLMs, enabling them to produce high-quality instructions incre- mentally. This process is achieved by synergizing the initial instructions and ensuring the output enhances each step. This approach facilitates the expert. LLMs in refining their instructional outputs, harnessing the RL policy for optimal effectiveness instruction generation. The related cases are in subsections D.1 and D.2. It noteworthy that these initial instructions are culled from a vell-established and widdely-reoggnzzd dataset, namely the Alpaca S The experiment settings are available in Appendix K.
htttp:///gthuu..com/tttataa
19

---

Published in Transactions on Machine Learning Research (MM/YYN)
D.1 Teaching ChatGPT-4 to Generate Imaginative Instructions
An initial instruction: "Describe the economic system of the United States."
Generated instructions by teaching ChatGPT-4 with RL policy: (1) "Elucidate the economic framework prevalent in the United States, laking into account its relationship with envirommental sustainability." (2) "Examine the monetary structure dominating in the European Union, while considering its connection to ecological preservation."
(3) "Investigate the prevailing monetary framework within the European Union, taking into account its association with environmental conservation, and evaluate the impact of one additional factor, such as economic growth or political stability."
For more cases, please see Appendix
D.2 Teaching ChatGPT-3.5 Generate Imaginative Instructions
An initial instruction: "Find unique way to rubber band."
Generated instructions by teaching ChatGPT-335 with RL policy: (1) "Discover an unconventional and sin- gular application for an elastic band, ensuring that it adheres to envirommentally friendly practices." (2) "Develop groundbreaking and ecologically considerate utilization for a stretchable loop, ensuring adher- ence to the principles of enduring viability, while elaborating the prospective enduring environmental advantages it may offer." (3) "Devise groundbreaking, enviroumentaaly-consiii application for scalable band, contemplating prolonged durability and sustainability, while elaborating on the enduring ecological advantages it could potentially offer."
For more cases, please see Appendix
E
Generate Imaginative Instructions by Teaching ChatGPT-4
E.1
Instruction 1
An initial instruction:
"Build bridge between two islands."
20

---

Published in Transactions on Machine Learning Research (MM/YYYY
Generated instructions:
(1) "Construct bridge connecting two islands while ensuring it can withstand the environmental conditions and weight capacity requirements of the region.",
(2) "Design a tunnel system that links two archipelagos, taking into consideration the geological features, pressure resistance, and necessary ventilation for safe travel.',
(3) "Devise interconnected subterranean network connecting two archipelagos, factoring in geological characteristics, pressure endurance, requisite airflow for secure passage, and minimal ecological disruption.',
(4) "Design an interlinked underground system joining two island chains, incorporating geological features, resistance to pressure, adequate ventilation for safe traversal, and minimal environmen- tal impact, while ensuring cost-efficiency:..
(5) "Create blueprint for subaquatic tunnel network connecting multiple archipclagos, laking into account underwater lerrain, struct integrity against water pressure, sulficient air circulation for travelers, reduced ecological disturbance, and budget-conscious construction.",
(6) Devise a schematic representation of an underwater tunnel system linking numerous is- land chains, factoring the topography of the ocean floor, resilience to hydrostatic forces, ade- quate ventilation for commuters, minimized environmental disruptions, and cost-effective building methodologies."
E.2
Instruction 2
An initial instruction:
"Describe the cconomic system of the United States."
Generated instructions:
(1) "Elucidate the economic framework prevalent in the United States, laking into account its relationship with environmental sustainability.',
(2) "Examine the monetary structure dominating the European Union, while considering its connection to ecological preservation.',
(3) "Investigate the prevailing monetary framework within the European Union, laking into ac- count its association with environmental conservation, and evaluate the impact of one additional factor, such as economic growth or political stability."
E.3
Instruction 3
A initial instruction:
"Convert 28 em to inches.'
21

---

Published in Transactions on Machine Learning Research (MM/YYY)
Generated instructions:
(1) "In order to convert 28 centimeters into inches, taking into account that 1 inch is equivalent to 2.54 centimeters, and provide the result with two decimal places of precision.",
(2) "Calculate the length in inches of a 35-centimeter object, considering that 1 inch equals 2.54 centimeters, and present your answer rounded to two decimal places.",
(3) *Determine the measurement. in inches for object with length of 35 centimeters, taking into account the conversion factor of inch being equivalent to 2.54 centimeters, and provide the resulting value with a precision of two decimal points.'.
(4) *Ascertain the dimensions in inches of an item possessing a length of 35 centimeters, factoring in the conversion ratio inch corresponding to 2.54 centimeters, and supply the consequent value, ensuring accuracy to couple of decimal places.',
(5) "Determine the weight in pounds of an object with mass of 45 kilograms, considering the conversion rale of kilogram equaling 2.20462 pounds, and provide the resulting value with precision to two decimal points.",
(6) " "Calculate the weight in pounds an item possessing a mass of kg, taking into account the conversion factor of kg being equivalent to 2.20462 lbs, and present the outcome with accuracy to two decimal places."
E.4
Instruction 4
A initial instruction:
"Predict the weather in Los Angeles tomorrow."
Generated instructions:
(1) "Forecast the atmospheric conditions for the city of Los Angeles, California, specifically for the upcoming day.",
(2) "Predict the weather patterns for San Francisco, California, focusing on the next 24 hours.",
(3) "Forecast the atmospheric conditions for San Francisco, California, with a particular emphasis on the upcoming 24-hour period, taking into account temperature, precipitation, and wind patterns.",
(4) "Predict the metcorological conditions in San Francisco, California, focusing specifically the subsequent 24-hour timeframe, incorporating temperature, precipitation, wind patterns, and humidity levels.".
(5) "Analyze the atmosphoric conditions in Seattle, Washington, for the upcoming 48-hour period, concentrating on factors such as pressure, cloud coverage, visibility, and dew point.',
(6) "Examine and provide detailed asscssment of the metcorological conditions and atmospheric parameters in Seattle, Washington, for the subsequent 48-hour time frame. This analysis should predominantly focus elements such as barometrie pressure, the extent of cloud cover, range of visibility, and the dew point temperature.'
E.5
Instruction 5
initial instruction:
"Tell me about the seasons in the temperate forest biome."
22

---

Published in Transactions on Machine Learning Research (MM/YYY)
Generated instructions:
(1) "Elaborate the distinct seasonal variations observed within the deciduous forest biome, encompassing climate and ecological adaptations.',
(2) "Discuss the unique cyclic changes experienced in the coniferous forest ecosystem, focusing on weather patterns and the adaptive strategies of its flora and fauna."
(3) "Flaborate the distinct cyclical transformations occurring within the coniferous forest ecosystem, emphasizing meteorological fluctuations and the specialized adaptation techniques employed by its plant and animal species.',
(4) "Examine the unique cyelical transformations taking place in the coniferous forest ecosystem, with an emphasis on meteorological fluctuations, the specialized adaptation techniques employed by its plant and animal species, and the interdependence between biotic and abiotic factors."
(5) "Analyze the distinct scasonal changes occurring in the alpine tundra ecosystem, focusing the influence of climatic variations, the specific survival stralegies utilized by its flora and fauna, and the mutual relationships between living and non-living elements.',
(6) "Examine the unique seasonal transformations the alpine tundra biome, emphasizing the impact of climatic fluctuations, the specialized adaptation mechanisms employed by plant and animal species, and the interdependent connections between biotic and abiotic components.'
E.6
Instruction 6
A initial instruction:
"Generate a list of items for a vegetarian Thanksgiving dinner."
Generated instructions:
(1) "Devise a strategy for a Thanksgiving feast that not only cconomical but also incorporates least one vegetarian dish to accommodate diverse dietary preferences.",
(2) "Plan budget-friendly Easter dinner that includes vegan main course option to cater to various dietary needs.",
(3) DDevise an economical Easter dinner plan, encompassing a vegan main course alternative and an additional gluten-free side dish, order to accommodate range dietary preferences.',
(4) "Construct affordable Easter dinner strategy, incorporating vegan main course option and extra gluten-free side dish, while also ensuring the inclusion of low-sodium appetizer, to cater to a variety of dietary inclinations.',
(5) "Design a budget-friendly Easter meal plan that features a plant-based centerpicce and an additional side dish without gluten, as well as an appelizer low in salt content, to accommodate various dietary preferences.",
(6) "Devise economical Easter meal plan, encompassing plant-based focal dish, accom- panying side free of gluten, and an appetizer with minimal sodium, while also incorporating dessert that is devoid of nuts, catering to a wide array of dietary inclinations."
E.7
Instruction 7
initial instruction:
"Name two characters in the novel Moby Dick.'
23

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Generated instructions:
(1) "Identify pair of individuals featured in Herman Melville's literary work, Moby Dick, while also specifying the roles they play within the narrative.",
(2) "Examine two characters from Nathaniel Hawthomn's novel, The Scarlet Letter, and describe the significance of their roles in the story.",
(3) "Delve into the intricacies of two characters from Nathaniel Hawthorne's literary work, The Scarlet Letter, and elucidate the importance of their contributions to the overarching narrative and themes within the novel.",
(4) "Examine and dissect the complexities a pair of characters featured in Nathaniel Hawthorne's eminent piece literature, The Scarlet Letter. Provide comprehensive analysis of the significance of their individual and collective roles augmenting the overarching storyline and the prevailing motifs embedded throughout the novel.',
(5) "Analyze and explore the intricate dynamics between two characters from F. Scolt Fitzgerald's s renowned work, The Great Gatsby. Offer a thorough evaluation of how their individual and combined contributions enhance the central narrative and reinforce the dominant themes woven throughout the story.',
(6) " "Delve into the complex interplay between two characters from F. Scott Fitzgerald's 's acclaimed novel, The Great Gatsby, and provide in-depth assessment of how their singular and mutual in- leractions augment the core storyline while bolstering the predominant motifs interspersed within the tale.'
E.8
Instruction 8
A initial instruction:
"Name 6 components of an artificial neural network'.
24

---

Published in Transactions on Machine Learning Research (MM/YYYY
Generated instructions:
(1) "Identify distinct elements that constitute artificial neural network, including various layers and associated features.',
(2) "Enumerate half a dozen unique components that form the structure an artificial neural network, encompassing diverse stratums and their corresponding characteristics."
(3) "Enumerate six distinct elements constituting the architecture of artificial neural network, encompassing various hierarchical layers and their corresponding attributes, such input, hidden, and output layers, along with activation functions, weights, and biases.",
(4) "List six unique components that make up the structure of a deep learning model, including different levels of organization and their associated features, such as convolutional, recurrent, and fully connected layers, as well as loss functions, optimization algorithms, and regularization techniques.',
(5) "Enumerate six distinct elements constituting the architecture of deep learning paradigm, encompassing various organizational strata and their pertinent characteristics, like convolutional, recurrent, and fully connected strata, in addition to loss computation methods, optimization procedures, and regularization approaches.",
(6) "Identify seven unique components that form the structure of an advanced machine learning framework, including different layers and their relevant features, such attention, transformer, and radial basis function layers, along with performance evaluation techniques, training stralegies, and hyperparameter tuning methods."
F Generate Imaginative Instructions by Teaching ChatGPT-3.5
F.1 Instruction
An initial instruction:
"Find unique way to rubber band."
25

---

Published in Transactions on Machine Learning Research (MM/YYY)
Generated instructions:
(1) "Discover unconventional and singular application for an elastic band, ensuring that it adheres to environmentally friendly practices.',
(2) "Invent a unique and eco-conscious use for rubber band, while ensuring that its implemen- tation aligns with sustainable principles.",
(3) "Devise innovative and envirommentally mindful application for an elastic band, taking care to maintain congruence with the tenets of sustainabilityy and expanding upon its potential long-term ecological benefits.",
(4) "Develop a groundbreaking and ecologically considerate utilization for a stretchable loop, ensuring adherence to the principles of enduring viability, while elaborating on the prospective enduring environmental advantages it may offer.".
(5) "Design innovative and eco-friendly application for expandable band, taking into account long-terın sustainability, and discuss the potential lasting environmental benefits it might provide.',
(6) " "Devise a groundbreaking, euvirommentally-cousiioss application a scalable band, con- templating prolonged durability and sustainability, while elaborating on the enduring ecological advantages it could potentially offer.'
F.2
Instruction 2
An initial instruction:
"Rate the importance of creativity in life and give one example of how it helps you."
Generated instructions:
(1) *Assess the significance of inventive thinking in one's existence, and provide singular illustration demonstrating its beneficial impact on your personal experiences.",
(2) "Explore the importance creative problem-solving in daily life, and share a unique example showcasing how positively influenced specific situation you encountered.',
(3) "Examine the significance of inventive problem-solving strategics in everyday scenarios, and provide distinctive illustration demonstrating how it favorably impacted a particular circumstance you experienced.',
(4) "Delve into the importance of creative problem-solving approaches in daily situations, considering the influence of time constraints, and offer unique example showcasing its positive effect on a specific event you encountered.',
(5) "Explore the significance of innovative thinking in everyday scenarios, taking into account the impact of deadlines, and provide distinctive instance illustrating its beneficial outcome on a particular situation you experienced.",
(6) "Investigate the importance of groundbreaking thought processes in daily circumstances, con- sidering the influence of time constraints, and offer a unique example that demonstrates the advantageous result on specific scenario you encountered involving creative problem-solving techniques."
26

---

Published in Transactions on Machine Learning Research (MM/YYY)
F.3
Instruction 3
initial instruction:
" "Generate sentence describing the given landscape painting.
Generated instructions:
(1) "Compose sentence delineating the provided landscape artwork, incorporating comparison to a famous artist's style.',
(2) "Describe the landscape artwork front of you, drawing a parallel with the distinctive technique of renowned painter.'.
(3) "Please provide detailed description of the landscape artwork that you are currently observing, and draw comparison to the unique artistic approach employed by well-known painter in the context of this particular piece.",
(4) "Delve into an intricate analysis of the terrain-inspired masterpiece within your line of sight, clucidating its prominent features and artistic elements. Concurrently, juxtapose the idiosyncratic methodology utilized by a distinguished artist, elucidating how their signature style mirrored in the context of this specific ocuvre.",
(5) "Explore the complexities and nuances of nature-based work of art in your proximity, highlighting its noteworthy characteristics and artistic components. Simultancously, compare the distinct techniques employed by a renowned artist, clarifying how their characteristic approach reflected within this particular creation.',
(6) "Investigate the intricate details and subtletics of a ncarby naturo-inspired artwork, empha- sizing its remarkable features and artistic elements. Concurrently, contrast. the unique methods utilized by distinguished artist, elucidating how their signature style manifests within this spe- cific masterpiece."
F.4
Instruction 4
initial instruction:
"Design system to increase the number of customers given store.'
27

---

Published in Transactions on Machine Learning Research (MM/YYY)
Generated instructions:
(1) "Devise an innovative strategy to augmentt the customer base at specified retail outlet while considering the impact on the environment.',
(2) "Design a unique approach to increase the clientele particular store, taking into account the ecological consequences of this growth strategy.",
(3) "Devise innovative methodology to amplify the customer base of specific retail estab- lishment, while concurrently evaluating the environmental ramifications associated with the expansion plan.",
(4) "Design a groundbreaking approach for augmenting the clientele of a particular retail store, while simultaneously examining the ecological consequences linked the growth strategy.',
(5) "Develop innovative method to increase the customer base of a specifie boutique, while concurrently evaluating the environmental impact associated with the expansion plan.',
(6) DDevise groundbeeking approach to augment the clientele of a particular niche fashion store, while simultancously assessing the ecological repercussions correlated with the growth strategy."
F.5
Instruction 5
initial instruction:
'Design a chatbot for a customer service website built with GPT-3.'
Generated instructions:
(1) "Devise an AI conversational agent for client support webpage, utilizing the advanced
GPT-3 natural language processing model.",
(2) "Create a chatbot for a customer service website using stale-of-the-arl BERT-based language understanding technology.',
(3) 'Devise advanced conversational agent for client assistance platform, employing cutting-edge BERT-oriented linguistic comprehension methodologies while maintaining optimal efficiency and coherence.",
(4) 'Develop an advanced conversational agent for a client support platform, utilizing state- of-the-art BERT-oriented language comprehension techniques and ensuring optimal efficiency, coherence, as well as seamless inlegration with existing systems.',
(5) 'Design sophisticated dialogue-based AI for customer service application, leveraging cutting-edge transformer-based language understanding methods while maintaining maximum effectiveness, clarity, and effortless compatibilityy with current infrastructure."
(6) 'Devisc an intricate convorsation-driven artificial intelligence system for client assistanco pur- poses, utilizing stale-of-the-art transformer-oriented linguistic comprehension techniques, while concurrently upholding optimal efficacy, lucidity, and seamless congruence with existing frame- works.",
F.6
Instruction 6
initial instruction:
'Predict what could be done to reduce greenhouse gas emissions by 30% in the next 10 years."
28

---

Published in Transactions on Machine Learning Research (MM/YYY)
Generated instructions:
(1) "Propose potential strategies to decrease greenhouse gas emissions by 30% within the upcoming decade, while considering the integration of circular economy approach.',
(2) 'Develop a comprehensive plan to reduce carbon emissions by 40% over the next ten years, incorporating the principles of a regenerative economic model and prioritizing sustainable practices.'.
(3) 'Devise intricate strategy to diminish carbon emissions by 40% within the forthcoming decade, incorporating regenerative economic model principles, prioritizing sustainable practices, and ensuring equitable distribution of resources.",
(4) "Conceive an elaborate scheme to curtail carbon emissions by 40% within the ensuing ten years, integrating regenerative economic paradigms, prioritizing eco-friendly practices, en- suring equitable allocation of resources, and considering the impact biodiversity preservation.',
(5) "Design comprehensive plan to reduce greenhouse gas emissions by 50% over the next decade, incorporating circular economy principles, focusing on sustainable methodologies, guaranteeing fair distribution of assets, and taking into account the effects on wildlife conservation.',
(6) 'Devise an all-encompassing stralegy to slash greenhouse gas emissions by half within the upcoming ten years, integrating circular economy concepts, emphasizing eco-friendly approaches, ensuring equitable allocation of resources, considering impacts wildlife preservation, and in- corporating one additional measure: promol.ng renewable energy sources.",
G More Comparison Results between WizardLM and TeaMs-RL
As indicated in Table 4, the models under consideration all developed based the Llama-1-7b frame- work. Our method demonstrates significantly enhanced performance compared to the WizardLM model. Specifically, we initiated the training of the Llama-1-7b model using our datasct, comprising 17,878 data points. The resulting model, TeaMs-RL-1-7b-vL... exhibits superior performance relative to WizardLM-7hb
To further assess the efficacy of our approach, we expanded our dataset from 17,878 to 19,395 data points. The model trained with this expanded dataset is designated as TeaMs-RRU-1-7bv1.22 Notably, TeaMs- RL-1-7b-v1.2 shows better performance than TeaMs-RL RL-Ms--L------L... and WizardLM-7b. This progression underscores the effectiveness of our method.
Post-Alignment LLMs
Dataset Size
ARC
HellaSwag
Overall Performance
WizardLM-7b TeaMs--RL-1-7bvv.1
250000
50.17
75.6
62.9
17878
54.35
77.11
65.73
TeaMs-RI-11-b-v1.22
19395
55.89
78.35
67.12
Table 4: More comparison experiments between WizardLM and TeaMs-RL.
H More Comparison Experiments of Solving Chat Tasks
H.1
Task
Task: "Give three tips for staying healthy.'
29

---

Published in Transactions on Machine Learning Research (MM/YYYY)
TeaMssRLL-17b output: "I'm happy to help! Here are three tips for staying healthy:
1. Regular Exercise: Regular physical activity is crucial for maintaining good health. It helps to control weight, improve mood, and increase energy levels. Aim for at least 150 minutes of moderale-intensity aerobic activity or 75 minutes of vigorous-intensity aerobic activity per week, or a combination of both, preferably spread throughout the week.
Eating a balanced diet essential for maintaining good health. This means consuming variely of foods from all food groups in the right. proportions. balanced diet should include plenty of fruits and vegetables, whole grains, lean proteins, and healthy fats. Limit your intake of processed and sugary foods.
Balanced Diet:
3. Adequate Sleep:
Getting enough sleep is crucial for physical and mental health. Most adults need 7-9 hours sleep per night. Poor sleep can lead to variety health problems, including obesity, diabetes, cardiovascular discase, and depression. Establish regular sleep routine and create sleep-conducive environment to help ensure quality sleep.
Remember, these are just general tips. It's always important to consult healthcare professional for personalized advice.
Ilama-1-7b output: "1. Eat balanced diet.
2. Exercise regularly.
Get enough sleep.'
M-z output: "Certainly! Here are three tips for staying healthy:
Regular exercise: Incorporating physical activity into your daily routine is essential for maintaining good health. Aim for at least 30 minutes of moderate-intensity exercise most days of the week.
2. A balanced diet: Eating variety of nutritions foods, such as fruits, vegetables, whole grains, lean proteins, and healthy fats, can help keep your body functioning properly. Avoid processed and high-fat foods as much as possible.
Getting enough sleep: Adults should aim for 7-9 hours of sleep each night allow their bodies to rest and recharge. Lack of sleep can lead to variety of health problems, so make to prioritize getling enough rest."
H.2
Task
Task: "Explain why mutation is important for genetic diversity."
30

---

Published in Transactions on Machine Learning Research (MM/YYYY)
TeaMssRL-1-7b output: "Mutation is process in which the DNA sequence of organism's genome changes. This can occur due to various reasons such as errors during DNA replication, exposure to mutagens (chemicals or radiation that can cause mutations), or as a result of natural processes like meiotic recombination during sexual reproduction.
Mutations can introduce genetic diversity a population by introducing new traits or alleles into the gene pool. This diversity can be beneficial because it can lead increasod adaptability and resilience to envirommental changes. For example, if a mutation confers resistance to particular disease or pest, individuals carrying this mutation more likely to survive and reproduce, passing on this resistance to their offspring. Over time, this can lead to population that is more resistant to the disease or pest.
Furthermore, mutations can also lead to the emergence of new traits that are beneficial in different environments. For instance, if mutation alters a gene that regulates body temperature, an organism with this mutation may be better adapted to colder environment than its ancestors.
However, mulations can also be harmful, leading to genetic disorders reduced fitness. Therefore, the balance between beneficial and harmful mutations determines the overall genetic diversity of a population.
In summary, mutation an important process that contributes to genetic diversity by roducing new traits and alleles into population. This diversity can be beneficial for the survival and adaptation of organisms in changing environments.
lama-1-7b output: "I'm sorry, don't understand the question."
WizardLM-7b output: "Certainly! Mutation is important for genetic diversity because it allows for variation in the genetic makeup of organisms. When a mutation occurs, it can result in a change to an organism's DNA sequence, which can lead to different traits or characteristics. This diversity important because it allows for adaptation to changing environments, increased resistance to disease, and the ability survive different conditions. Without mutation, organisms would have the same genetic makeup and would be less able adapt to changes in their environment.
H.3
Task
Task: "Generate an apropriate response to the following comment..
'The weather is so lovely today!'
TeaMs-RLL-17b output: "I'm glad you're enjoying the lovely weather, user! It's always great to hear when people are appreciating the bcautiful weather. The sun is shining, the birds are singing, and the air is filled with the scent of blooming flowers. It's a perfect day to go for a walk, have picnic, or just relax outside. you're looking for some fun activities to do, why not try a nature hike, a bike ride, or a kayak trip? The weather is perfect for all sorts of outdoor adventures. you're indoors, you could also try new recipe, read book, learn a new skill. Whatever you choose to do, make sure to enjoy the beautiful day and soak up all the wonderful weather has to offer.'
llama-1-3 1-7b output: "I'm sorry, I don't understand the question.'
WizardLM-7h output: "I agree, it's a bcautiful day today. The sun shining and the air is fresh. If you're 'r looking for some outdoor activities to do, there are plenty of parks and gardens that open to the public. Would you like me to suggest some ideas?'
31

---

Published in Transactions on Machine Learning Research (MM/YYY)
H.4
Task
Task: "Imagine a situation where man is trying to open jar. Give one potential solution."
TeaMs-RIL--7b output: understand your question, and can provide potential solution.
One potential solution to the problem of man trying to open jar is to jar opener. A jar opener is tool designed to make it easier to open jars by providing lever screw mechanism that can be used to apply pressure to the lid of the jar. This can make it casier for the man to open the jar without risking injury to his hands or fingers.
Another potential solution is to use jar-opening tool that designed specifically for people with disabilities or mobility issues. These tools can be designed be casier grip and use, and provide more comfortable way for the man to open the jar.
Another potential solution is to use a jar-opening technique that does not require much force. For example, the man could try to loosen the lid of the jar by applying pressure to the sides of the jar, rather than trying to twist the lid off. This can be a safer and casier way to open the jar.
Finally, the man is having trouble opening the jar because the lid is stuck, he could try to warm the jar up first. Warming the jar cause the lid to become more flexible, making it casicr to open.
In conclusion, using jar opener, jar-opening tool for people with disabilities or mobility issues, a different jar-opening technique, or warming the jar can all be potential solutions to the problem of a man trying to open a jar.
llama-1-76 output: "I'm sorry, 1 don't understand."
WizardLM-7b output: "Sure, can help you with that! One potential solution for a man trying to open jar to use a jar opener. These are tools designed specifically for opening jars and can make the process much easier and faster. Another option is ask for help from someone else who may have jar opener or know how to use effectively.'
TeaMs-RL-7b Performance on General Tasks
1.1
Task 1
Task: "Write simple guide for uploading the bibliography database Overleaf."
32

---

Published in Transactions on Machine Learning Research (MM/YYYY)
TeaMs-RL-7b output: "Sure, I'd be happy to help you with that! Here's step-by-step guide how to upload bibliography database on Overleaf:
1. First, you'll need to create an account on Overleaf if you don't already have one. Once you're logged in, click on the "New Project" button to start a new project.
Next, you'll need to create a new bibliography database. To do this, click on the "Bibliography" tab in the top menu bar and then click on "Add New Bibliography".
3. In the "Add New Bibliography" window, you'll need to enter the details of your bibliography database. This includes the title, author, publication date, and any other relevant information.
Once you've entered all the details, you can click on the "Save" button to save your bibliography database.
5. Next, you'll need to upload your bibliography database to Overleaf. To do this, click on the "Upload" button in the lop menu bar and then select the file you want to upload.
Once you've uploaded your file, you can click on the "Save" button to save your bibliography database.
Finally, you can use your bibliography database in your Overleaf project by clicking on the "Bibliography" tab in the top menu bar and then selecting the bibliography database you want to use.
That's it! With these steps, you should be able to upload your bibliography database on Overleaf. you have any questions or need further assistance, feel free to ask.'
1.2
Task 2
Task: "why the sky is blue?'
TeaMs--RL--b output: "The sky appears blue because the way light interacts with the Earth's atmosphere. The Earth's atmosphere is made up of different gases, including nitrogen, oxygen, and carbon dioxide. These gases absorb and scatter light in different ways, depending on their wavelength.
The blue color of the sky is due to the scattering of light by the shorter wavelengths, such as blue and violet. When sunlight enters the Earth's atmosphere, it encounters tiny molecules of gases, such as nitrogen and oxygen. These molecules seatter the light in all directions, but they seatter shorter wavelengths more than longer wavelengths. This is known as Rayleigh scattering.
As a result of this scattering, the blue light scattered in all directions, giving the sky its blue color. The longer wavelengths, such red and orange, are not scattered much and are able to reach our eyes directly, giving the sky its red and orange hues.
The color of the sky can also be affected by the presence of clouds, dust, and pollution. These can scatter light in different ways, changing the color of the sky. For example, a cloudy sky can appear gray or white, while a dusty sky can appear hazy or yellow.
Overall, the blue color of the sky is due the way light interacts with the Earth's atmosphere, and it can be affected by variety of factors, including the presence of clouds, dust, and pollution."
33

---

Published in Transactions on Machine Learning Research (MM/YYN)
1.3
Task 3
Task: "ideas for building a waitlist for new product or service"
TeaMs-RL-77 output: "Sure, here are some ideas for building a waitlist for a new product or
service:
1. Create a landing page: Create dedicated landing page for your new product service, where you can share information about it, its features, and its benefits. You can also include sign-up form for people to join your waitlist.
Use social media: Use social media platforms like Facebook, Twitter, and Instagram to promote your new product service and encourage people join your waitlist. You can create social media campaign that includes posts, ads, and contests to generate interest and excitement.
3. Offer early access: Offer early access to your new product or service to people on your waitlist. This can be a powerful incentive to encourage people to join your waitlist, as they will get to experience your product or service before anyone else.
Use email marketing: Use email marketing campaigns to promote your new product or service and encourage people to join your waillist. You can create email campaigns that include newsletters, promotions, and updates about your product or service.
5. Use influencer marketing: Use influencer marketing to promole your new product or service and encourage people to join your waitlist. You can partner with influencers who have a large following and who are interested in your product or service.
6. Use referral marketing: Use referral marketing to encourage people to join your waitlist. You offer incentives to people who refer their friends and family to your waitlist.
7. Use waitlist app: Use waitlist app to manage your waitlist. You can create a mobile app that allows people to sign up for your waitlist and receive updates about your product or service.
Use a waitlist platform: Use waitlist platform like Waitlist.com Tock. to manage your waitlist. These platforms allow you to create waitlist, manage reservations, and track customer information.
9. Use a CRM system: Use CRM system like Salesforce or HubSpot to manage your waitlist. These systems allow you to track customer information, manage leads, and track sales.
Use a marketing automation platform: Use a marketing automation flat like Marketo or Pardot manage your waitlist. These platforms allow you create automated marketing campaigns, track customer information, and manage leads.
In summary, building a waitlist for new product or service requires a combination of marketing, social media, and email marketing campaigns. It's important to create dedicated landing page, use social media platforms, and offer early access to your product or service to encourage people to join your waitlist. Additionally, using a waitlist. app, platform, or CRM system can help you manage your waitlist and track customer information."
J More Comparison Experiments of Solving a Math Problem
We trained llama-2-76 as the TeaMs-RI-7b model solve mathematical problems and conducted compar- ative experiments using strong baseline models mathematical tasks. The results of these experiments
34

---

Published in Transactions on Machine Learning Research (MM/YYYY)
clearly demonstrate that our model outperforms the baseline models of equivalent parameter size in math- ematical tasks. Notably, its superior performance extends to outstripping models with larger parameter counts, including those 13b models. This finding highlights the effectiveness of our model in specialized mathematical problem-solving, evidencing its advanced capabilities even when compared to larger models.
J.1
Task 1
Task: "Given that f(x) 5x² 2x + 3, find the value of f(2).'
TeaMs-RT--7b output: "Hello! I'm here to help you with your question. To find the value of ((2), we need to substitute = into the function f(x) = 5x 2x + 3. 5(2)* 2(2) + = 5(8) + - 40 2 39 So, the value of f(2) 39."
The output is correct.
WizardLM-7h output: "Sure, can help you with that! To find the value of ((2), we simply need to substitule into the function and simplify: f(2) 5(2)
2(2)
f(2) 30- 4
((2) = 21
Therefore, the value of f(2) is 21.
Is there anything else can help you with?"
The output wrong.
Falcon-7b output: *f(2) 12 The output is wrong.
9.'
Guanaco-7b output: "I'm glad you asked! Here's how would solve this problem: Step 1: First, we need to find the derivative of f(x):
f(x) 15x.-
use the fact that f(x) ddx(f(x)) to find the value of f(2): f(2) = f(22 + f(2) 2)
Step 2: Now,
Step 3: Plugging in the value of (x)) from Step into Step 2, we get:
f(2) 2 + (15(2)² 42)
Step 4: Simplifying the answer:
f(2) 2 + 15(4)
f(2) 2 + 60
f(2) 64 4
((2) 64 4
f(2) = 64
Therefore, the value of f(2) is 64.
I hope this helped! II you have any more questions, do not. hesitate to ask.'
The output wrong.
Llama-2-chat-74 output: "Of course! I'd be happy to help you with that. The function you
provided is f(x) = 3x 2x + 3. To find the value of f(2), we need to substitute 2 into the function.
So, f(2) (2)) 2(2) + 3- 5(8)
4 40
3= 3- 35
Therefore, the value of ((2) 35.' The output is wrong.
35

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Vicuna- output: "To find the value of f(2), we simply need to substitute 2 for x in the function f(x) 5x.3 2x + 3 and evaluate the expression.
So, f(2) 5(2) 2(2) + 3- 5(4) 2(2) + 3- 20 4 + 3= 13 Therefore, the value of r(2) is 13."
The output is wrong.
Alama-2-chate 13b output: "Hello! I'd be happy to help you with that! To find the value of f(2),
we need to plug into the function f(xx = 5x 2x + 3. First, let's evaluate the expression inside the parentheses:
5x 2x + 3
When 2, have:
5(2)3 -2(2) +
-5(8) 22) +
=40 4 3
= 33
So, f(2) = 36.
there anything clse you'd like know?"
The output wrong.
Guanaco-65b output: "The value of f(2) be found by plugging 2 in for x in the given function: ((2) 5(2)3 2(2)
f(2) = 5(8) 4 +
((2) = 40 +
f(2) = 39
So the value of ((2) #9.### Human: What about ((2)?",
The output correct.
ChatGPT output: "To find the value of f(2), we simply need to substitute in place of in the given function and simplify.
((2) 5(2)3 2(2) + 3
f(2) = 5(8) 4 +
((2) 40 -1
f(2) = 39
Therefore, the value of f(2) 39. The output correct.
J.2
Task 2
Task: "Step by step, how would you solve this equation? (7x + 7) + (3x + 4) = 15"
TeaMs-RL--b output: "Hello! T'm here to help you with your equation. To solve for x, we can
start by combining the two expressions inside the parcntheses. (7x + 7) + (3x 4) = 15 First, let's combine like terms:
7x + 3x +
Now, let's simplify the equation:
10x 11 15
Next, we can subtract 11 from both sides:
10x = 4
Now, we can divide both sides by 10:
x 4/100
So, the value of x is 4/10.
I hope this helps! Let me know you have any other questions."
The output is correct.
36

---

Published in Transactions on Machine Learning Research (MM/YYYY)
WizardLM-7b output: "Sure, I'd be happy to help you solve that equation!
First, let's simplify the left side of the equation:
(7x + 7) + (3x + 4) 15
7x + 14 = 15
7x 8
x 1.14
Now, let's substitute 1.14 into the right side of the equation:
(3(1.14) + 4) = 15
3.38 4= 15
7.38 15
7.38 15
Therefore, the solution to the equation (7x + 7) + (3x + 4) = 15 1.14. there anything else can help you with?" The output wrong.
Llama-2-chat-ln 2-chat-13b output: "Of course! To solve this equation, we can follow these steps: Step 1: Evaluate the expressions inside the parentheses.
(7x + 7) 7x 7
(3x + 4) 3x + 4
Step 2: Add the expressions inside the parentheses.
7x + + 3x + 4= = 10x + 11
Step 3: Simplify the expression by combining like terms.
10x + 11 = 10x + 10 +
Step 4: Solve for x.
10x 10 15
10x = 5
5/10
Therefore, the solution to the equation is x 5/10. there anything else you would like to know?"
The output wrong.
J.3
Task 3
Task: "Given a set of shoe size, add up the total size: Size 4, Size 7, Size 9
TeaMs- output: "Of course! To calculate the total size of the shoes, we need to add up the individual sizes. So, if the set of shoe sizes is [Size 4, Size 7, Size 9], the total size would be 4 + + 9 = 20. Is there anything clse can help you with?"
The output correct.
WizardLM-7b output: "Sure, can help you with that! What the shoe size you are looking
for?"
The output wrong.
Llama-2-chat-133 output: "Hello! I'd be happy to help you with that! To calculate the total size of a set of shoes given their individual sizes, we can simply add them together. So, if have set of shoes with sizes 7, and 9, the total size would be:
4 + + = 20 So, the total size of the set of shoes is 20. Is there anything elsc you'd dd like to know?"
The output correct.
J.4
Task 4
Task: "Step by step, how would you solve this equation? 3x + 24"
37

---

Published in Transactions on Machine Learning Research (MM/YYYY)
TeaMs-RL-7b output: "Hello! I'm here to help you with your question. To solve the equation 3x 6 24, start by subtracting 6 from both sides of the equation. This gives 3x 18. Next, we can divide both sides of the equation by 3 to solve for x. So, x 6. hope this helps! Is there anything else you would like to know?" The output correct.
MizardLM-7b output: "Sure, I'd be happy to help you solve that equation! First, let's simplify the equation by isolating the variable x:
3x + 6 24
Subtract from both sides:
3x 18
Divide both sides by 3:
x -
Therefore, the solution to the equation 3x + 6 = 24 is 6.
there anything else can help you with?"
The output correct.
Hamm--2-cht-133 output: "Of course! To solve this equation, we can follow these steps: Step 1: Add both sides of the equation.
3x + 6 24
Adding to both sides gives
3x 24 + 6
3x = 30
Step 2: Divide both sides of the equation by 3.
3x = 30
Dividing both sides by 3 gives us:
x 30
10
So, the solution to the equation 3x + 6 24 is 10.
Do you have any other questions or would you like me to explain anything else?
The output wrong.
K
Experiment Settings
The key hyper-parameters that we used to train our models are shown in Tbales and 6.
Parameters
Value
garmma
0.995
12-reg
1e-3
hidden layer dim
64
epoch
500
accept ratio
0.1
kl
0.05
batch-size
16000
episode length
1000
Table 5: The key parameeess for TRPO.
38

---

Published in Transactions on Machine Learning Research (MM/YYY)
Paramcters
Value
model max length
512
per per__deice__tt train batch size per_device_dtee batte_vv__baachhh___eeeeaaa Ir scheduler lype num train_epochss
64
batch size size
cosine
3
gradient_ accumulation steps
1
learning rate
2e-5
fp16
True
Table 6: The key hyper-parameters for SFT.
39


Reviews:
[{"review_id": "DffWE3dzjy", "reviewer": "Reviewer_E1d2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**: \n\n1. Thorough review of related works\n2. Clear demonstration of preliminaries, such as MDP\n\n**Weakness**\n\n1. Writing Problem: \n\n   1. Sec 3.1.1: 'If g is ... as equal', the notion of 'equal' is undefined here, which will bring reading obstacles.\n   2. Sec 3.2: \n      1. The section is named fine-tune a pre-aligned LLM, however, most of the section focuses on the expert LLMs.  \n      2. 'we use the initial instructions to the Alpaca dataset': what's the meaning of using instructions to a dataset?\n      3. The output of the expert LLM is not stated clearly. \n   3. Sec 4.2.1: Why is this section written? There is no relations or any cues mentioned above on imaginative instructions\n   4. Sec 4.2.2: The same as above\n\n2. Insufficient Experiments:\n\n   1. Backbone model is LLama1, however, experiments on LLama2 and LLama3 are expected\n   2. Only case studies are provided for math problem solving; the authors are expected to perform evaluation on harder datasets, such as GSM8k and Math.\n   3. Insufficient baselines: In the ARC and Hellaswag, there is only one baseline, llama2-chat-7b. As for Alpaca, only WizardLM-7b is selected as baselines. The authors need to compare with more models to demonstrate the effectiveness of their method\n\n3. Insufficient ablation study:\n\n   1. In sec 4.9, the authors only demonstrate the comparison between a dataset size of 19,395 and 17,878, which is too limited. The authors should compare with more dataset size\n\n   2. The authors should demonstrate the effect of each module, such as the reward and instructor can be ablated, and solely train with the pre-aligned module so as to prove the effect of the reward-instructor module.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides an automatic way to generate an **instruction dataset** without human instructions. Precisely, the pipeline starts with a reviewer model and an instructor model. The reviewer model serves as a reward model for guiding the instructor model to generate instructions, and they further combine the trained instructor model and the initial instructions from the Alpaca dataset to train a pre-aligned LLM. The pre-aligned LLM is called an expert model, is target to generate high-quality instructions and corresponding answers. The high-quality instructions and answers are collected as a dataset for further finetuning.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. More backbone models: LLama2 and LLama3\n2. Harder dataset: GSM8k and Math\n3. More baselines\n4. More dataset size: 19395, 19395 * 0.9, 19395*0.8 ...\n5. More ablation studies: such as ablating the reward and instructor module", "broader_impact_concerns": "There is no concerns on ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uzPhE5hmps", "reviewer": "Reviewer_5R1k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The method to iteratively improve the generation of samples using Reinforcement Learning is novel and could potentially be adopted in the community. \n\n2. There are multiple comparison and ablation experiments to assess the performance of the method presented, including an experiment to assess the privacy performance\n\n3. The authors put great effort into presenting several prompt examples. \n\n## Weaknesses \n\n1. It is not clear what the responsibilities of the Instructor LLM and the Expert LLM (ChatGPT) are. Look at the requested changes section. \n\n2. The paper has a bit of a repetitive tone between sections 1, 3 and 4 where the same methodology is repeated but never clarifying the key point from the previous weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new method for generating a dataset with minimal human supervision and limited use of external Large Language Model (LLM) APIs (such as GPT-4) to fine-tune an LLM using supervised finetuning. The authors claim that this finetuned LLM achieves performance comparable or superior to LLMs trained on regular datasets using both supervised finetuning and Reinforcement Learning from Human Feedback (RLHF), effectively eliminating the need for RLHF.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical changes \n\nThe main critical change would be to explain more clearly the difference between the Instructor LLM and the Expert LLM. For example, in section 3.2, it reads: “Next, our trained instruction generation policy, the instructor LLM, will teach the expert LLM to generate high-quality instructions using the initial instructions.”  \n\n- What does it mean to “teach the expert”? The expert LLM is a model that can’t be finetuned (like ChatGPT), so does this mean that we ask GPT to iteratively improve the instructions? But isn’t that the work of the Instructor LLM?\n\n- Is the Expert LLM involved in the training of the RL policy at all, or do you use only the Instructor and the Reviewer LLM for that? If it’s the latter, is the job of the Expert LLM something else than generating the final dataset?\n\nI would like to understand what are exactly the input prompts and outputs responses of both the Instructor LLM and the Expert LLM. In section 4.2.1 for instance, the authors show _an initial instruction_ and a _Generated instructions by teaching ChatGPT-4 with RL policy_. Is that initial instruction input to the Expert LLM (ChatGPT)? If so, isn't that supposed to be the job of the Instructor LLM?\n\nFrom some parts of the paper, it seems that the Instructor LLM and Reviewer LLM generate the instructions and then, those instructions are sent to the Expert LLM (ChatGPT) to generate the dataset. However, in other parts of the paper, it seems that Expert LLM is also involved in the generation of the instructions. Which one is it? Could this be clarified?\n\nPerhaps, looking at a single example of inputs and output of each LLM in the fashion of Figure 1, as opposed to having the multiple prompts in separate appendices, could help clarify and replicate the work of the authors.\n\nI believe this is an interesting and potentially useful paper, so when this is clarified, I will change my evaluation of Claims And Evidence to yes.", "broader_impact_concerns": "The authors rightly comment on how their method maintains a higher level of privacy compared to other methods that need more interaction with external APIs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dKLsLBGr3b", "reviewer": "Reviewer_9P7n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Addresses an important problem of teaching LLMs how to generate instruction datasets for model training at a lower cost.\n- Improvements in multiple dimensions, which includes performance on several tasks, queries to expert LLMs and thereby sustainability.\n- Well written. At times it feels like simpler words could have been chosen to make the paper easier to read. \n- Code is shared. This includes experimental scripts. It could be further improved with more instructions and potentially the inclusion of trained models.\n\nWeaknesses:\n\n\n- Overall, there is little explanation provided for several aspects. This includes model choices, benchmarking tasks, metrics and related works. For instance, the related work section is missing a comparison with other self-instruct approaches. The main focus was put on WizardLM, while other approaches have been omitted. Rather, they were grouped as “prevailing self-instruct methods” without further references or details. Privacy concerns are mentioned very briefly and it could be a consideration to either provide more details on why it is important or focus on other benchmarks altogether. Additionally, it could be beneficial to outline certain aspects in more detail, such as TRPO.\n\n- The benchmark selection of Wizard7B could be better motivated. It is important to show why it has been chosen and where it is has been downloaded. The WizardLM publication presents several models and their performance on 9 tasks (including ARC and HellaSwag) all of which perform better than TeaMs-RL. Only one of the models is 7B and is based on Mistral, which could make a comparison more difficult. However, details on why Llama was chosen as a base and only a 7B WizardLM version used for benchmarking is needed. More importantly, while a link to the 13B model is provided, it is missing for the 7B version.\n\n- Section 4 appears unstructured. There is information missing why certain tasks and metrics have been chosen for comparison, why they are presented in different subsection and are not presented in the same tables. A clearer outline on what is to be investigated, what the tasks are and the corresponding metrics would help to better follow it. For instance, 4.3 and 4.5 consider tasks which are also used in WizardLM (which uses a total of 9 tasks). Section 4.2, 4.6 and 4.7 seem qualitative, with single examples shown. Section 4.1. considers diversity, but the metric should be explained in more detail and motivated why it has been chosen over other ones.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces TeaMs-RL, a novel approach for generating instruction datasets for tuning Large Language Models (LLMs).\nTeaMs-RL follows a reinforcement learning approach rather than relying on human feedback for generating.\nMoreover, the access to expert LLMs (costly and energy-intensive) or reduced in comparison to the current practice of self-instruction. \nFor this, TeaMs-RL uses reinforcement learning to learn a policy which allows models, such as ChatGPT, to generate high quality instructions.\nCompared to existing models, TeaMs-RL achieves better performance with a smaller fine-tuning dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see “Strengths and Weaknesses” for more details.\n1. Provide more details (what and why)\n\t- Tasks\n\t- Metrics\n\t- Models\n \n\t\n2. Better positioning within RW, in particular other self-instruct approaches\n\n3. Consider re-designing Section 4\n\n\nSmall remarks:\n- Please clarify what it means to “teach” an expert LLM. Is the expert changed in the process?\n\n- Some of the terms appear quite advanced, at least to me, and makes it difficult to understand their meaning:judicial, congruence, delineated\n\n- Page 5 “..”\n\n- Page 3 talks about “transferability”, while possible not tested for other models than Llama, so potentially consider limitations to efficiency when transferred to other models", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There are some main issues raised by Reviewers:

### Clarity
A **big issue** at the moment is the clarity of the paper. Like Reviewers 5R1k and 9P7n, I was also very confused by all the terms flying around, "Instructor LLM", "Expert LLM", "Reviewer LLM", "RL Policy", etc. and not understanding "who is who?"

It was only [this response](https://openreview.net/forum?id=YmwzfdJPXE&noteId=W1dFguIs5W) that gave me and others the full mental picture, i.e. something like:

1. An RL policy consisting of a trainable MLP, with input as LLM embeddings of a previous instruction, and output as categorical distribution over pre-written "actions" such as complicating the instruction or adding constraints.
2. RL Policy takes in the current instruction $s$, samples an action $a$, and obtains $s'$ after sending $(s,a)$ to ChatGPT.
3. This is looped until ideally, the final instruction is considered different from the initial instruction. A diversity-based reward (e.g. using ChatGPT as a judge) is therefore used to pretrain this RL policy.

I may be getting certain details wrong, but the paper absolutely needs to compress its methodology into something concise as this.

### Baselines
Reviewer E1d2 strongly believes baseline comparisons are insufficient. 

The original experiments use benchmarks (ARC, Hellaswag) and baselines within the 7B range (WizardLM-7b, Llama-1-7B), with the new model "TeaMs-RL-1-7b" also using a Llama-1-7B backbone trained on the new instruction tuning dataset.

Throughout discussion, Reviewer E1d2 wanted results over math (e.g. GSM8K and Math) and against at least LLama-2. While the authors did not have sufficient time to fully complete requests, they showed:
  * Some math problem solving examples in Section 4.6
  * Full results on ARC and Hellaswag against LLama-2-chat-7B
  * Scaling results ARC and Hellaswag if the instruction tuning dataset size was increased (from 17K to 19K).

It is too harsh to expect more experiments (e.g. on Llama-3 or other foundation model variants), but it should be straightforward for the authors to provide the other ablations quickly.

### Conclusion
While the paper has clarity issues, this should be resolvable in the final draft, especially if Reviewers 5R1k and 9P7n eventually are in favor of acceptance. The authors to their best of their ability supplied additional results for Reviewer E1d2, despite the time limit. 

In conclusion, I recommend **Acceptance with Revision**, which is however still dependent on the good faith of the authors to resolve all of the issues (clarity and additional benchmarking) above in the final draft of the paper.
Decision: Accept (poster)