CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment (2024)

Abstract: Language models trained on large-scale corpus often generate harmful responses that are harmful and contrary to human values. A prevalent approach for human alignment is reinforcement learning from human feedback (RLHF), utilizing algorithms such as proximal policy optimization (PPO). However, these methods are often characterized by complexity, instability, and substantial resource consumption. Considering that existing large language models (LLMs) like ChatGPT are already relatively well-aligned and cost-friendly, researchers propose to align the language model with human preferences from AI feedback. Nevertheless, the common practices, that unidirectionally distill the responses, are constrained by the inherent capability of LLMs. To address it, we introduce CycleAlign, a framework that distills alignment capabilities from the parameter-invisible LLMs (black-box) to the parameter-visible models (white-box) in an iterative manner. CycleAlign iteratively improves both the white-box and black-box models by integrating static and dynamic in-context learning and a belief alignment method. Empirical results illustrate that the model fine-tuned by CycleAlign remarkably exceeds existing methods, and achieves the state-of-the-art performance in alignment with human value.1

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated superior capabilities in processing various complicated tasks (Liu et al., 2023d; Wu et al., 2024; Zhu et al., 2023; Tu et al., 2023; Chen et al., 2024; Cheng et al., 2024), benefiting from the extensive training corpus and model parameters (Brown et al.,
* Equal contribution. † Corresponding author: Rui Yan (ruiyan@ruc.edu.cn). 1The code of this work is available at https://github.
com/hongjx175/CycleAlign.
2020; Bubeck et al., 2023; Chowdhery et al., 2022; Touvron et al., 2023a,b; Du et al., 2021; OpenAI, 2023). Nevertheless, models trained on the corpus collected from diverse web sources could not be effectively guided, and are prone to generate harmful, toxic and criminal contents (Bai et al., 2022b; Ouyang et al., 2022). Therefore, aligning language models with human preferences has emerged as a pivotal focus in the ongoing research.
Reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) has been employed to align language models with human preferences. Generally, the popular RL method PPO (Schulman et al., 2017) is utilized to optimize the foundation language model, with a reward model as the guidance. However, its complex architecture poses a challenge for hardware devices and it exhibits an unstable property during training. Recently, the emergence of ranking-based alignment methods has resolved the stability and hardware-consumption problems through shifting from the RL framework to supervised fine-tuning (SFT) (Song et al., 2023; Rafailov et al., 2023; Yuan et al., 2023). Nevertheless, the need for extensively annotated data renders them costly and labor-intensive.
14596
Considering existing LLMs like ChatGPT are well aligned, the reinforcement learning from AI feedback (RLAIF) methods are proposed to introduce automatic AI supervision (Bai et al., 2022b; Kim et al., 2023) to replace the manual annotation. However, common practices that distill instructionfollowing responses in a unidirectional manner are limited by the inherent capability of LLMs (Xu et al., 2024). Meanwhile, researchers (Burns et al., 2023) recently have demonstrated that the small model (GPT-2) could supervise the large model (GPT-4) and improve the ability of the latter. Consequently, we propose a novel framework CycleAlign to better align the parameter-visible whitebox model with the parameter-invisible black-box model by iterative interactions.
As shown in Figure 1, we introduce the incontext learning (ICL) (Min et al., 2022; Rubin et al., 2021; Ren et al., 2021) as the pivot to enhance black-box LLMs in this process. Given an instruction, we prompt the white-box model to generate multiple responses. Then, the black-box LLM ranks these responses with the help of the humancraft ranking prompt and static in-context demonstrations. The ranking signal will be utilized to optimize the white-box model and help it generate more harmless and helpful responses. Additionally, the generation probability of responses could be deemed as a ranking judgment from the perspective of the white-box model. As we know, within a certain range, LLMs tend to perform better as the number of high-quality in-context demonstrations increases. (Brown et al., 2020). Combining the ranking judgment from the white-box and black-box model, we could extract the consistent (or agreement) rankings as the pseudo label, which will be used as the dynamically appended demonstrations to improve the black-box LLM. Benefiting from the static and dynamic demonstrations, the black-box LLM could better rank the responses generated from the white-box model. After the iterative interaction, the alignment of the whitebox model will be improved with the help of an enhanced black-box LLM.
We conduct experiments on the human preference dataset HH-RLHF (Bai et al., 2022a) to investigate the effectiveness of CycleAlign regarding helpfulness and harmlessness. Compared with the previous methods, CycleAlign could improve the alignment ability and take state-of-the-art performance in generating harmless and helpful responses. In summary, our main contributions are
as follows:
• We present a new framework CycleAlign, which utilizes collaboration between the black-box LLMs and the white-box models, to align the latter with human preferences in an iterative manner.
• We enhance the ranking capability of the blackbox LLMs by employing static and dynamic incontext demonstrations under the interactive scenario.
• We validate the effectiveness of the CycleAlign framework in generating harmless and helpful responses by extensive experiments.

heading: 2 Related Work
text: RL-based Methods for Human Alignment. Reinforcement learning (RL) techniques have been widely applied to the human alignment of LLMs, which employ RL algorithms, such as Proximal Policy Optimization (PPO) to optimize the responses generated by LLMs (Yang et al., 2023). These approaches typically consist of three stages: 1) SFT: conducting SFT to enable the LLMs to follow instructions; 2) Reward modeling: training a reward model based on extensive paired responses of comparisons; 3) RL-based optimization: employing the RL algorithm to optimize the SFT model with well-trained reward model. At stage 2), RL from Human Feedback (RLHF) collects human-labeled pairs of responses (Bai et al., 2022a; Ouyang et al., 2022) while RL from AI Feedback (RLAIF) utilizes aligned LLMs (e.g., ChatGPT) to compare the pairs of responses (Bai et al., 2022b; Lee et al., 2023). Ouyang et al. (2022) proposed InstructGPT which employed RLHF for optimization. Bai et al. (2022a) employed RLHF to train a helpful and harmless assistant. Bai et al. (2022b) trained a harmless AI assistant through self-improvement based on a helpful AI assistant, without any human labels identifying harmful outputs. Lee et al. (2023) suggested that RLAIF could exhibit comparable performance to RLHF. Overall, these approaches all employed an RL algorithm (e.g., PPO) which is often complex, unstable, and resource-demanding.
Supervised Fine-tuning for Human Alignment Due to the complexity, high resource requirements, and instability of RL methods, people have begun to explore SFT methods to directly optimize the language models for human alignment. Rafailov et al.
(2023) bypassed the reward modeling stage and directly aligned the LMs with preference data, using a binary cross entropy objective for optimization. Similarly, Yuan et al. (2023) utilized the pair-wise responses of comparisons to enable the LMs to learn the preference knowledge. Song et al. (2023) extended the pair-wise comparison to accommodate preference rankings of any length. Liu et al. (2023a) combined opposite responses to fine-tune models, with hindsight feedback as the prompt prefix. Liu et al. (2023b) constructed a sandbox of LLMs as a simulated human society to collect interaction data with feedback for fine-tuning. These methods either relyed on extensive human labels or only unidirectionally distilled preference knowledge from aligned LLMs into unaligned LMs, ignoring the unaligned model can also give feedback to the aligned LMs to improve the aligning process. Our proposed CycleAlign utilizes the collaboration between aligned and unaligned models to improve human alignment.

heading: 3 Methodology
text: In this section, we describe CycleAlign which utilizes the collaboration between black-box LLMs and white-box models to align the latter with human preferences. As figure 2 shows, we introduce the static and dynamic demonstrations to break the inherent bottleneck of black-box LLMs and help the better alignment distillation to while-box models by iterative interactions.

heading: 3.1 Cyclical Collaborative Framework
text: To alleviate the instability of the RL algorithm and the costly human labels, we replace human feedback with AI feedback from the black-box LLMs and use ranking-based SFT to optimize the whitebox models. Existing methods only distill preferences unidirectionally from well-aligned black-box LLMs, ignoring the feedback of unaligned whitebox models. Consequently, we design a cyclical framework to facilitate the collaboration between them.
The framework is shown in Figure 2. For each interaction, we prompt the white-box model to generate multiple different responses for a given instruction. With the help of well-aligned black-box LLMs (e.g. ChatGPT), we could prompt them to rank these responses by their alignment degrees, utilizing the in-context learning method. Recently, researchers (Burns et al., 2023) have demonstrated that the small model (GPT-2) can supervise the large model (GPT-4) and improve the ability of the latter. Inspired by that, we also consider the belief (ranking results) of the white-box model about the alignment degrees, reflected by the generation probabilities of the responses. By incorporating the belief from while-box and black-box models, we could obtain the consistent rankings of the responses and utilize them as the pseudo in-context demonstrations to improve the ability of black-box LLMs.
With the iterative interactions between whitebox and black-box models, both of them will be enhanced.

heading: 3.2 ICL with Static-Dynamic Integration
text: LLMs have demonstrated the powerful capability of ICL (Brown et al., 2020; Xie et al., 2021; Min et al., 2022), which learns the patterns hidden within the demonstrations and generalizes to the specific tasks by few-shot prompting (Dong et al., 2023). In this work, we combine static and dynamic demonstrations to break the capability bottleneck of LLMs caused by utilizing solely static demonstrations.
We manually craft the prompt template and a static demonstration, which can be seen in Appendix A.1 and A.3. Then we continuously update the demonstrations during the training process, i.e., dynamic demonstrations. LLMs would make mistakes easily with only static demonstrations, which become the bottleneck of them. Besides, humancrafted static demonstrations could not adapt to the habit of white-box models, resulting the suboptimal distillation performance. Dynamically appending the demonstrations that fit the white-box models into the black-box is meaningful and helpful.
Specifically, for a given input, the white-box model could generate multiple responses and then we can obtain the rankings of them according to their generation probabilities, as the belief of the white-box model. Meanwhile, the black-box LLMs also can give rankings of these responses. The consistent rankings between black-box and whitebox models will represent not only the belief of the black-box LLM but also the confidence of the white-box model. We use the consistent rankings as the pseudo labels and append them to the demonstrations for black-box LLM to improve its ICL performance of judgment. During training, the white-box model is progressively aligned. The distribution of the generated responses will gradually converge toward human preferences. The generated responses will be more challenging to rank, so ranking these responses will exploit the capability of the black-box LLM. Meanwhile, the ranking results of the white-box model will be more and more accurate in terms of the degree of alignment, making us believe that they contain useful signals. With the training cycle progressing, both the white-box model and the black-box LLM will be enhanced benefiting from their collaboration, resulting in better alignment with human preferences of the white-box model.
How do we extract the agreement rankings? We assume that the rankings from the black-box
LLM are more accurate in general. In addition, since responses generated by the white-box model continuously improve with training, the rankings of responses that align more closely with human preferences have a higher reference value for the black-box LLM. So we extract the Longest Common Subsequence (LCS) of the two ranking results with the highest black-box rankings.
Our experiment results indicate that our ICL with Static-Dynamic Integration method enhances the ranking accuracy of black-box LLM and achieves better alignment performance of the white-box model.

heading: 3.3 Optimization
text: Recently, ranking-based SFT methods have been applied for alignment as an alternative to RL algorithms. Given a set of responses, human preferences can be expressed as rankings of the responses. Ranking-based SFT methods directly incorporate the ranking information in a contrastive manner into the fine-tuning stage (Rafailov et al., 2023; Yuan et al., 2023; Song et al., 2023; Wang et al., 2023b). We bring the two ranking-based optimization objectives from RRHF (Yuan et al., 2023) and PRO (Song et al., 2023) into our framework.
Specifically, given the white-box model π, an instruction x and n possible responses {yi}n1 with preference order y1 ≻ y2 ≻ · · · ≻ yn, the rankingbased SFT objective can be formulated as:
L = Lrank + λLsft, (1)
where
Lsft = − 1 |y1| ∑
t
logPπ(y 1 t |x, y1<t). (2)
The Lrank can be calculated by PRO or RRHF as follows:
LRRHF = − n∑
i=1
i−1∑
j=1
max{0, pi − pj}, (3)
LPRO = − n−1∑
j=1
exp(pj)∑n i=j exp(pi) , (4)
where pi denotes the probability of generating yi
conditioned on x, pi = 1|yi| ∑ t logPπ(y i t|x, yi<t).

heading: 4 Experiments

heading: 4.1 Settings

heading: 4.1.1 Datasets
text: We conduct experiments on HH-RLHF (Bai et al., 2022a)2, a human preference dataset about helpfulness and harmlessness. It contains about 170k dialogues, each of which has a context and a pair of responses along with an annotated preference label. This dataset contains four subsets, which are Harmlessbase, Helpfulbase, Helpfulonline and Helpfulrejection respectively. The statistics of them can be found in Appendix A.2. We clean the dataset referring code of OpenAssistant3. In our framework, the performance of the white-box model will become stable after being trained on about 1,000 examples of data, similar to the previous findings (Lee et al., 2023). Thus, we sample 1,000 contextualized questions across the four subsets of HH-RLHF and evaluate the model performance on each subset.

heading: 4.1.2 Evaluation
text: We use quantitative and qualitative approaches to evaluate the harmlessness and helpfulness of a language model. For quantitative evaluation, a welltrained reward model, whose training data involves HH-RLHF, is utilized to assess the responses generated by different models, similar to previous works (Song et al., 2023; Yuan et al., 2023). Because the reward model may not completely reflect human preferences, manual assessment should be involved. Meanwhile, GPT-4 judgment recently has become a popular and relatively reliable evaluation method (Wang et al., 2023a; Pezeshkpour and Hruschka, 2023; Zheng et al., 2023). Thus, human annotators and GPT-4 are employed for qualitative evaluation. They are required to compare the responses based on the criterion of harmlessness and helpfulness. To avoid the order bias of compared responses in GPT-4, we shuffle the orders of the compared responses and utilize chain-of-thought (CoT) (Wei et al., 2022). Finally, we calculate the average win rates of different models. More implementation details can be found in Appendix A.5.

heading: 4.1.3 Baselines
text: We compare our CycleAlign with zero-shot prompting, in-context learning (Brown et al., 2020) and
2https://github.com/anthropics/hh-rlhf 3https://github.com/LAION-AI/Open-Assistant
CoT (Wei et al., 2022), as well as imposing the recent alignment methods on these models.
Specifically, we involve unaligned models including LLaMA-7B (Touvron et al., 2023a), Alpaca-7B (Taori et al., 2023) and aligned models including ChatGLM-6B (Du et al., 2021) and ChatGPT. The considered existing aligning methods include RLHF method PPO (Schulman et al., 2017), SFT method RRHF (Yuan et al., 2023), and PRO (Song et al., 2023) which involves human and AI feedback. The detailed description can be found in Appendix A.4.
Due to that our CycleAlign is an optimizationflexible framework, we equip CycleAlign with RRHF and PRO, and note them as CycleAlignRRHF and CycleAlignPRO respectively.

heading: 4.2 Main results
text: The main results of our experiments can be found in Table 1. Upon the LLaMA-7B and Alpaca7B, we reproduce the state-of-the-art alignment method PRO. The results of PPO and RRHF are cited from Song et al. (2023). The effectiveness of our CycleAlign framework on alignment could be illustrated from the following perspectives.
Compared to zero-shot backbones like LLaMA and Alpaca, as well as these models with ICL and CoT, it is obvious that models significantly outperform them after alignment. It indicates that existing foundation models or models fine-tuned for instruction following are under-aligned with human values and can generate harmful and unhelpful responses, even with ICL and CoT. Besides, ChatGLM and ChatGPT, which have been aligned with human preference data, perform well in generating harmless and helpful responses. And overall they perform even better with ICL and CoT. Considering that ChatGPT is well-aligned and cost-friendly, we propose CycleAlign to better align models with it in a low-resource manner.
Compared to previous alignment methods, the model equipped with CycleAlign obtains a remarkable improvement in alignment. It is noteworthy that the previous alignment methods (PPO, RRHF, and PRO) fine-tune models on the entire training set of HH-RLHF augmented with responses generated by ChatGPT, provided by (Song et al., 2023). Specifically, CycleAlign increases 7.03 reward score on Harmlessbase and 5.31 reward scores in total for RRHF when the backbone is LLaMA. It also increases about 1.27 reward scores for PRO, indicating the effectiveness of iterative cycle aligning
with the help of black-box LLMs. Overall, the CycleAlignPRO based on Alpaca exhibits state-of-the-art performance in alignment compared with all the traditional alignment methods and has the approximate performance of ChatGPT. With CycleAlign, the model could generate more harmless and helpful responses to satisfy the demands of users.

heading: 4.3 GPT-4 and Human Evaluation
text: In recent developments, GPT-4 has demonstrated certain consistency with human judgment, leading to its extensive application in evaluations (Liu et al., 2023c; Mao et al., 2023). In our study, we employed both GPT-4 and human annotators to assess and compare the responses generated
by CycleAlignPRO and PRO, with Alpaca serving as the backbone. The evaluation results, presented in Table 2, present similar conclusions.
The sampled results across all datasets reveal a consensus among humans and GPT-4 that models fine-tuned by CycleAlignPRO demonstrate better alignment with human preferences. This statement, however, seems to stand in contrast with the assessments derived from the reward model, as illustrated in Table 1. According to the reward model evaluation, CycleAlignPRO falls short of matching the performance of PRO on the Harmlessbase subset. Nonetheless, both human and GPT-4 evaluations suppose that CycleAlignPRO generates much less harmful content compared to PRO. This inconsistency might be rooted in the limitations inherent to the current reward model. Besides, the models fine-tuned by CycleAlignPRO manifest markedly superior performance in the Helpfulrejection subset as GPT-4 evaluation, and in Helpfulbase according to human assessment.
These findings cohesively indicate that through iterative interactions with black-box LLMs, whitebox models are capable of achieving a more refined alignment with human preferences.

heading: 4.4 Ablation Study
text: In this section, we conduct ablation studies to verify the effectiveness of each part of our design.
Dynamic demonstration (D2) and ICL With the model continuously updated during the training process, the distribution of the generated responses is ever-shifting. So it is necessary to dynamically examine the accuracy of ranking results returned from the black-box LLM. We take the rankings from the reward model used for evaluation as ground truth and calculate the ranking accuracy of black-box LLM at the last iteration of every step. As shown in Figure 3, after removing D2, the ranking accuracy of black-box LLM declines. Especially after removing all of the ICL components, the performance of black-box LLM severely deteriorates. The bottleneck in the ranking performance of ChatGPT indirectly affects the alignment performance, which shows a similar trend in Table 3 regarding the ranking accuracy of ChatGPT.
Agreement ranking When we dynamically add only rankings from the black-box LLMs to demonstrations, (i.e., without white-box model rankings demonstrations, denoted by w/o WRD), the ranking accuracy of the black-box LLM and the alignment performance of the white-box model drops; and only utilizing rankings of the white-box model (w/o BRD) leads to declining ranking accuracy and even worse alignment performances. These results indicate that the signals from the white-box model help the black-box LLM to rank the evershifting responses and consequently contribute to better alignment.
The aforementioned experimental results illustrate that ICL and dynamic demonstrations consisting of agreement rankings used for bridging the cycle bring enhanced alignment performance for the unaligned white-box models. This results in the
generated responses being more in line with human preferences, i.e., harmless and helpful.
Cycle framework To validate the effectiveness of the iterative cycle framework, we align the model with vanilla ranking-based SFT, i.e., without cycle iterations (w/o Cycle). For a fair comparison, we sample 5,000 contextualized questions from HH-RLHF. We sample responses from the white-box model and request the black-box LLM for rankings as well as a better response only once for each question. In this case, the two settings (w/o Cycle and CycleAlign) involve an approximately equal number of parameter updating iterations. The results can be found in Table 3, which show that our CycleAlign achieves superior alignment performance in all aspects. The result demonstrates our iterative cycle framework can more adequately align the white-box models.

heading: 4.5 Evolution over Iterations
text: We conduct detailed analyses of each part of the cycle over iterations to demonstrate the effectiveness of our framework. We focus on understanding how the agreement, black-box performance, and whitebox performance evolve throughout the training process.
Agreement The training of our models involves 1,000 steps, where each step involves updating the white-box model for a maximum of five iterations. If the rankings from the white-box model and blackbox LLM are the same at some iteration, we stop the current step and move on to the next one. Our assessment of the level of agreement between the two models is based on the running mean iteration times per current step k during training. The lower the mean iteration times, the better the agreement.
As seen in Figure 4, the number of needed iterations decreases rapidly in the first 300 steps and then keeps stable. It indicates that the agreement
Table 3: Ablation study. The results are scores calculated by the reward model. The numbers in bold indicate the best performance. For a fair comparison, we set the number of demonstrations under w/o D2 and w/o Cycle as 2, close to other settings.
Settings Avg #Demo Harmlessbase Helpfulbase Helpfulonline Helpfulrejection Total
CycleAlignPRO 1.86 71.32 67.89 66.53 68.92 68.97 w/o D2 2 (pre-set) 71.77 65.37 64.99 66.34 67.36 w/o ICL 0 71.96 64.37 64.03 65.93 66.88 w/o WRD 1.81 69.81 66.08 65.75 67.23 67.41 w/o BRD 1.93 67.99 66.65 65.73 67.64 67.21 w/o Cycle 2 (pre-set) 70.30 66.57 65.51 67.57 67.76
between the rankings from the white-box and blackbox models improves over the CycleAlign training steps.
White-box performance We monitor the progress of the performance of the white-box model by assessing the reward scores on the validation set during training, seen in Figure 4. The performance shows rapid improvement within the first 50% of the training steps, then reaches a plateau, indicating that the alignment degree of the white-box model increases over the training process.
Black-box performance Because we update the dynamic demonstration set that is initiated with a static example by iterations within each step, the black-box LLM evolves within each step. We calculated the ranking accuracy of the black-box LLM at nth iteration for steps that iterate 3-4 times for analysis. As Table 4 shows, the ranking accuracy of the
black-box LLM is ever-improving over inner iterations. This suggests that the ranking performance of the black-box LLM improves by iterations with the dynamic demonstrations of agreement rankings.
Intuitively when the alignment degree of the white-box model is low in the first place, it may hurt the ranking performance of the black-box model. We conduct an in-depth analysis of that. We calculate the running mean ranking accuracy of the black-box model and the running mean difference of reward score (delta score) of the generated responses over steps. We assume that a higher delta score indicates a bigger difference between the responses to rank, making it easier for the black-box LLM to rank them. As seen in Figure 5, in the first
place, the delta score is high, and the ranking accuracy is high too. Then, the delta score drops, which means that the responses become more difficult to rank. The ranking accuracy first drops, which indicates that the demonstrations fail to provide sufficient help for ranking at first. But in the following steps, the ranking accuracy increases though the delta score continues to drop. This suggests that when the white-box model improves, the agreement rankings can better help with the black-box LLM. In conclusion, rankings from the white-box model may hurt initially, but they help across the whole training process overall.

heading: 4.6 Iterative Number Analysis
text: In this section, we investigate the influence of the threshold of interactive times in each step, i.e. the optimal setting about the maximum iterative number N between the black-box LLM and the whitebox model. As shown in Figure 6, the model performance displays a tendency to increase first and then decrease. We find that it does not need too many interactions because the performance will saturate when in-context demonstrations continuously increase. For this consideration, we set the maximum iterative number N as 5 to obtain the best performance.

heading: 4.7 Case Study
text: In Table 5, we compare responses from PRO and our CycleAlign to different contexts. 1) Both models answer informatively about "Santana’s music"; however, our CycleAlign model provides additional context, details, and engagement, proving better for user interaction. 2) Regarding queries on illegal activities, both models discourage such behavior, emphasizing law adherence and ethics. Our model, however, offers a more comprehensive response, providing alternative legal suggestions and demonstrating a commitment to promoting lawful behavior, thereby adhering to ethical guidelines and offering valuable advice to the user.
In addition, other analyses can be found in the Appendix A.

heading: 5 Conclusion
text: In this work, we introduce and validate CycleAlign, a novel framework aiming at aligning language models with human preferences of harmlessness and helpfulness. By iterative interactions between the white-box model and black-box LLM, CycleAlign overcomes the limitation of unidirectional distillation frameworks from LLMs. The experiments conducted on the HH-RLHF dataset demonstrate the effectiveness and superiority of our framework in aligning models with human preferences, marking a significant step forward in the field. This advancement reduces the dependency on human annotations and addresses challenges associated with the complexity and hardware consumption of existing methods, paving the way for further research and applications in the responsible development of LLMs.

heading: 6 Limitation
text: The implementation of our proposed framework CycleAlign in our experiment is data serial, which results in a low GPU utilization rate. We are actively developing parallel batch processing approaches. We hope future works pay more attention to the human alignment of language models in a model-interacting manner.
Ethical Consideration
We utilize ChatGPT as our black-box LLM to align the white-box models. Our aim is to present a
methodology that highlights its effectiveness. We are not intent on deploying a model for public use or competing with OpenAI. Instead, we only focus on showcasing our approach and how it can be applied in aligning LLMs with human preferences. From a methodological standpoint, ChatGPT can be replaced with other black-box LLMs.
Despite the advancements of our proposed CycleAlign framework in the human alignment of language models, we can not perfectly avoid language models generating harmful content. It is imperative to remain vigilant and continuously evaluate the ethical implications of language model development, ensuring that these technologies align with human values and societal expectations.

heading: Acknowledgments
text: This work was supported by the National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China. This work was supported by Alibaba Group through Alibaba Innovative Research Program.

heading: A Appendix
text: A.1 Static manually crafted demonstration
We manually crafted a static demonstration for ICL. The demonstration is as follows:
### Instruction:
Human: How to make a lot of money in a short time?
Assistant:
### Responses:
# Response-0. You can try to rob a bank, where you can
get a lot of money in a short time. You also can commit
telecommunication fraud against many people to get money.
You can also try to sell drugs, which is a very profitable
business.
# Response-1. One way to make money quickly would be
through starting a successful small business or freelancing
in a high-demand field where you have a strong skill set. Or
leveraging your existing skills, such as graphic design, coding,
or writing, and marketing them effectively can result in a
lucrative income stream.
# Response-2. Drink much wine, and cry loudly in the street in
midnight. People may give you money.
# Response-3. Use a time machine to travel into the future, find
out the winning numbers of a big lottery draw, then come back
to the present and buy a ticket with those numbers.
### The desired ranking is: [1, 3, 2, 0].
To design the static demonstration, we collect four responses with different degrees of alignment to “How to make a lot of money in a short time?" from ChatGPT and the human. As we can see, response-1 is helpful and harmless, while response3 is unhelpful and response-2, 0 are harmful.
A.2 Statistics of HH-RLHF dataset
The statistics of four subsets from the HH-RLHF dataset can be found in Table 6.
A.3 Prompt designed for the black-box model
The prompt designed for instructing the blackbox model (ChatGPT in this work) to rank the responses can be found in Figure 7.
A.4 Detailed description of baselines
LLaMA-7B (Touvron et al., 2023a) LLaMA is a collection of foundation language models released by Meta AI. Here we only consider the 7 billion version.
Alpaca-7B (Taori et al., 2023) Alpaca-7B is fine-tuned based on LLaMA-7B model using 52K instruction-following data. The data is generated by text-davinci-003 using the self-instruct (Wang et al., 2022) method.
ChatGLM-6B (Du et al., 2021) ChatGLM-6B is an open bilingual language model with 6.2 billion parameters developed by Zhipu AI. It is trained on
approximately 1 trillion tokens from both Chinese and English corpus and is further enhanced with supervised fine-tuning, feedback bootstrapping, and RLHF.
ChatGPT ChatGPT is a powerful large language model developed by OpenAI. It is fine-tuned from the GPT-3.5 series by introducing RLHF. Here we use gpt-3.5-turbo-0613 API version.
Besides, we compare with prevalent alignment methods like PPO, RRHF, and PRO, which are all aligned with the whole training dataset of HHRLHF augmented with responses from ChatGPT.
PPO (Schulman et al., 2017) Proximal Policy Optimization (PPO) is a popular algorithm in the field of reinforcement learning. It has been used to optimize the language model for aligning with human preferences.
RRHF (Yuan et al., 2023) Response Ranking for Human Feedback (RRHF) evaluates and ranks model-generated responses to ensure matching human preferences. It requires only 1 to 2 models during tuning and simplifying various aspects of the process.
PRO (Song et al., 2023) Preference Ranking Optimization (PRO) extends the Bradley-Terry comparison method to rank responses generated by LLMs according to human preferences, offering an alternative to complex and unstable reinforcement learning approaches like PPO.
A.5 Implementation details
The LLaMA-7B (Touvron et al., 2023a) and Alpaca-7B (Taori et al., 2023) are the backbones of white-box models in our experiment. We utilize ChatGPT, gpt-3.5-turbo-0613, through API provided by OpenAI as the black-box LLM. We apply the CycleAlign framework to optimize these two models with the help of DeepSpeed ZeRO-2 (Ren et al., 2021) and Huggingface.Accelerate (Gugger et al., 2022). The reward model used for quantitative evaluation is trained on preference datasets including HH-RLHF by OpenAssistant4. We set the weight factor λ to (l − 1)2, where l is the number of candidate responses (l = 3 in this work). We set batch size as 1, epoch as 1, learning rate as 5e− 5, and maximum sequence length as 512. The threshold of the interaction times N is set as 5. All of the experiments are done on a single A100 40G GPU, and each CycleAlign fine-tuning costs about 7 hours.
A.6 Human annotators
4https://huggingface.co/OpenAssistant/oasst-rm-2pythia-6.9b-epoch-1
A.7 Evaluation on the unseen dataset We evaluate the fine-tuned white-box models on an unseen dataset called BeaverTails (Ji et al., 2024), a human-preference dataset for the alignment of LLMs. Specifically, we randomly sample 2,000 prompts from BeaverTails as the test set and use white-box models to generate responses. Then we evaluate the responses to the prompts with the value model beaver-7b-v1.0-reward trained on BeaverTails. The results can be found in Table 8. As the results show, our approach outperforms the baseline methods under all setups. These results justify the generalization of models trained under our framework.

heading: Future Work
text: We hope future works pay more attention to the human alignment of language models in a model-interacting manner. Ethical Consideration
We utilize ChatGPT as our black-box LLM to align the white-box models. Our aim is to present a
methodology that highlights its effectiveness. We are not intent on deploying a model for public use or competing with OpenAI. Instead, we only focus on showcasing our approach and how it can be applied in aligning LLMs with human preferences. From a methodological standpoint, ChatGPT can be replaced with other black-box LLMs. Despite the advancements of our proposed CycleAlign framework in the human alignment of language models, we can not perfectly avoid language models generating harmful content. It is imperative to remain vigilant and continuously evaluate the ethical implications of language model development, ensuring that these technologies align with human values and societal expectations