TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback (2024)

Abstract: Reinforcement Learning from Human Feedback (RLHF) leverages human preference data to train language models to align more closely with human essence. These human preference data, however, are labeled at the sequence level, creating a mismatch between sequence-level preference labels and tokens, which are autoregressively generated from the language model. Although several recent approaches have tried to provide token-level (i.e., dense) rewards for each individual token, these typically rely on predefined discrete reward values (e.g., positive: +1, negative: -1, neutral: 0), failing to account for varying degrees of preference inherent to each token. To address this limitation, we introduce TLCR (Token-Level Continuous Reward) for RLHF, which incorporates a discriminator trained to distinguish positive and negative tokens, and the confidence of the discriminator is used to assign continuous rewards to each token considering the context. Extensive experiments show that TLCR leads to consistent performance improvements over previous sequence-level or token-level discrete rewards on open-ended generation benchmarks.

heading: 1 Introduction
text: With the emergent properties of Large Language Models (LLMs) in relation to the scale of both the model and data, there has been a notable performance improvement, particularly in various language generation tasks including summarization (Chang et al., 2023), translation (Zhang et al., 2023), and question-answering (Yoon et al., 2022, 2023). Central to these rapid performance gains in language generation is the Reinforcement Learning from Human Feedback (RLHF), which employs reinforcement learning techniques to fine-tune LLMs in alignment with human preferences (Christiano et al., 2017; Bai et al., 2022; Ouyang et al., 2022).
*Equal contribution †Co-corresponding authors
However, a fundamental limitation of the current RLHF approach is its dependency on sequencewise (i.e., sparse and holistic) rewards, which stems from the inherently holistic nature of humanlabeled preference data (Figure 1-(a)). While this approach effectively directs overall model behavior, it tends to miss the detailed significance and context of specific words and phrases within sequences. This oversight suggests significant room for enhancement in existing frameworks by introducing token-wise (i.e., dense) feedback, which
14969
could more accurately capture the subtleties of language generation.
There have been several attempts to incorporate such granular feedback into the training process. Recent methods (Guo et al., 2024; Cao et al., 2024) employ stronger LLMs to identify regions within a model’s output that require refinement, thereby enabling a denser reward signal. However, these methods still have a limitation in reflecting finegrained preferences due to the use of a few predefined discrete reward values (Figure 1-(b)).
To this end, we propose TLCR (Token-Level Continuous Reward), a novel reward model to provide token-level continuous rewards for finegrained RLHF (Figure 1-(c)). In particular, we obtain continuous rewards efficiently and effectively from the predictive confidence of categorical preferences (positive, neutral, or negative) for each token, which are predicted by the token-level preference discriminator. In order to train such a discriminator, our method compensates for the lack of token-level preference labels in an existing sequence-level human preference dataset by utilizing an external mature LLM, such as GPT-4 (OpenAI, 2023). This process involves instructing the external LLM to revise a given output text, subsequently followed by analyzing the (1) added, (2) deleted, and (3) substituted tokens between the original and revised outputs and labeling token-level preferences.
After training the discriminator with the created token-level preference labels, we use it during reinforcement learning to provide fine-grained dense rewards to a generated text from the language model. During this process, we find that instead of using fixed reward values for each token based on its preference (e.g., using +1 for positive, -1 for negative, and 0 for neutral), assigning values based on the prediction confidence of the discriminator allows superior guidance. Such an approach allows for a continuous range of rewards that reflect the varying degrees of preference associated with each token based on the context.
In detail, our contributions can be summarized as follows:
• To address the challenges of sequence-level rewards and the inability of token-level discrete rewards to reflect varying degrees of token preference, we introduce Token-Level Continuous Reward (TLCR), that provides a continuous spectrum of rewards for each token during Reinforcement Learning from Human Feedback (RLHF).
• For this, TLCR leverages confidence values from a token-level preference discriminator, and to train the discriminator, we use an external mature language model to refine responses, then assign token-wise preference labels by calculating the minimum edit distance between the original and revised responses.
• Through extensive experiments and analysis, we show the effectiveness of TLCR in enhancing fine-grained RLHF compared to the traditional sequence-level and token-level discrete reward mechanisms. To the best of our knowledge, our work is the first to achieve substantial performance gain on open-ended generation benchmarks using token-level rewards.

heading: 2 Related Works

heading: 2.1 Reinforcement Learning from Human Feedback.
text: Foundation models have achieved notable performance improvement due to increased model size and data volume. With these advancements, ensuring safe and reliable real-world applications has become a priority (Yoon et al., 2024). In this aspect, Reinforcement Learning from Human Feedback (RLHF) aims to integrate human judgments and preferences into the reinforcement learning framework to train models, such as language models, to align more closely with human values and expectations (Christiano et al., 2017; Bai et al., 2022; Ouyang et al., 2022). In this approach, a reward model is initially trained based on human feedback, usually pairwise preference ranking, using the Bradley-Terry preference model (Bradley and Terry, 1952). This reward model is then utilized to guide the model via reinforcement learning frameworks, usually Proximal Policy Optimization (PPO) (Schulman et al., 2017).
However, despite its effectiveness, RLHF has complexities involving multiple stages and models, demanding substantial resources. These challenges are pronounced in large-scale implementations, with recent research highlighting potential instability issues (Zheng et al., 2023b; Gao et al., 2023). To tackle such limitations, Hydra-PPO (Santacroce et al., 2023) utilizes switchable architecture to integrate multiple models involved in PPO training into a single LLM, while ReMax (Li et al., 2023b) adapts REINFORCE (Williams, 1992), simplifying the framework by eliminating the need for extra models used in PPO.
Recent studies have also been exploring alternatives to conventional reinforcement learning (RL) frameworks for efficient training. Direct Preference Optimization (DPO) (Rafailov et al., 2023) simplifies LLM training by using log-likelihood as an implicit reward, avoiding the need for separate reward modeling.

heading: 2.2 Fine-grained Feedback
text: The majority of LLM fine-tuning with human feedback, including RLHF (Ouyang et al., 2022) and DPO (Rafailov et al., 2023), typically utilize trajectory-wise rewards learned from pairwise preference learning. This approach involves assessing and providing feedback on LLM responses as a whole rather than each individual word or phrase within the sequence.
While trajectory-wise rewards have been effective in guiding the overall behavior of LLMs, they may present limitations in terms of the granularity and specificity of the feedback. This might lead to less precise tuning of the model, since specific aspects of responses that could be improved or are particularly well-aligned with desired outcomes may not be individually identified and adjusted. This highlights the potential need for more fine-grained feedback mechanisms in RLHF for LLMs. For instance, Wu et al. (2023) shows that fine-grained rewards enable more detailed and specific guidance for each phrase of the LLM response, potentially leading to more nuanced and effective learning and alignment with human preferences and intentions.
One of the most significant challenges in utilizing fine-grained rewards for aligning LLMs is the lack of datasets with fine-grained feedback, since most existing datasets provide trajectory-wise or holistic labels. This gap in data availability necessitates innovative approaches to generate or infer finegrained signals from the available trajectory-wise human feedback. Yang et al. (2023) defines the trajectory-wise reward as an aggregation of individual token-wise rewards, which are learned through standard preference-based reward model training. However, selecting an optimal aggregation function for token-wise rewards is not straightforward and depends largely on heuristic approaches. Also, the lack of additional fine-grained information or constraints will result in a sub-optimal reward function.
To address this issue, recent techniques leverage external advanced LLMs, such as GPT-4 (Ope-
nAI, 2023), to provide more detailed and accurate fine-grained insights for learning dense rewards. Approaches such as FIGA (Guo et al., 2024) and RLMEC (Chen et al., 2024) utilize off-the-shelf LLMs to revise the generated response with minimal edits, pinpointing areas within responses that need fine-tuning and providing specific guidance at the token level. Meanwhile, DRLC (Cao et al., 2024) asks external LLMs to identify the positive and negative segments within an original response, utilizing them as intrinsic rewards to guide model adjustments. On the other hand, ABC (Chan et al., 2024) extracts fine-grained credits from the trained reward model itself, particularly using attention maps for token-specific reward reallocation.
Despite the advancements these methods have limitations. Some can only offer provide predefined values for rewards (such as FIGA, and DRLC) and cannot clearly differentiate between positive and negative impacts (such as ABC). Others suffer from an inadequate reward model, necessitating a heavy dependency on stronger regularization (such as RLMEC). In contrast, our approach offers continuous-scale rewards that encompass both positive and negative feedback without the need for excessive regularization, leading to a more refined and effective learning.

heading: 3 Preliminary
text: In this work, we focus on aligning the language model to human preference in terms of helpfulness and harmfulness of the generated texts through Reinforcement Learning from Human Feedback (RLHF).
Language Generation as an MDP We consider the language generation procedure as a Markov Decision Process (MDP) as detailed in Ramamurthy et al. (2022), which is defined by the tuple (S,A,P,R, γ, Tmax) using finite predefined vocabulary tokens V . An episode in the MDP starts with a human instruction or a history of human-assistant conversation pairs followed by a human instruction x = (x0, ..., xl). This is used as our initial state s0 = (x0, ..., xl), where s0 ∈ S and S represents every possible states with xi ∈ V . At time step t, an action in the environment at ∈ A is generated by the next token prediction given current state st by a policy Language Model (LM) πθ from V . For t ≥ 1, the transition function P : S × A 7→ ∆S appends at to the end of the state st = (x0, ...xl, a0, ..., at−1). This process
continues until the current t exceeds Tmax or the end of sentence (<eos>) token is generated, yielding the generated sequence y = (a0, ..., aT ). A reward function R : S × A 7→ R assigns a value to each transition.
Sequence-level Reward in RLHF Typically, in RLHF for language models, a sequence-level (i.e., sparse) reward model is used, which maps the concatenation of the input x and the generated output y to a single scalar value. This scalar value is solely assigned to the last generated token or the end of sentence (<eos>) token aT . Formally, this process of sparse reward modeling used in previous RLHF frameworks can be defined as:
rt = { Rϕ(x, y), if t = T 0, otherwise.
(1)
Token-Level Discrete Reward in RLHF In contrast to the sparse reward approach, token-level discrete rewards in RLHF assign a predefined discrete value at each time step t ∈ [0, T ], reflecting the preference for each token, formalized as:
rt =    +1, if at is positive −1, if at is negative 0, if at is neutral.
(2)
4 TLCR: Token-Level Continuous Reward
In this section, we introduce an RLHF framework that utilizes a discriminator-based token-level re-
ward model that allows continuous-scale dense reward signals for RLHF. Section 4.1 details the procedure for obtaining token-level preference labels. In Section 4.2, we elaborate on the training procedure of the token-level preference discriminator. Subsequently, Section 4.3 outlines how the trained discriminator is utilized for continuous dense reward signals during the RL phase.

heading: 4.1 Token-Level Preference Labeling
text: Human preference data (Bai et al., 2022; Cui et al., 2023) is generally comprised of a prompt x = (x0, ..., xl), accompanied by its chosen response yc = (a c 0, ...a c Tc ) and the rejected response yr = (ar0, ..., a r Tr ). The categorization of responses as being labeled either "chosen" or "rejected" reflects the holistic evaluation of preferences.
As illustrated in Figure 2-(a), we utilize an external mature Language Model LMext as a reviser to assign token-wise preference labels for the response. The process begins by presenting the model with both the prompt and the rejected response to refine the rejected response to align more closely with human preferences. Considering the difficulty in explicitly defining "good human preference," we reduce the ambiguity during the process by also providing the chosen response yc to LMext. This allows the revising model to compare the rejected and chosen responses, understand the reasons behind the chosen response’s preference, and adjust the rejected response to better meet human preference criteria. Throughout this refine-
ment process, we instruct the model to make minimal modifications to the rejected response. This minimal modification ensures the core message remains intact while optimizing preference alignment, making it easy to identify which particular tokens are responsible for the output preference, a more straightforward approach than comparing yc and yr directly. Formally, given an instruction I (the full details of the instruction used are described in Appendix A), prompt x, rejected response yr, and chosen response yc, the LMext produces the modified response given as
ym = (a m 0 , ..., a m Tm) = LMext(I, x, yr, yc). (3)
Motivated by previous works (Liu et al., 2022; Guo et al., 2024), once we obtain the modified response ym, we use the Levenshtein Distance (LD) (Levenshtein, 1965) to determine the preference of each token. The LD calculates the minimum number of edits—(1) additions, (2) deletions, or (3) substitutions—required to transform one sentence into another through dynamic programming. By identifying the edited tokens regarding LD, we can determine which tokens contribute to a response being preferred or rejected.
When calculating the LD of going from yr to ym, we assign a token-wise preference labels pr = (pr0, ...p r Tr ) and pm = (pm0 , ..., p m Tm
) for yr and ym, respectively. Specifically,
prt = { negative, if art is deleted or substituted neutral, otherwise, (4)
pmt = { positive, if amt is added or substituted neutral, otherwise. (5)

heading: 4.2 Discriminator Training
text: Following the token-level preference labeling process, we train a discriminator Dϕ that differentiates between these positive and negative tokens (Figure 2-(b)). The output space of Dϕ is binary, where the prediction of 1 indicates positive and 0 indicates negative preference of the token.
To train the discriminator, we employ a binary cross-entropy loss function. Formally, for a given token at with its corresponding preference label pt, the loss function L for Dϕ is defined as follows:
L = −pt log(Dϕ(at|x, a0:t−1)) − (1− pt) log(1−Dϕ(at|x, a0:t−1)), (6)
where Dϕ(at|x, a1:t−1) represents the discriminator’s predicted probability that the t-th token at is
positive given the previous contexts, and pt = 1 for positive labels, pt = 0 for negative labels.
For tokens labeled as neutral, we introduce a soft labeling strategy to encourage the discriminator to remain unbiased towards either class. Specifically, neutral tokens are assigned a uniform probability distribution (i.e., pt = 0.5 for neutral labels), implying an equal likelihood of being positive or negative.

heading: 4.3 Discriminator-Guided Token-Level Continuous Reward
text: As illustrated in Figure 3, we utilize the trained discriminator to obtain token-level signals during the reinforcement learning phase. In detail, our discriminator-guided token-level reward mechanism employs a normalization formula,
rt = 2 ·Dϕ(at|x, a0:t−1)− 1 ∀t ∈ [0, T ], (7) to translate the discriminator’s confidence in token preference classification into a scale ranging from -1 to 1. A maximum reward of 1 is granted to tokens identified with high confidence as aligning positively, signaling their strong adherence to encouraging the model to favor such tokens in future responses. Conversely, tokens considered negative, marked by a confidence score of 0, receive the minimum reward of -1, indicating a deviation from the desired preferences and advising the model to avoid such tokens. Tokens that the discriminator views with neutral confidence scored at 0.5 are assigned a reward of 0, reflecting their ambiguous
contribution to preference alignment and suggesting no immediate need for the model to alter its approach for such tokens.
Given our token-wise reward approach, we adapt our reinforcement learning phase to align with the original PPO (Schulman et al., 2017) implementation, diverging from the conventional RLHF framework with a sparse reward setting (contextual bandit) (Ouyang et al., 2022).

heading: 5 Experimental Settings
text: This section presents the datasets, models, compared baselines, training details, and the evaluation metrics used in the paper.
Dataset Throughout the experiments, we use the full-hh-rlhf dataset (Bai et al., 2022) which is centered on improving the helpfulness and harmlessness of the language model generation. This dataset includes 112k training instances and 12.5k instances for evaluation. Every instance in the dataset features a prompt, along with a chosen response considered preferable and a rejected response, offering a clear basis for performing model alignment for better helpfulness and harmlessness. Following the setting in (Li et al., 2023b), we randomly divide the dataset into three parts: 20% for supervised fine-tuning (SFT), 40% for reward model learning, and 40% for reinforcement learning for reward maximization.
Model During the Supervised Fine-Tuning (SFT) phase we use Llama-2-7B (Touvron et al., 2023) as our base model. Similarly, for the reward model training, we initialize our discriminator Dϕ with Llama-2-7B. During the RL phase, we initialize our policy model with the trained SFT model. For the external language model LMext, we use gpt-40125-preview1.
Training Details We use DeepSpeed-Chat (Yao et al., 2023) framework for performing RLHF, where PPO is the default algorithm. Within this framework, our proposed TLCR is integrated to provide token-level rewards to enhance the PPO algorithm. Experimental details are presented in Appendix B.
All experiments were conducted 3 times with different random seeds, and the average value is reported. The experiments were conducted using 8 x NVIDIA A100 80GB PCIe.
1https://platform.openai.com/docs/models/gpt-4-and-gpt4-turbo
Baselines For comparison, we evaluate against a variety of baselines: SFT and DPO (Rafailov et al., 2023), where no preference data is utilized; sequence-level reward PPO (PPOseq) (Schulman et al., 2017) and ReMax (Li et al., 2023b), both employing sparse rewards; as well as FIGA (Guo et al., 2024), which introduce fixed token-level rewards.
Moreover, to verify that the performance improvements are not solely due to the use of GPT4 for creating synthetic data, we conducted an additional experiment, labeled PPOsynthetic. This experiment used pairs of rejected samples and their synthetically modified responses generated by GPT-4, applied to traditional sequence-level PPO (Schulman et al., 2017). Additionally, we include TLCRfixed as a variant of our approach using fixed token-level rewards. Specifically, TLCRfixed assigns a reward of +1 to the positive, -1 to the negative, and 0 to the neutral token instead of normalized confidence values. A token is considered neutral if the max confidence is below 0.6.
Evaluation Automatic evaluation of instructionfollowing language models are becoming a primary metric for preference evaluation with leaderboards2 for ranking. In light of this trend, , we report MTBench (Zheng et al., 2023a) and AlpacaEval (alpaca_eval_gpt4) (Li et al., 2023a) for our evaluations. Also, we perform human evaluation on 100 random samples from the full-hh-rlhf test set.

heading: 6 Results

heading: 6.1 Results on MT-Bench
text: Table 1 shows the comparison of the MT-Bench (Zheng et al., 2023a) evaluation results. MT-Bench is a multi-turn benchmark that measures the ability of LLMs to engage in coherent, informative, and engaging conversations. The process of eval-
2https://huggingface.co/spaces/lmsys/chatbot-arenaleaderboard
uation involves prompting GPT-4 to act as judges and asses the quality of the models’ responses on 80 questions on a scale of 1 to 10. The result shows that among the baselines, TLCR achieves the highest preference scores with the overall score being 5.04. Notably, compared to the sequence-level (i.e., sparse) rewarded PPO (PPOseq), our TLCR, which is a token-level (i.e., dense) rewarded PPO, achieves 0.36 increase in performance.
Compared to the results from SFT, PPOsynthetic shows better performance, yet it underperforms the PPOseq trained on the original data. Furthermore, a significant performance gap was observed between PPOsynthetic and TLCR, which was trained on the same synthetic data. Moreover, our TLCR outperforms TLCRfixed by 0.15. These results indicate that the modified synthetic dataset from GPT-4 alone cannot give a satisfactory signal during the alignment process without our targeted continuous token-level reward.

heading: 6.2 Results on AlpacaEval
text: Figure 4 shows the evaluation results for 805 test questions from the AlpacaEval dataset (Li et al., 2023a). This evaluation method entails presenting GPT-4 with two responses generated by different models and determining the binary win rate based on GPT-4’s preference. We achieve the higher win rate among all baselines; TLCR achieves win rates of 84.89%. 80.12%, 77.19%, 66.65%, 77.31%, and 60.76% over SFT, DPO, PPOseq, ReMax, FIGA, and TLCRfixed respectively.

heading: 6.3 Human Evaluation
text: We present the human evaluation result in Figure 5. Given the generations from SFT, PPOseq,
TLCRfixed, and TLCR, five human annotators3 are asked to choose the most preferred answer. We report the ratio of samples selected as the most preferred. The results show that TLCR achieved the highest preference rate, receiving an average of 45.23% of the total votes.

heading: 7 Ablation Studies

heading: 7.1 Reward Comparison
text: In Figure 6, we compare reward scores across different methods: SFT, PPO, TLCRfixed, and our proposed TLCR. Across all evaluated reward metrics, SFT consistently records the lowest scores due to its lack of training for maximizing any specific reward score.
Figure 6-(a) shows the reward score obtained using sequence-level rewards, which were the direct maximization target for PPOseq during reinforcement learning. Notably, TLCR (ours), despite not being explicitly maximized with this reward, shows a reward score closely competitive to that of PPOseq, obtaining a score of 4.13 compared to 4.28 from PPOseq. On the other hand, TLCRfixed lags behind in achieving comparable sequence-level reward scores, with only obtaining a score of 3.04.
3Human annotators all have TOEFL iBT score above 100 and possess a bachelor’s degree or higher.
Moving to Figure 6-(b), we look into the evaluation based on token-level discrete rewards, a value for which TLCRfixed explicitly maximized. Here, the limitations of sequence-level feedback become apparent, as evidenced by PPOseq’s underperformance (achieving a score of only 25.7) in enhancing token-level generation quality. Notably, TLCR also exceeds this aspect, effectively capturing token-level preference.
Lastly, Figure 6-(c) shows the outcomes evaluated on token-level continuous rewards, the core metric our TLCR model was trained to maximize. Consistent with previous trends, PPOseq shows insufficient results (with a score of 17.0), further emphasizing the shortcomings of sequence-level feedback in refining token-level outputs. TLCR achieves the highest score in this category as well.
Overall, these findings reinforce the inherent limitations of traditional sequence-level and tokenlevel discrete reward feedback mechanisms and demonstrate the advantages of our TLCR approach in achieving token-level improvements.

heading: 7.2 Alignment of Discriminator with Human Preference
text: Since the discriminator assigns the token level continuous reward during PPO, it is important to analyze the discriminator’s alignment with human preferences. Directly obtaining token-level human labels is challenging due to the fine granularity required. Therefore, we examine the test set of the token-level preference obtained by GPT-4 using our token-level preference labeling framework, and the results show a token-level preference prediction accuracy of 81.4% from our trained discriminator. This high accuracy indicates that the discriminator effectively mirrors token-level preferences, confirming the effectiveness of our approach in aligning with human preferences at the token level.

heading: 7.3 The Effect of Positive and Negative Token-Level Rewards
text: In this section, we explore TLCR’s performance under two conditions: excluding negative rewards (where we consider only rt ≥ 0 from Equation 7, denoted as TLCRw/o negative) and excluding positive rewards (where we focus on instances where rt ≤ 0 in Equation 7, denoted as TLCRw/o positive). Similar to Zheng et al. (2023b), we monitor the PPO training process by using action space modeling metrics, such as perplexity, response length, and KL divergence between the policy model and the SFT model to show informative details of the training stability of each setting.
In Figure 7, under conditions excluding negative rewards, the model is observed to increase the generation length to increase the number of positive tokens, thereby inflating its reward. Conversely, sentence length decreases to minimize negative token generation when positive rewards are not used. It can be viewed as ’reward hacking’, resulting in highly elevated KL divergence and perplexity. This result shows the importance of considering both positive and negative rewards in training to prevent reward hacking and ensure accurate preference training.

heading: 7.4 Qualitative Results
text: Figure 8 illustrates the qualitative analysis of tokenlevel preference predictions made by our trained Discriminator Dϕ. For the response A to Question 1, incorrect details like ’08’ from ’2008’ are identified as negative due to the misinformation. Con-
versely, the subsequent tokens are predicted as positive due to their accurate information. Similarly, in the case of response B, ’Elon Musk’ receives a negative prediction, attributed to the inaccurate information.
Furthermore, in the response B to Question 2, the term ’samoosa’ is marked as unfavorable, likely due to an apparent spelling error, contrasting with the favorable assessment of ’samosa’ in response A, which is spelled correctly.
For Question 3, the response B exhibits hallucinated outputs, characterized by repetitive, nonterminating sentences. While the initial sentence in both response A and response B is awarded a favorable preference, the following hallucinated segments in response B are marked with a negative preference.
These examples highlight the value and usefulness of token-level preference in providing detailed feedback on sentence quality. As illustrated, a generated sequence can include a mixture of accurate and inaccurate information or even correct and helpful responses impaired by minor spelling errors in specific words. This fine-grained approach to feedback is crucial for distinguishing the subtle quality of language model outputs.

heading: 8 Conclusions
text: In this study, we introduced the Token-Level Continuous Reward (TLCR), a novel reward model aimed at providing detailed, token-based continuous rewards for Reinforcement Learning from Human Feedback (RLHF). By utilizing the prediction confidences from a discriminator to assign rewards at the token level, our approach offers a sophisticated reward system that adjusts to the varying degrees of relevance each token holds within its context. The superiority of TLCR over traditional sequence-level and token-level discrete reward mechanisms has been established through detailed experiments and analyses.

heading: 9 Limitations
text: Current work uses a 7B parameter model, trained with a single dataset partitioned for three stages. Also, the current reward model training utilizes the offline static dataset, which might cause a distribution mismatch during policy model training. Moreover, the token-wise preference discriminator exhibits a bias due to the ambiguity of good human preference, implying room for improvement.
Future works can include model-data scaling, iterative updating of the reward model and the policy model, and combining multi-objective preference (e.g., safety reward and helpfulness reward) learning for more fine-grained guidance.

heading: 10 Broader Impact
text: This paper presents a method to improve LLMs via RLHF, targeting fine-grained alignment with human preferences and values. We hope our work will help reduce potential risks caused by LLM generations.

heading: 11 Acknowledgements
text: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00951, Development of Uncertainty-Aware Agents Learning by Asking Questions), Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT) [RS-2021II212068, Artificial Intelligence Innovation Hub (Seoul National University)] and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)(No. RS-2019-II190079, Artificial Intelligence Graduate School Program, Korea University).

heading: A Prompt Used for Instructing the External Language Model

heading: B Experimental Details
text: We use the Adam optimizer, setting β1 = 0.9 and β2 = 0.95. We set the maximum sequence length, including prompts and responses, to 512. Further details for each stages of the PPO process using the Deepspeed-Chat (Yao et al., 2023) framework is detailed as follows:
• Supervised Fine-Tuning (SFT): We use a batch size of 16 per device, a learning rate of 10−5, and use a cosine learning rate schedule without warm-up. We train for 2 epochs.
• Reward Model Training (i.e., Discriminator Dϕ training): Our reward model training stage involves training the discriminator. Similar to the SFT training, we use the batch size of 16 per device, with a learning rate of 10−5, and use a cosine learning rate schedule without warm-up. A weight decay of 0.1 is used, and we train for 2 epochs.
• Reinforcement Learning with PPO: For the RL stage, we use ZeRO-2, gradient checkpointing, and offload. The learning rate is set to 10−6, and we use a cosine learning rate schedule without warmup. We train for 1 epoch. The KL penalty coefficient is set to 0.1. We use ZeRO-3 for the reference model and the reward model, as they do not require training. The hybrid training engine is used.

heading: C Generated Response Comparison

heading: Future Work
text: Future works can include model-data scaling, iterative updating of the reward model and the policy model, and combining multi-objective preference (e. g. , safety reward and helpfulness reward) learning for more fine-grained guidance