DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling (2024)

Abstract: The performance of the reward model (RM) is a critical factor in improving the effectiveness of the large language model (LLM) during alignment fine-tuning. There remain two challenges in RM training: 1) training the same RM using various categories of data may cause its generalization performance to suffer from multi-task disturbance, and 2) the human annotation consistency rate is generally only 60% to 75%, causing training data to contain a lot of noise. To tackle these two challenges, we introduced the idea of Mixture-of-Experts (MoE) into the field of RM for the first time. We propose the Double-Layer MoE RM (DMoERM). The outer layer MoE is a sparse model. After classifying an input into task categories, we route it to the corresponding inner layer task-specific model. The inner layer MoE is a dense model. We decompose the specific task into multiple capability dimensions and individually fine-tune a LoRA expert on each one. Their outputs are then synthesized by an MLP to compute the final rewards. To minimize costs, we call a public LLM API to obtain the capability preference labels. The validation on manually labeled datasets confirms that our model attains superior consistency with human preference and outstrips advanced generative approaches. Meanwhile, through BoN sampling and RL experiments, we demonstrate that our model outperforms state-of-the-art ensemble methods of RM and mitigates the overoptimization problem. Our code is available at: https://github.com/quanshr/DMoERM.

heading: 1 Introduction
text: After an initial stage of pre-training and subsequent instruction fine-tuning (Quan, 2024), large language models (LLMs) undergo a crucial stage of high-quality alignment fine-tuning based on Reinforcement Learning with Human Feedback (RLHF) to improve their abilities (Ouyang et al., 2022; Stiennon et al., 2020). During the RLHF process,
a reward model (RM) often needs to be trained, which acts as a proxy of human preferences and assigns scores to the outputs of the LLM. The scores are then used as reward signals to optimize the LLM through reinforcement learning (RL). In this process, the LLM and RM are interdependent and iteratively optimized, and the RM is expected to be highly consistent with human preferences. In addition, during the inference stage, the RM can also be augmented with Best-of-n (BoN) sampling strategies to further enhance the quality of the outputs of the LLM (Ouyang et al., 2022; Nakano et al., 2021).
Training of reward models relies on the data derived from human annotators who manually rank the varying outputs under a single input by their preferences. However, many studies have found that agreement rates among human annotators typically only range between 60-75% (Ziegler et al., 2019; Stiennon et al., 2020; Dubois et al., 2023), thereby introducing a minimum of 25% noise within the labeled dataset. One important reason for this phenomenon is the multifaceted nature of evaluation—it is often observed that one response may excel in one aspect while simultaneously falling short in another. This multifaceted evaluation conundrum has been exemplified in previous studies (Dai et al., 2023; Ganguli et al., 2022; Bai et al., 2022), which illustrate the inherent tensions between enhancing helpfulness and harmlessness. As these attributes can at times be inversely related, adjudicating between a response that is more helpful yet potentially less harmless poses a significant challenge for comparative assessment. We further validate this perspective through empirical studies.
In this study, we pioneer the integration of the Mixture-of-Experts (MoE) framework (Jacobs et al., 1991; Lepikhin et al., 2021; Du et al., 2022) into Reward Modeling. Our approach employs a double-layer MoE architecture. The outer layer
7006
comprises a sparse MoE model specifically designed to avoid multi-task disturbance (Standley et al., 2020). As shown in Figure 1, We categorize inputs into several distinct tasks and use a pre-trained router to route the inputs to their corresponding task-specific expert. This strategy can also facilitate distributed deployment and enhance the model’s capacity and capabilities without a commensurate increase in computational demands (Rajbhandari et al., 2022; Xue et al., 2022).
Subsequently, within each inner layer lies a dense MoE model, which is tailored to the specific capabilities set required for its category. For instance, in roleplay scenarios, we divide it into six core capabilities, including personality and emotional investment, conversational sense, empathy ability and so on (details are in Appendix Table 12). We obtain preference labels on these single capability points by calling a public API, which greatly reduces annotation costs and is sufficient to achieve satisfactory results in our experiments. Considering that capability points are equivalent to a decomposition of tasks in a low dimensional space, using low-rank adaptation (LoRA) (Hu et al., 2022) finetuning will be very suitable. Each LoRA fine-tuned model effectively becomes an expert in scoring on a singular capability point. Lastly, we aggregate the outputs from these expert models into a unified one-dimensional score with an MLP to determine the final reward value. We believe this methodology can improve the interpretability and performance of RMs since it is just like the Chain of Thought (CoT) for RMs. Both preference consistency and optimization evaluations indicate our model is more effective in optimizing LLMs and can mitigate the overoptimization problem against other state-of-the-art RM ensemble methods.

heading: 2 Related Work

heading: 2.1 Mixed-of-Experts
text: Mixture-of-Experts or MoEs was introduced early for machine learning applications (Jacobs et al., 1991; Jordan and Jacobs, 1994), where researchers control the allocation of different weights to different models through gate networks to mitigate interference between different types of samples. MoE can enhance the model’s generalization capability by decomposing complex tasks into several subtasks, which will help avoid multi-task disturbance (Standley et al., 2020) and meanwhile confer greater flexibility in development (Ma et al., 2018). Recently, much ongoing research has focused on the top-k (e.g., top-1 or top-2 in many works) activation MoE model (Ramachandran and Le, 2019; Clark et al., 2022; Dai et al., 2022), since it can be leveraged to enlarge parameter count and enhance model capability while keeping computational complexity nearly unchanged for both training and inference due to its sparse activating nature (Shazeer et al., 2017; Fedus et al., 2022). While MoE has achieved great success in the field of large generative language models (Shen et al., 2023; OpenAI, 2023), how to efficiently train more effective RM with MoE architecture remains largely unexplored.

heading: 2.2 Reward Model Ensembling
text: Reward Model Ensembling has been tried in the field of safe RLHF (Dai et al., 2023). The research is based on a widely observed imagination: the pursuit of greater helpfulness and harmlessness may often conflict in practice (Ganguli et al., 2022; Bai et al., 2022). Another weave of research ensemble RMs through multi-objective reward modeling (Ramé et al., 2023b,a) or weight-averaged reward modeling (Ramé et al., 2024), but they are struggling to formulate non-linear relationships. In addition, some research works (Coste et al., 2024; Eisenstein et al., 2023; Zhai et al., 2024) have found that training multiple reward models and aggregating them by changing the data training order, batch size, and learning rate can alleviate the overoptimization problem (Gao et al., 2023) of RM and increase its performance. However, the aggregation methods they chose were only 1) mean, 2) min, and 3) mean minus std, and the performance of aggregation was very dependent on the diversity of several models (Zhai et al., 2024), which required a lot of attempts in experiments.

heading: 3 Empirical Study

heading: 3.1 Multi-Task Training
text: It has been frequently observed that using irrelevant training data to train LLMs will cause their generalization performance to decrease in other tasks (Dong et al., 2023; Wen et al., 2023). This also applies to RMs. Dai et al. (2023) found that training RMs on harmlessness and helpfulness simultaneously will lead to the model getting suboptimal results on both types of data. In order to further explore whether data of different categories interfere with each other, we selected the preference data for three tasks: roleplay, chitchat, and text creation. We train on different combinations of training sets and testing on all test sets. The results are shown in Table 1.
We find that using separate category data give the best results in this category, and using data from other categories may affect the generalization ability under original task.

heading: 3.2 Annotation Consistency
text: The consistency rate of manually labeled preference data is generally only 60-75% (Ziegler et al., 2019; Stiennon et al., 2020; Dubois et al., 2023), which brings a lot of noise to the training data. Inspired by the fact that Chain-of-Thought (CoT) can improve the accuracy of reasoning (Wei et al., 2022; Zhou et al., 2023), we try to improve the consistency rate through CoT. We first conduct experiments on humans. We select the text creation subtask and randomly divide 200 pairs of responses into two groups. For the first group, we let three annotators directly rank the preference and record the average agreement rate between any two anno-
tators, which is represented as A. For the second group, we divide the text creation into five capability points: 1) intent conformity, 2) expressiveness, 3) readability, 4) content richness, 5)logic, and ask the annotators to first score the content in each single capability point, and then evaluate the overall preference. The average agreement rates on five capability points are represented as B-1 to B-5, respectively, and the final overall agreement rate is B-f. We record these results in Figure 2, and a screenshot of the annotation interface is shown in Figure 8 in the Appendix.
We find that the consistency on capability points is significantly higher than the consistency of directly evaluating the overall preference, and the method of evaluating capability points first can increase the final overall consistency rate.

heading: 4 Methodology

heading: 4.1 Outer Layer MoE
text: Our first layer is a sparse MoE structure, and only the top-1 expert is activated each time. We divide the input into five categories according to tasks: text creation, roleplay, objective knowledge QA, subjective knowledge QA, and chitchat, and train an MoE for each category. When new input comes, we use a small frozen top-1 gating network pretrained on category labels to act as the router. Formally, we have
chosen = 4
argmax e=0
E(te|x) (1)
y = RMchosen(x) (2)
Here t0, . . . , t4 represent five tasks respectively, x represents input, RM0, . . . ,RM4 represent the
expert RMs corresponding to each task, and y represents the RM’s output.

heading: 4.2 Inner Layer MoE

heading: 4.2.1 Modeling
text: For each task, we first obtain a base task-specific model RMtbase from a general RM through training on task-specific preference data. Then, we divide the tasks into distinct capability points. Capability points are equivalent to a decomposition of tasks in a low-dimensional space. Define the input space as X , we need to learn an expert RMti : X → Zi. In this work, we obtain RMti by performing LoRA fine-tuning on RMtbase .
Let Z = Z0, . . . , Zk−1. After we have learned k experts, we use an aggregation network to aggregate the output results of k experts to produce the final output; that is, we learn RMt : Z → R. This is a Markov process and X → Z → R construct a homogeneous Markov chain.
We employ a FCN following each expert RM ti to serve as the value head to generate onedimensional scores, which further maps to the range of [0, 1] using the sigmoid activation function. Let W (base) represent the initial base model RM tbase, ∆W (i) represent the fine-tuned LoRA network learned for capability point i, and wi and bi represent the FCN network associated with capability point i. The score ri for capability point i is then expressed as follows:
zi = (W (base) +∆W (i))x (3)
ri = σ(wizi + bi) (4)
To obtain a single score as the final reward from multiple experts, we concatenate all lowdimensional representations of experts and use a two-layer MLP to aggregate them. Then the final reward score r is:
z = k−1 ⊕ i=0 zi (5)
r = σ(W1 PReLU(W0z +B0) +B1) (6)
Note that our MLP does not act on the final scalar outputs, but rather on the multiple low-dimensional decomposition without being fed into FCNs, as we believe there may be underlying correlations between different capability points, which can be learned using MLP in their low-dimensional embedding space.
We use logsigmoid as the loss function, which is also the most commonly used loss function for training RMs, where k represents the number of responses in a piece of data:
L = − 1( k 2 )E(x,yw,yl)∼D[log(σ(r(yw)− r(yl)))]
(7)

heading: 4.2.2 Training
text: Figure 3 illustrates the training framework of each inner layer MoE. We use a pre-trained RMtbase as the base general RM, and then perform the following three phases of training in sequence:
• Phase 1: Task-Specific Training. Use 60% of the task-specific preference data to fullparameter fine-tune on RMtbase to get base task-specific model RMtbase .
• Phase 2: Capabilities Training. Use the data with capability point labels (the method of obtaining these labels is introduced in Section 4.3) to train RMt using LoRA fine-tuning to obtain RMt0 , . . . ,RMtk−1 . Each time, a new linear head is learned from the original linear head of RMtbase .
• Phase 3: Ensemble Training. Remove the original FCN on each expert RM and use an MLP to aggregate RMt0 , . . . ,RMtk to obtain the final model RMt, and use the remaining 40% task-specific preference data to train it. During this phase, RMtbase and the LoRA layers are frozen, and only the newly added MLP layer is trained.

heading: 4.3 Capability Point Labels Obtaining
text: Since it is costly to obtain all preference labels on each capability point, instead of manually sorting or scoring, we use the method of calling the public Ernie Bot API1 to obtain the comparative preference of the single capability point in each response pair, which significantly reduces the labeling cost.
To avoid the positional bias inherent in LLMs, we employ a strategy that involves swapping positions and requiring each pair to be processed twice. We then selectively retain only those data pairs that exhibit consistency in twice-calling. This method also doubles as a data cleansing technique, as it effectively filters out pairs with minimal discrepancies, which may introduce potential noise into training data. The prompt template is shown in both Chinese (Table 13) and English (Table 14) in the Appendix.
Our approach does not require additional data since the task-specific data from training Phases 1 and 3 can be directly utilized as raw response
1The ERNIE Bot API has similar functions to ChatGPT, but it is much cheaper and can achieve the same level of proficiency in Chinese: https://cloud.baidu.com/doc/ WENXINWORKSHOP/s/flfmc9do2
pairs, which enables us to acquire capability point preferences based on them. In our experiments, we reutilize all the task-specific data as raw response pairs during Phase 2.

heading: 5 Experiment Setup

heading: 5.1 Model
text: We use Qwen-1.8B-Chat2 as both the policy model and the base reward model, which is an open source Chinese-English bilingual Transformer-based large language model proposed by Alibaba Cloud. To make a fair comparison, we use the same model as the base model of our baseline ensemble methods.

heading: 5.2 Baseline
text: We use a single RM baseline and collect three stateof-the-art ensembling methods for reward models from a range of papers (Coste et al., 2024; Eisenstein et al., 2023; Zhai et al., 2024). All baseline methods and our model are trained and evaluated with the same dataset.
• Single RM
We use the training of a single reward model as the most basic benchmark.
• Mean Optimization
Mean optimization simply takes the mean of the outputs of the different ensemble members:
Rµ(x) = 1
k
k−1∑
i=0
ri(x) (8)
• Worst-Case Optimization
Worst-case optimization (WCO) creates a conservative estimate by choosing the lowest reward from the ensemble at every step:
RWCO(x) = k−1 min i=0 ri(x) (9)
• Uncertainty-Weighted Optimization
Uncertainty-weighted optimization (UWO) calculates reward by combining the average reward across all models in an ensemble with the intra-ensemble variance, weighted by a
2https://huggingface.co/Qwen/Qwen-1_8B-Chat
coefficient λ. Mathematically, this objective is given by:
RUWO(x) =
Rµ(x)︸ ︷︷ ︸ mean
− λ 1 k
k−1∑
i=0
(ri(x)−Rµ(x))2
︸ ︷︷ ︸ variance
(10)

heading: 5.3 Dataset
text: Our prompt dataset are very diverse and can be mainly classified into five categories: roleplay, chitchat, subjective knowledge QA, objective knowledge QA, and text creation, with some others (including logical reasoning, mathematical calculations, code understanding and generation, translation, etc). Our dataset is over 98% Chinese. We ensure that the training and test sets contain no intersection and heuristically eliminate duplicate prompts. We also filter prompts containing personally identifiable information (PII).
Our data has a turn range of 1 to 27, with an average of 3.72 (a turn contains a user query and an LLM response). Each sample’s final query has multiple responses generated through either automated or manual processes. These responses are then assigned preference rankings through manual labeling. A detailed statistic of our data is shown in Table 2.
We employ a rigorous annotation process with clear documentation guides to direct our annotators (we have shown an example in our GitHub repository). Each piece of data is evaluated by two annotators to ensure quality, and the final ranking is established through discussion to reach a consensus. We have observed that the consistency rate between each pair of annotators reached 74% on average.

heading: 6 Result

heading: 6.1 Training Phases
text: In this section, we use roleplay as an example to demonstrate the progress of the model at different training phases. By recording the accuracy of the reward model’s ranking on the pairs of responses in training and testing sets, we can intuitively display the training results. The results are shown in Figure 4. We find:
• In Phase 1 (Figure 4a): the accuracy of the training set improves rapidly, while the improvement of the test set dataset is slow, eventually stabilizing at 56%.
• In Phase 2 (Figure 4b to 4g): we divide the roleplay task into six capability points, namely 1) personality and emotional investment, 2) conversational sense, 3) empathy ability, 4) manifestation of relationship traits, 5) personalized characteristic expression, and 6) content richness. We find that depending on different single capability points, an accuracy of 80-86% can be achieved on the test set. Note that the training and testing set labels here are only one capability point and not an overall preference.
• In Phase 3 (Figure 4h): the improvement in the test set was rapid, reaching a peak of 68%. Compared with the 56% accuracy in Phase 1, it can be proven that our method of training multiple experts based on different capability points and aggregating those experts can significantly improve the model’s performance.

heading: 6.2 Preference Consistency Evaluation
text: Since RM is essentially an imperfect proxy for human preferences, testing the consistency rate of trained RM using human-labeled preference data is a direct and effective evaluation method. Given a pair of preference data, we use the trained RMs to assign scores to each response and record the consistency between the sorting of scores and the sorting of manual labels. Higher consistency rates (or accuracy) mean better performance of RMs as the proxies for human preferences. Compared with the methods introduced in Section 5.2, we add the GPT-4 generative evaluation benchmarks for a more comprehensive evaluation. Their implementation details are introduced in Appendix B.3.3.
We present the results in Table 3. There are two noteworthy findings: firstly, our DMoERM achieves the best results in all categories and overall experiments, with a 6 to 8 percentage point improvement compared to other methods. This indicates that our training method can better learn human preferences without increasing the amount of training data or model parameters. Secondly, the DMoERM-w/o-Outer model removes the outer layer and instead uses the same capability point partition to train on various categories. It achieves the second effect and still has a significant improvement compared to other methods, making it a viable alternative when memory or task-specific data is limited.

heading: 6.3 Optimization Evaluation
text: In the optimization evaluation, we use BoN and PPO as optimization strategies to optimize the same policy model. Since we have verified in section 3.1 that the outer MoE will improve performance by dividing different categories to train different models, for the sake of fair comparison, we restrict the task within roleplay to comparing the effects of the corresponding inner MoE with other aggregation methods. Under the same degree of optimization measured by KL divergence, we use another pre-trained reward model3 fine-tuned on our taskspecific data with a larger number of parameters as
3https://huggingface.co/IDEA-CCNL/ Ziya-LLaMA-7B-Reward
the referee to judge the outputs of results, which we called the gold reward model. Model strategies that score higher under the gold reward model are considered better. We evaluate BoN for a maximum of nmax = 60, 000 samples, which roughly equals 10 nats of KL. The reported results represent the mean outcomes derived from a set of 500 distinct prompts. For PPO, we train for 3, 000 PPO steps and compute the average results derived from three distinct random seeds. We give further details on implementation and other hyperparameters in Appendix B.3.
We showcase the results in Figure 5. Both BoN and PPO optimization results demonstrate that our model consistently outperforms alternative integration approaches when policy models are optimized to the same degree. Moreover, after n = 8, 000 (KL ≈ 8 nats) in the BoN optimization experiment, our model maintaine stability without signs of overoptimization, unlike other ensemble methods that exhibit varying degrees of overoptimization. These findings suggest that our model outperforms baselines in optimizing LLMs and is capable of addressing the issue of overoptimization.

heading: 7 Conclusion
text: In this work, we propose DMoERM to enhance RMs’ performance. The outer layer MoE divides inputs into different tasks to avoid multi-task disturbance, while the inner layer MoE reduces the impact of data noise by learning LoRA experts on
different capability points. Preference consistency experiments demonstrate our model is more representative of human preferences. Optimization evaluations indicate our model is more effective in optimizing LLMs and can mitigate the overoptimization problem.

heading: Limitations
text: Although we attempt to reduce annotation costs by calling public LLM API instead of manual labeling, it is still costly as using the ERNIE Bot API will cost approximately $3, 000 in total, and using the ChatGPT API will cost about ten times more. However, if the annotation standards for human are predetermined in advance, it will not significantly bring more annotation costs. In our annotation process, we found that due to the annotators spending most of their time understanding the queries and various responses, the proposed annotation method (first annotates the preferences of pre-determined capability points and then annotates the overall preferences) only reduces the annotator’s speed by about 10%, but can increase the annotation consistency by 5 percentage points.
Another problem is the training time, with about 80 NVIDIA A100 GPU hours to train one inner MoE, which is about eight times longer than training a traditional single RM with the same amount of parameters. While many works focus on exploring efficient training methods, we reserve this problem as a future work.

heading: A Additional Related Work

heading: A.1 Reinforcement Learning with Human Feedback
text: Reinforcement Learning with Human Feedback (RLHF) is a foundational method for fine-tuning language models to align with human preferences. RLHF has been applied to a variety of tasks, including text summarization (Stiennon et al., 2020) and improving the helpfulness and harmlessness of language models (Bai et al., 2022). In particular, InstructGPT (Ouyang et al., 2022) employs a three-step RLHF process that includes a supervised learning technique and the PPO algorithm (Schulman et al., 2017), which has proven to be effective for ChatGPT. Despite its success, RLHF encounters several challenges, such as low sample efficiency (Snell et al., 2023; Gülçehre et al., 2023) and overoptimization (Gao et al., 2023). Since we require no additional data to improve performance and mitigate overoptimization, our method works in both two aspects.
RLHF heavily depends on reward modeling as a proxy for human preferences. Recent research has attempted to bypass the reward modeling step (Yuan et al., 2023; Song et al., 2023). Specifically, Direct Policy Optimization (DPO) aims to refine policies by classifying human preference data without reward models. Although this
method is simpler to implement and offers training stability, more recent studies reveal several advantages of using reward models. Investigations into the robustness of reward-model-based strategies suggest they are more resistant to overfitting due to the limitations of KL regularization (Azar et al., 2023). Moreover, in comparison to DPO, rewardmodel-based RLHF shows great advantages on outof-preference samples (Li et al., 2023).

heading: A.1.1 the Overoptimization Problem of RMs
text: As the learned reward model is only a proxy for the true reward function, optimizing it may not always result in an improvement according to true human preferences. In practice, optimizing a (fixed) learned reward model almost always leads to improvement according to this learned reward model but only improves according to the true reward model (i.e., humans) for some initial period, after which performance often begins to regress. This phenomenon is referred to as overoptimization.

heading: B Additional Experiment Setup

heading: B.1 Hyperparameters
text: Since in different training phases there are different sets of model parameters that need to be trained, we use different learning rates to better adapt to these three phases and allocate different proportions of training set for Phase 1 and Phase 3. The settings are shown in Table 4.
We set the minibatch size to 1 and conduct an evaluation on the validation set every 100 training steps. If the best result on the validation set does not improve for 20 consecutive evaluations, we implement early stopping and use the best-performing model for the next training phase or the final testing if it is already Phase 3.
The settings for the LoRA components are presented in Table 5, applicable to both our LoRA experts and the LoRA ensembles used in the baseline comparisons.
The PPO hyperparameters and generation hyperparameters are shown in Tables 6 and 7, respectively.
All experiments are run on a single machine with eight NVIDIA A100 80G GPUs, and we use Adam optimizer for the optimization process.

heading: B.2 Ensemble Creation for Baseline Methods
text: To create an ensemble for the Mean, WCO, and UWO baselines, under Coste et al. (2024); Eisenstein et al. (2023) guidance, we train a fixed number of proxy reward models using identical data and hyperparameters. Each model, however, is initialized with different random seeds. This leads to variations in the random initialization of the scalar reward head that is added on top of the pre-trained language model, as well as in the data shuffling order. Table 5 presents the LoRA parameters for
these models. We have trained an ensemble of five reward models, aligning with the configurations used in previous works. This number is also comparable to the number of LoRA experts in our model, which ranges from five to six, depending on the numbers of capability points allocated under a category.

heading: B.3 Optimization Method
text: B.3.1 Best-of-n Sampling Best-of-n (BoN) sampling, also called rejection sampling, is a simple inference-time optimization method (Nakano et al., 2021; Ouyang et al., 2022). For a given prompt, n responses are generated from the policy model, and the answer with the highest proxy reward model score is returned. To evaluate the degree of optimization, the KL divergence 4 is defined analytically as a function of n:
KLbon = log n− n− 1 n
(11)
In our experiments, we evaluate BoN for a maximum of nmax = 60, 000 samples, which roughly equals 10 nats of KL.

heading: B.3.2 Proximal Policy Optimization
text: Proximal Policy Optimization (PPO) (Schulman et al., 2017) is a policy-gradient-based online reinforcement learning method that maximizes a given reward function by repeatedly performing small incremental updates to the policy. PPO is the standard algorithm used in fine-tuning language models based on human feedback (Ouyang et al., 2022; Bai et al., 2022; Stiennon et al., 2020; Zheng et al., 2023). When using PPO to fine-tune a language model, a KL penalty term is added during the reward calculation to regularize the policy by preventing it from deviating far from the initial policy:
RPPO(q, r) = R(q, r)− βKLPPO (12)
where πPPO is the policy being optimized and πinit is the initial (pre-trained) language model. The naive way to calculate KL divergence between the PPO-optimized policy πPPO and the initial model is as follows:
4A recent work (Beirami et al., 2024) claims to have proved that this boundary is not so accurate and provided a tighter boundary defined by the binary entropy function. At present, they have not been widely recognized, and even if they are correct, our experimental conclusions are still valid in terms of trends.
KLPPO(π PPO, πinit) = E(q,r)∼πPPO [log πPPO(r|q) πinit(r|q) ]
(13) However, this estimator suffers from high variance and may yield negative values. Consequently, we employ the following estimator (Coste et al., 2024):
KLPPO(π PPO,πinit) =
E(q,r)∼πPPO [ 1
2 (log πPPO(r|q) πinit(r|q) ) 2]
(14)
We train a total of 3, 000 PPO steps, and the PPO parameters are shown in Table 6.

heading: B.3.3 Generative Baseline
text: Apart from RM-based methods, if only the preference order of response pairs needs to be given, a simple way is to call the public language model API through zero-shot or few-shot. It can also be combined with BoN or DPO to optimize LLM. For a more comprehensive evaluation, in preference consistency experiments, we have added the GPT-4 baseline.
• GPT-4 (OpenAI, 2023) We use the most advanced gpt-4-1106-preview 5 as the evaluator to evaluate a pair of preference data. Each time, we swap the positions of the responses and make two requests, and we will re-request until we get identical results from two requests as the final results.
We try to find well-prompted instruction and use both zero-shot and one-shot to evaluate and do not use few-shot for comparison, for we find it will make the context much longer and the result is not better than one-shot. There are also some other generative evaluation methods. Since they have reported in their papers that their performance is just similar with (Ke et al., 2023) GPT-4 or even worse than (Wang et al., 2023) GPT-4, we just use GPT-4 to represent these methods.

heading: C Discussion on Model Interpretability
text: The single traditional RM only outputs the final reward score and we have no clues about why a
5https://platform.openai.com/docs/models/ gpt-4-and-gpt-4-turbo We called the API in January 2024.
response gets its score, which makes it have poor interpretability. However, our DMoERM first learns latent embedding at different capability points under specific tasks, and the final score is learned from and based on an ensemble of embeddings of different capability points. By using the FCN trained in Phase 2 to act on the corresponding hidden embedding, we can know the score on each capability point of a response. In this way, we can identify which aspects of a good response are effective and which aspects of a poor response are not. Similarly, if the overall consistency of the model is not satisfactory, we can also identify the problem and prescribe targeted solutions by analyzing the performance of each single capability point.
The method that our DMoERM checks the task of inputs, works on different capability points, and aggregates latent embeddings of each expert to obtain the final reward is just like the Chain-ofThought of the reward models, which can enhance the reasoning (give the reward under the context) ability of reward models as well as the interpretability of the reward models.

heading: D Additional Experimental Result

heading: D.1 Larger Model Performance
text: We further apply the preference consistency evaluation to the Qwen7B-Chat and Qwen14B-Chat models. Except for adjusting the learning rate to adapt to different parameter sizes ([5e−8, 2.5e−5, 5e− 7] for the 7B model and [3e−8, 1.5e−5, 3e−7] for the 14B model, during the three training phases), the other parameter settings and dataset partitioning are consistent with the paper, and the specific experimental results are shown in the Table 10.
We find that our DMoERM achieved the best results among all categories and all model sizes (in bold), while DMoERM-7B achieved the second best results (is underlined). Meanwhile, in the comparison between our DMoERM in different parameter sizes, the final results for different categories improve by about 1.9% for 7B and 3.6% for 14B against the smallest 1.8B model, indicating that our model structure can benefit from scaling up the size of parameters. The increase in consistency rate is not too significant when scaling up due to the presence of certain noise in the labeled data (with observed 74% consistency among human annotators), which leads to it being challenging to further improve when consistency rates near the 70% mark. Meanwhile, due to the amount of train-
ing data is fixed, the larger models may not have been sufficiently trained, which may also be a factor contributing to the less significant improvement.

heading: D.2 Human Evaluation
text: In the process of BoN and PPO optimization, we both set three checkpoints (n = 30, 1000, 6000 for BoN and 1000, 2000, 3000 PPO steps for PPO) and use 100 prompts at each checkpoint for human evaluation. Specifically, for the different outputs obtained from these prompts under optimizations from different RMs, we require annotators to sort them according to their preferences, and we record the winning rates of our model against other ensemble methods in Figure 6.
We have two findings: Firstly, compared to BoN, the comparison improvement under PPO optimization is slower. Secondly, in both optimization experiments, the winning rates of our model increase steadily as optimization progresses, ultimately reaching approximately 87% for BoN and 68% for PPO.

heading: D.3 OOD Optimization Evaluation
text: In contrast to several previous works (Gao et al., 2023; Coste et al., 2024; Eisenstein et al., 2023; Zhai et al., 2024) where the RM training set, optimization set, and evaluation set are typically independently and identically distributed (IID), our following experiments involve using out-ofdistribution (OOD) RM training sets in PPO I and OOD evaluation sets in PPO II (since BoN is a training-free optimization method used in the inference stage, the optimization prompt set is directly the evaluation prompt set, so we only conduct OOD optimization experiments on PPO). We use AlignBench (Liu et al., 2023), a publicly and popularly used alignment benchmark in Chinese, to evaluate the effectiveness of different models. We use the DMoERM-w/o-Outer for comparison and set the reward models and policy models in the same model sizes of 7B. Due to the models being larger, we compare the optimization results of different RMs using GPT-4 as the referee, which we believe has the most sufficient capability to evaluate the generation results and is naturally fair. The RM training set, the optimization set, and the evaluation set used in PPO I and PPO II are represented in Table 8 and the experimental results are shown in Figure 7.
We have two significant findings: First, PPO I and PPO II show consistent trends, both steadily improving the winning rate against the baseline method during the optimization process. Second, In PPO II, the winning rate initially increased more rapidly (before 1, 000 steps), while the growth in PPO I was more stable and persistent in all optimization periods.
Intuitive analysis: Due to the fact that in PPO II, the optimization set and training set are IID, the reward model can provide a clear reward signal during optimization, enabling training to quickly achieve positive optimization; conversely, In PPO I, while the optimization and evaluation sets are IID, they are OOD with the training set, making
the reward signal is not such clear and strong, but advantages can gradually be reflected in the continuous optimization process.

heading: D.4 Annotation Quantitative Experiment
text: Since our method requires more annotation costs, to provide a more comprehensive evaluation of our method, we quantified the annotation consumption and maintained consistency between our method and the baseline method in terms of annotation consumption. In this experiment, we compared DMoERM-w/o-Outer with other baseline models in all three model sizes on the entire dataset. Due to the need that our model requires each piece of data 5 times additional annotations on intent conformity, logic, conversational sense, content richness, and readability, we only randomly sample one-sixth of the data to train our model, while other models use all the data to control the consistency of the number of annotations. The experimental results are shown in Table 9.
We also recorded the performance of our DMoERM-w/o-Outer at the end of the training Phase 1 as DMoERM-Phase1. At this point, it is equivalent to a Single RM trained with only one-
sixth of the data, so the effect is poor. But when the three training phases ended (denoted as DMoERMPhase3), its performance improved by 9.3 percentage points on average and exceeded all baseline models in three model sizes.
Note that this experiment is not very fair because in our empirical experiments, we found that our annotation method only reduces the annotator’s speed by about 10%, but can increase the annotation consistency by 5 percentage points. In the experiment, we only used one-sixth of the data, but exceeded all the baseline methods by at least 1.4 percentage points, indicating the compelling advantages of our method.

heading: D.5 Qualitative Sample
text: Throughout our experiments, quantitative metrics play a pivotal role as they enable the rapid and comprehensive evaluation of various methods. In this section, we offer a concise insight into the qualitative aspect of the approaches discussed in this study. Specifically, for a given prompt, responses from the ultimate policies (n = 60, 000 for BoN and 3, 000 steps for PPO) of each method in Figure 5 are provided.
The main findings are as follows. Firstly, the brevity of BoN’s response is due to the differing manner in which policy optimization occurs compared to PPO. Secondly, there are indications of failure and overoptimization in the other ensemble RMs. As shown in Table 11, for BoN, this manifests in inadequate answers. For PPO, this means bad answers that are incorrectly long and repetitive. These are obvious signs of overoptimization. Lastly, we observe that even in scenarios where other ensemble RMs struggle, our DMoERM is capable of yielding robust qualitative outcomes.
Query 如何简洁地描写点心的香味从盒子里 飘出 How to briefly describe the aroma of the dim sum floating out of the box
Mean (PPO)
盒子轻轻打开，点心的香气瞬间弥 漫。那是糯米的温润，豆沙的甜蜜， 还有一丝丝清香的茶叶味。一阵阵， 一股股，绵绵不绝，像是诗一般的甜 美旋律，缭绕在空气中，点心的清香 让人垂涎欲滴，心生欢喜。 The box opens gently, and the aroma of the dessert instantly spreads. It is the warmth of glutinous rice, the sweetness of red bean paste, and a trace of fresh tea flavor. Wave upon wave, continuous and unending, like a poetic sweet melody lingering in the air, the fragrance of the dessert makes people salivate
and feel delighted.
WCO (PPO)
点心的香味从盒子里缓缓飘出，一次 又一次地扑鼻而来，像是阵阵潮汐， 持续不断地冲击着味蕾。每一波香气 都带着诱人的酥脆和甜蜜，让人忍不 住想要尝一口。这就是点心的魅力， 它的香味像是魔法，一次又一次地吸 引着人们。 The aroma of the dim sum slowly wafts out of the box, hitting the nostrils over and over again, like waves of the tide, constantly impacting the taste buds. Each wave of fragrance brings tempting crunch and sweetness, making it hard to resist taking a bite. This is the charm of dim sum, its fragrance is like magic, attracting people over and over again.
UWO (PPO)
点心的香气，如同精灵的歌唱，娓娓 地从盒子的缝隙中溢出。一次，两 次，每一次都像是在空气中划出一个 甜美的弧线。轻轻一闻，那香气又再 次被唤醒，从盒子里流淌出来，一次 又一次，像是永无止境的旋律，让人 沉醉。 The aroma of the dim sum, like the singing of elves, spills out of the cracks of the box in a continuous stream. Once, twice, each time it seems to draw a sweet arc in the air. With a gentle sniff, the aroma is awakened again, flowing out of the box, over and over again, like an endless melody that intoxicates the senses.
DMoERM (PPO)
点心的香气如轻烟，缓缓从精致的盒 子中逸出，弥漫在空气中。那甜美的 芬芳，仿佛有魔力般，无声地诱惑着 每一个嗅觉细胞，让人不由自主地靠 近，沉醉在这份难以抗拒的美味预告 中。 The aroma of the dim sum, like light smoke, escapes slowly from the exquisite box and pervades the air. That sweet fragrance, as if magical, silently tempts every olfactory cell, drawing people irresistibly closer, intoxicated by this irresistible preview of deliciousness.

heading: Future Work
text: While many works focus on exploring efficient training methods, we reserve this problem as a future work