InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

Authors: yuchun  miao
sen  zhang
liang  ding
rong  bao
lefei  zhang
dacheng  tao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:21:52

Abstract: Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models with human values, reward hacking, also termed reward overoptimization, remains a critical challenge. This issue primarily arises from reward misgeneralization, where reward models (RMs)  compute reward using spurious features that are irrelevant to human preferences. In this work, we tackle this problem from an information-theoretic perspective and propose a framework for reward modeling, namely InfoRM, by introducing a variational information bottleneck objective to filter out irrelevant information.
Notably, we further identify a correlation between overoptimization and outliers in the IB latent space of InfoRM, establishing it as a promising tool for detecting reward overoptimization.
Inspired by this finding, we propose the Cluster Separation Index (CSI), which quantifies deviations in the IB latent space, as an indicator of reward overoptimization to facilitate the development of online mitigation strategies. Extensive experiments on a wide range of settings and RM scales (70M, 440M, 1.4B, and 7B) demonstrate the effectiveness of InfoRM. Further analyses reveal that InfoRM's overoptimization detection mechanism is not only effective but also robust across a broad range of datasets, signifying a notable advancement in the field of RLHF. The code will be released upon acceptance.

Main paper: 
 introduction: 
 With the advent of large language models (LLMs), reinforcement learning from human feedback (RLHF) has emerged as a pivotal technological paradigm to align models' behaviors with human values [57, 33, 4, 25] . One of the core stages of RLHF is reward modeling, where a proxy reward model (RM) is learned to mimic human preference by training on a preference dataset that contains sets of responses with human rankings. Then a reinforcement learning (RL) stage follows to align the LLM with human preferences by optimizing rewards from the learned proxy RM. Despite empirical success, RLHF has been criticized for its vulnerability and instability [6] . One widely revealed cause is reward hacking, also known as reward overoptimization, a phenomenon where the policy model's optimization, though seemingly effective under the proxy RM, actually diverges from the true human objectives [57, 41, 16] . This issue can be manifested in various ways, from copying styles without generating meaningful content to exhibiting excessive caution in responses [10, 51] .One primary cause of reward overoptimization in the reward modeling process is reward misgeneralization [6] , where RMs may incorrectly generalize training data, resulting in poor proxies for actual human preference. This problem arises because the same set of human feedback can be interpreted in multiple ways by RMs, even when ample training data is available [40] . Consequently, RMs tend to depend on spurious features-those unexpected or contingent elements that correlate with the ranking InfoRM distinguishes itself by enhancing RM generalizability through mutual information modeling. Additionally, a distinct feature of InfoRM is its overoptimization detection mechanism, which can guide parameter selection and algorithm design in subsequent RLHF. Specifically, the RM encoder is derived from the standard RM, with modification to the final layer. labels but are irrelevant to actual human preferences, such as length bias [38] . Over-exploiting such information results in RM overfitting, which significantly undermines its generalizability and poses a notable challenge for RM in handling the dynamic response distribution during the RL stage, leading to an unstable RL process [45, 29] . Current efforts in mitigating reward overoptimization mainly include incorporating Kullback-Leibler (KL) divergence as constraints [44, 49, 33] , enlarging the scale of RM [16] , employing composite RMs [10, 14, 30, 36] , optimizing preference dataset [56] , and specifically addressing response length bias [7, 38] . However, none of these approaches take the aforementioned reward misgeneralization issue into account.In this work, we propose a new reward modeling framework from an information-theoretic perspective, namely, InfoRM, which effectively addresses the aforementioned reward misgeneralization issue. InfoRM takes inspiration from the recent advancements in deep variational inference and mutual information (MI)-based learning theory [34, 18, 52] . Specifically, we translate the reward modeling problem into optimizing a variational information bottleneck (IB) objective function. This approach aims to filter out information irrelevant to human preferences from the IB latent representation, which acts as a crucial intermediary between the RM outputs and the corresponding human preferences; please see Figure 1 for comparison between standard RM and InfoRM.The advantages of our framework are two-fold: Firstly, benefiting from the MI modeling, InfoRM eliminates human preference-irrelevant information from the IB latent representation to achieve generalizable human preference modeling. This approach directly addresses the reward misgeneralization challenge by ensuring that only pertinent features that genuinely reflect human preferences are retained within the IB latent space. Supporting experiments are detailed in Appendix D. Secondly, InfoRM also stands out for its potential in overoptimization detection. In particular, we discover a correlation between reward overoptimization and the emergence of numerous outliers in the latent IB space of InfoRM, a phenomenon not observed in RM without IB. Motivated by this observation, we design the Cluster Separation Index (CSI) as an indicator of reward overoptimization, which identifies such outliers by quantifying the deviations of RLHF model-generated sample distributions; please see Section 5 for experimental validation. The proposed CSI not only facilitates parameter adjustments in InfoRM within real-world scenarios when lacking the gold RM but also provides an informative tool for online mitigation strategies such as early stopping; see Appendix E.2 and G.Building on these advantages, our method mitigates the risk of reward overoptimization in RLHF, resulting in enhanced RLHF performance, as illustrated in Figure 2 . We summarize our main contributions as follows:• We introduce InfoRM, a new reward modeling framework based on information theory principles, to tackle the reward misgeneralization challenges by bottlenecking the irrelevant information.• We propose CSI, an effective indicator for reward overoptimization detection, derived from our insight into the correlation between overoptimization and outliers in the IB latent space of InfoRM.• We empirically demonstrate that InfoRM significantly outperforms standard RMs in RLHF performance, particularly in mitigating reward hacking. Furthermore, our metric for detecting reward overoptimization has proven both effective and robust, marking a significant advancement in RLHF. 
related work: 
 Our work draws inspiration from two lines of research, i.e., reward overoptimization in RLHF and information bottleneck-family methods.Overoptimization! Figure 3 : An example of reward overoptimization in RLHF characterized by a declining gold score (i.e., actual human preference) and a rising proxy score (i.e., proxy RM preference).Reward hacking, also termed reward overoptimization, presents a prominent challenge in RLHF, stemming from the limitations of imperfect proxy RM for human preference [21, 57, 41] . In practice, optimizing a learned proxy RM typically results in improvements according to this proxy. However, it only enhances performance in line with the gold RM-actual human preference-for an initial period, after which the performance often starts to deteriorate; please see Figure 3 for an illustration.To mitigate this issue, a widely adopted strategy is introducing KL divergence penalty to regulate the output deviation of the policy model from the supervised finetuning (SFT) model [44, 49, 33] . Although this strategy occasionally works in alleviating reward overoptimization, it inherently restricts the optimization landspace and is prone to overfitting [3] , resulting in degraded RLHF performance [16] . Alternatively, enlarging RM scale [16] , implementing RM ensembles [10, 14] , and composing RMs from multiple perspectives [30, 36] , have been explored to address this issue. Scaling up network size or quantity, as proposed by these approaches, presents limited feasibility and may incur significant costs, especially for models with billions of parameters [51] . Moreover, recent efforts to optimize RM training datasets [56] , and address the specific issue, i.e., response length bias [7, 38] , continue to overlook the human preference-irrelevant information in reward modeling, which perpetuates the issue of reward misgeneralization.Our approach is distinct from existing methods by specifically targeting the underlying challenge of reward misgeneralization-a fundamental driver of reward overoptimization. Consequently, our InfoRM, not only significantly reduces reward overoptimization via a single RM, but offers a valuable tool for detecting this phenomenon during RL stage, which facilitates parameter selection in real scenarios without gold RM and development of online mitigation strategies, such as early stopping.Information bottleneck (IB) is a well-established technique for learning an informative and compact latent representation as a balance between the conciseness and predictive power [42, 39, 43] . To address the challenge of optimizing the corresponding mutual information, Alemi et al. [1] presents a variational approximation to the IB objective. This paradigm has successfully extended to various scenarios [19, 18, 12, 52] . Inspired by these works, we introduce the IB principle into reward modeling in RLHF and derive an optimizable variational bound for this ranking problem. Notably, while the aforementioned methods primarily use IB for extracting target-related information, our work makes a step forward by further exploring the informative and compact nature of the learned IB latent representation space, leading to the development of a tool for detecting reward overoptimization. To the best of our knowledge, this is the first effort to connect IB with RLHF and demonstrate its effectiveness in the context of LLM.The response is marked as a Hacking sample if it refuses to directly answer the query due to being overly concerned and sensitive about security, thus being unhelpful to the general query that doesn't directly pertain to securtiity issues. 
preliminary: 
 Reward modeling aims to learn a proxy RM that mimics the underlying human objective, providing the human preference rankings y of response sets from human preference datasets where each sample is denoted as x = x w , x l . Here, x w and x l denote the chosen and rejected samples, respectively. 2Following Bradley-Terry Model [5] , by employing the learned proxy RM r θ (x), the preference distribution p θ (y) = p θ x w x l can be formulated as:EQUATIONwhere r θ (•) represents the learned proxy RM and θ collects the model parameters. Standard reward modeling approaches typically regard this problem as a binary classification task and optimize a negative log-likelihood loss [44, 49, 4] :EQUATIONwhereD = {(x i , y i )} N i=1 = {(x w i , x l i )} N i=1is the human preference dataset, 3 and σ(•) is the logistic function. Within the domain of LLM, the proxy RM is commonly initialized with the SFT model. Subsequently, it integrates an extra linear layer at the final transformer layer, producing a single scalar prediction for the reward value. Nonetheless, as discussed in Section 1, this paradigm is prone to reward misgeneralization during the training process, focusing too much on the trivial aspects of training samples while neglecting meaningful information relevant to human preferences. As a result, although the model may exhibit exceptional performance on training data, it tends to struggle with generalizing to unseen data. This limited generalizability of RM leads to the reward overoptimization phenomenon, a critical concern in the subsequent RL process, which necessitates the generalizability of RM to the constantly evolving sample distributions.Addressing the challenge of reward misgeneralization necessitates the capacity of RM to efficiently capture information pertinent to human preferences while discarding the irrelevant details, which aids in preventing overfitting to the human preferences-irrelevant information present in the training samples, thereby significantly enhancing model generalizability [52] .To this end, we tackle these challenges by reformulating the reward modeling process from an information theoretic perspective. Specifically, we quantify the human preference irrelevance and the utility of a latent representation for reward prediction in information-theoretic language. We first denote the random variables corresponding to RM input, the latent representation, and the human preference ranking as X, S, and Y , respectively. 4 By assuming a Gaussian distribution for the latent representation S, we define I bottleneck = I (X; S|Y ) and I preference = I (S; Y ) to provide quantitative measures for the irrelevance of human preferences in latent representation and the utility of latent representation for reward prediction respectively, where I denotes the MI. Therefore, the objective of our information-theoretic reward modeling framework J(θ) can be formulated as follows:EQUATIONwhere β is a trade-off parameter, and θ encompasses all the parameters in this objective. In Eqn. (3), the latent representation S essentially provides an information bottleneck between the input samples X and the corresponding rankings Y . Due to the high dimensionality of the input sample space, it is non-trivial to evaluate these two MI. Thus, given a human preference dataset D = {(x i , y i )} N i=1 and θ = {φ, ψ}, we instead optimize a variational lower bound J VLB :J(φ, ψ) ≥ J VLB (φ, ψ) = E (x,y)∼D [J preference -βJ bottleneck ]J preference = p φ (s|x) log q ψ (y|s)ds where r(S), J preference , and J bottleneck denote the variational approximation of the marginal distribution p(S), 5 the lower bound of I preference , and the upper bound of I bottleneck , respectively. Here, p φ (s|x) extract latent representations, and q ψ (y|s) handles ranking prediction based on the generated representation. The parameters of these two functions are collected in φ and ψ, respectively.EQUATIONIn our practice, the functions p φ (s|x) and q ψ (y|s) are modeled by an LLM with an extra head f φ (•) for representation generation, and an MLP g ψ (•) for reward prediction, respectively. Notably, p φ (s|x) is modeled as a multivariate Gaussian with a diagonal covariance structure, where the mean and covariance are both determined by the output of the encoder f φ (x), i.e., f µ φ (x) and f σ φ (x). Referring to Eqn. ( 4), the objective for our information-theoretic reward modeling reads:EQUATIONwhere h φ (x, ) = f µ φ + f σ φ (x) . w and l are independently sampled from N (0, I) for each input sample. L preference and L bottleneck are the estimates of J preference and J bottleneck in Eqn. (4), respectively. Detailed derivation is provided in Appendix A, and related pseudocode is provided in Appendix J.1.Remark I: Although InfoRM focuses on reward modeling, our ultimate goal is to mitigate reward overoptimization in RLHF by addressing the reward misgeneralization issue. Thus in subsequent experiments, we evaluate RLHF model performance to demonstrate the effectiveness of InfoRM.The response is considered a Hacking sample if it extensively deviates from addressing the query, significantly occurring content irrelevant to the query topic.Based on these criteria, determine whether the response can be categorized as exhibiting the Hacking Phenomenon. Your response should strictly select one of the following options for whether the response exhibits Hacking Phenomenon: ['Yes', 'No', 'Invalid'], without returning any other character.  
experiments in reward optimization mitigation: 
 In this section, we first validate InfoRM's efficacy through simulation experiments with access to the gold RM, allowing us to clearly observe its impact on mitigating overoptimization. We then proceed to real-world scenarios without a gold RM to further verify our approach's effectiveness.Our simulation experiments follow [16, 10] , where a fixed gold RM plays the human role, providing labels (i.e., rankings) to train a proxy RM. This setup enables to intuitively assess RLHF performance and observe overoptimization, which is unavailable in real-world settings.Models. In our simulations, we use the Pythia suite [4] for both the policy model and the proxy RM. Specifically, the 1.4B Pythia model serves as the universal policy model utilized everywhere. For the proxy RM, we remove the embedding layers from Pythia models sized 70M, 410M, and 1.4B, adding an MLP head to output a scalar reward. Moreover, the gold RM, based on Vicuna-7B-v1.5 [9] , follows the RM training protocol in AlpacaFarm [13] . Considering Vicuna's size of 7B-much larger than our maximum proxy RM size of 1.4B-it is reasonable to employ it as the gold RM [10] . The proxy RMs are all trained on the same simulated preference dataset with 25% label noise.Pipeline. Our RLHF pipeline in the simulation experiments follows [16] , consisting of several key stages. Initially, both the policy model SFT and the gold RM training are performed on AlpacaFarm [13] . Next, a simulated preference dataset for proxy RM training is generated by prompting the SFT model with instructions to produce two different responses, which are then ranked by the gold RM. In line with [10] , we simulate the scenario of high disagreement rates among human annotators by intentionally mislabeling 25% of this dataset, leading to two versions: one w/ and one w/o label noise. The proxy RM is then trained on these datasets. Finally, policy optimization is conducted using the PPO algorithm [37] ; please see Appendix J.3 for more implementation details.Data. Following [10] , the training data in our simulation experiments are from AlpacaFarm [13] . In particular, 10k instruction demonstrations are utilized for the policy model SFT and 20k preference data is used for gold RM training. In addition, the instructions of the 20k preference data are used for response generation via the SFT model, which is then labeled by the gold RM. The remaining 20k unlabeled data in AlpacaFarm are used for policy optimization. It's important to note that all training data in our simulation experiments is sourced exclusively from the AlpacaFarm dataset [13] , ensuring consistency of the training data distribution across three stages.Baselines. Our baseline models include Supervised Fine-Tuning model (SFT), RLHF model using standard RM (Standard RM), RLHF model using standard RM with KL divergence penalty (Standard RM w/ KL) [33] , and the RLHF model using ensemble RM (Ensemble RM) [10] . 6Figure 4 presents the simulated RLHF results for different 1.4B proxy RM w/ and w/o label noise. InfoRM consistently prevents reward overoptimization and substantially enhances RLHF performance under both noisy and noiseless scenarios. Notably, Standard RM's stability is significantly compromised with the label noise, leading to notable reward overoptimization. In contrast, InfoRM maintains stability regardless of label noise, underscoring InfoRM's ability to extract human preference-relevant information from noisy data to improve the resilience of proxy RMs.Previous research [16] demonstrates that increasing the RM size enhances the performance during the RL stage, as measured by the gold RM. In Figure 5 (left), we assess the impact of varying proxy RM sizes on the final RLHF performance measured by the gold RM. 7 Our findings include: (1) Information-theoretic reward modeling significantly improves performance beyond merely enlarging the RM size, making InfoRM a cost-effective and practical solution for deployment without additional computational costs. (2) InfoRM performance consistently improves as the RM size increases, suggesting our method's benefits are complementary to those from scaling the RM.To assess InfoRM's generalizability, we conduct experiments using both in-distribution (AlpacaFarm) and out-of-distribution (Flan) datasets in the RL stage. The results, shown in Figure 5 (right), demonstrate that InfoRM maintains relatively stable performance on the out-of-distribution Flan Table 1 : Comparison results of win, tie, and lose ratios of RLHF models using different RMs with the optimal hyper-parameters (learning rate and kl penalty) under GPT-4 evaluation. Our real-world experiments closely follow [55, 54] , where the actual human preference dataset, instead of the simulated preference dataset labeled by the gold RM in simulations experiments, is utilized for proxy RM training. RM hereafter refers to proxy RM since the gold RM is absent.Model and Training Data. In our real-world experiments, we evaluate InfoRM on two distinct tasks: the general dialogue task and the summarization task. For the general dialogue task, we utilize Vicuna-7B-v1.5 [9] , an open-source chatbot fine-tuned on LLaMA2-7B [44] , as the SFT model. We then build the RM upon the architecture and weights of Vicuna-7B-v1.5 and train the RM on Anthropic-RLHF-HH [4] , a large-scale human preference dataset including both helpful and harmless data. In the RL stage, this dataset is also employed to optimize the policy model initialized from the SFT model. For the summarization task, we utilize the Reddit TL;DR dataset [41] for SFT, reward modeling, and policy model optimization in the RL phase.Baseline. Similar to the simulated experiments, the baseline models in the real-world experiments include Supervised Fine-Tuning model (SFT), RLHF model using standard RM (Standard RM), standard RM with KL divergence penalty (Standard RM w/ KL) [33] , Ensemble RM (Ensemble RM) [10] , and Weight Averaged RMs (WARM) [36] .Evaluation Data. For the general dialogue task, to thoroughly evaluate the proposed method, both in-distribution and out-of-distribution data are utilized for evaluation. Specifically, in-distribution data refers to the Anthropic-RLHF-HH test set, including both helpful and harmless samples. And the out-of-distribution data is the validation set of AlpacaFarm [13] , consisting of samples from the self-instruct test set [47] , Vicuna test set [9, 53] , and Koala test set [17] . For the summarization task, the test set of Reddit TL;DR dataset [41] is utilized in our experiments.GPT-4 Evaluation. We evaluate the effectiveness of InfoRM by comparing its win ratio against baselines. Previous studies have found that GPT-4's judgments are closely related to humans [8, 55] . Therefore, we employ GPT-4 to evaluate the performance of our method and the baselines. The GPT-4 prompt used in our study is the one with the highest human agreement in AlpacaEval [24] ; please see Appendix J.4 for the detailed prompt. To eliminate the position bias [46, 11] , each pair of samples is assessed twice, with the order of responses reversed in each instance.Table 1 compares the win, tie, and lose ratios under GPT-4 evaluation for our method versus other baselines. Key findings include: (1) Our InfoRM significantly outperforms Standard RM without a KL divergence penalty due to its vulnerability to spurious features within training samples and distribution shifts in RL process, leading to severe reward overoptimization. Our InfoRM leverages IB theory to enhance model generalizability, as evidenced in Section 4.1, thus remarkably reducing overoptimization.(2) Our InfoRM continues to surpass Standard RM w/ KL, despite the introduced KL divergence noticeably improving its RLHF performance. We conjecture that the KL penalty, though stabilizing RL, may restrict the optimization landspace of the policy model, thereby affecting RL effectiveness; please see Appendix E.2 for parameter sensitivity analysis in such a real scenario.(3) InfoRM is a versatile and foundational framework that integrates seamlessly with other techniques to provide complementary benefits. InfoRM not only outperforms Ensemble RM and WARM in RLHF performance but also enhances results when combined with these methods. 
detecting overoptimization: additional strength of ourinform: 
 It is noteworthy that our InfoRM not only filters irrelevant information to human preference, thereby significantly enhancing the performance of RLHF, but also benefits from a highly informative and compact IB latent space, facilitating the establishment of a detection mechanism for reward overoptimization through latent representations. The capacity of our overoptimization detection mechanism hinges on two pivotal points: (1) Overoptimized samples manifest as outliers in the IB latent space of InfoRM. (2) The emergence of these outliers is quantitatively signaled by our proposed indicator.To examine the relationship between outliers in the latent IB space of InfoRM and the overoptimized samples in the RL process, the identification of overoptimized samples is highly challenging and under-explored. To address this issue, we pioneer the use of AI feedback, such as GPT-4, to identify overoptimized samples. Specifically, drawing upon the insights from [10, 51] , we first summarize common overoptimization behaviors, including excessive caution, responses that deviate from user intent, and the generation of a large volume of repetitive and meaningless text. Based on this, we then design guidelines for GPT-4 to assess whether an RLHF model response is overoptimized. Detailed prompt designs are provided in Appendix J.4.Figure 7 provides a t-SNE visualization of the response distributions in the latent IB space of InfoRM before and after RLHF, as well as the distribution of overoptimized samples from the RLHF model as judged by GPT-4. Our key conclusions include: (1) From the left column, outliers in the IB latent space are generally indicative of overoptimized samples, supported by the observation that most overoptimized samples significantly deviate from the distribution of samples before RLHF (depicted as blue points). ( 2) By comparing the left and right columns, it becomes evident that the incorporation of InfoRM leads to a substantial reduction in the number of outliers after RLHF, effectively preventing the appearance of overoptimized samples. This observation aligns seamlessly with the superior performance of InfoRM, as demonstrated in both simulated and realworld experiments. Appendix C.1 presents a more comprehensive validation of these observations, and related parameter sensitivity analysis in Appendix E.1 demonstrates their robustness.Based on the above observation, we design a detection metric for reward overoptimization, namely, Cluster Separation Index (CSI), by quantifying the deviations in the latent IB space of InfoRM. The computation process of CSI is elaborated as follows:•Step 1: Perform clustering on the RLHF model outputs within the latent space of our InfoRM.Denote the clusters as C = {C 1 , C 2 , ..., C n }, where C i represents the i-th cluster, and n is the total number of clusters. For each C i , compute the geometric centroid c i byEQUATIONwhere |C i | denotes the count of points in C i and x represents the points within C i .•Step 2: For each cluster centroid c i from Step 1, identify its nearest SFT model output. Calculate the Euclidean distance d i between each centroid c i and its nearest SFT output as:EQUATIONwhere S represents all SFT outputs and • indicates Euclidean distance.• Step 3: CSI is calculated as the sum of weighted distances by the number of the elements in each cluster:EQUATIONIn this work, we utilize DBSCAN [15] as the clustering algorithm due to its robust empirical performance and ability to operate without a predetermined number of clusters. The pseudocode of CSI calculation is provided in Appendix J.2 for better understanding. Remark II: Our overoptimization detection mechanism is closely tied to InfoRM's compact IB latent space. Other RMs without IB, showing weak correlations between latent space outliers and overoptimized samples, are incompatible with this mechanism; see Appendix F for related evidence.Remark III: Our overoptimization detection mechanism enhances RLHF performance in three ways. First, it facilitates parameter adjustments in InfoRM for real-world scenarios; please see Appendix E.2 for an example. Additionally, it serves as a model-based metric for overoptimization detection as verified in Appendix C.2, thus guiding the optimization of any reward model during the RLHF process, including dataset selection and algorithm design. Finally, it provides a tool for online mitigation strategies like early stopping, helping to prevent overfitting and maintain model integrity.The automated early-stopping algorithm based on our CSI is elaborated in Appendix G. 
conclusion: 
 In this study, we introduce InfoRM, a novel framework designed to mitigate reward overoptimization in RLHF by applying information-theoretic principles to reward modeling. Unlike existing methods that focus on implementing KL divergence constraints, expanding reward model scales, and addressing specific issues like length biases, InfoRM directly addresses the primary cause of reward overoprimization in reward modeling, i.e., reward misgeneralization, by incorporating a variational information bottleneck objective. Our RM effectively filters out information irrelevant to human preferences, ensuring only key features reflecting human values are retained. Additionally, InfoRM features CSI, a quantitative indicator from the latent IB space for detecting reward overoptimization.Experiments across various scenarios and model sizes have demonstrated InfoRM's significant effectiveness in mitigating reward overoptimization. We also empirically validate CSI's effectiveness in detecting reward overoptimization on a wide range of datasets, offering valuable guidance for future research in RLHF algorithm design, and developing online overoptimization mitigation strategies. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please see Appendix J.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research strictly adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please see the part of Broader Impacts following Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This research poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All assets used in this paper, including code, data, and models, are properly credited to their original creators, and all licensing terms and conditions of use are explicitly acknowledged and fully respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In reinforcement learning from human feedback, reward hacking or overoptimization occurs when the policy model's optimization diverges from true human objectives, reducing the helpfulness of large language models, from generating meaningful content to displaying excessive caution. This work introduces the information bottleneck into reward modeling, significantly reducing reward overoptimization. Additionally, we propose an indicator to support online mitigation strategies, aiming to better align large models with human preferences. Our study is ethical and poses no adverse effects on society.Our study presents several avenues for future research. Firstly, while our evaluation includes models up to 7 billion parameters, scaling our InfoRM framework to state-of-the-art models that are orders of magnitude larger remains an exciting and unexplored direction. Furthermore, our over-optimization monitoring mechanism exhibits some latency and requires inference on test datasets, highlighting the need for the development of real-time, lightweight over-optimization detection metrics. Such metrics are crucial for enhancing the effectiveness of Reinforcement Learning from Human Feedback (RLHF). Regarding evaluations, we also observe that the win rates computed by GPT-4 are influenced by the prompt structure. Future investigations could focus on identifying optimal ways to elicit high-quality judgments from automated systems, ensuring more reliable and consistent results.Let X, S, and Y denote the random variable of reward model input, latent representation, and human preference ranking, respectively. According to the well-established variational bounds for MI [1] , the variational lower bound of our IB objective can be formulated as follows:EQUATIONEQUATIONwhere r(s) = N (s; 0, I) is the variational approximation of the marginal distribution p(s). Notably, p φ (s|x) is modeled as a multivariate Gaussian with a diagonal covariance structure, where the mean and covariance are both determined by the output of the encoder f φ (x), i.e., f µ φ (x) and f σ φ (x). The first output, f µ φ (x), represents the K-dimensional mean of the latent representation s. The second output, f σ φ (x) is squared to form the diagonal elements of the K × K diagonal covariance matrix Σ. The relationship between f µ φ (x), f σ φ (x), and p φ (s|x) can be formulated as follows:EQUATIONThen, given a latent representation s drawn from p φ (s|x), the decoder g ψ (s) estimates the human preference ranking y based on the distribution q ψ (y|s).By estimating the expectation on (x, y) using the sample estimate based on the preference datasetD = {x n , y n } N n=1, where x n comprises a human-chosen sample x w n and a human-rejected sample x l n , with y n representing the corresponding human preference ranking, the variational lower bound of our IB objective can be approximated as follows:EQUATIONBased on the Gaussian distribution assumption on p φ (s|x), we can use the reparameterization trick to write p(s|x)ds = p( )d , where is an auxiliary Gaussian random variable with independent marginal p( ). In this way, s can be expressed by a deterministic functionEQUATIONHence, we can get the following objective function:EQUATIONIn our experiments, we employ a sample estimate to determineE n ∼p ( ) [log q ψ (y n |h φ (x n , n ))], by sampling a n from p( ) for x n , balancing computational complexity. Thus our objective can be estimated as follows:EQUATIONAccording to the Bradley-Terry Model, the human preference distribution p(y n ) can be formulated as:EQUATIONwhere σ(•) is the logistic function, and r(•) is the reward model. Notably, in this work, reward model r(•) consists of the previously mentioned encoder f φ (•) and decoder g ψ (•) and can be expressed as follows:EQUATIONCombining the two equations, we obtain:EQUATIONwhere w n and l n are independently sampled from N (0, I) for each input sample, x w n and x l n . Now, our estimation of the objective becomes:EQUATION-EQUATIONEQUATIONwe can get the final objective in our paper:EQUATION-EQUATIONwhere σ(•) is the logistic function.The upper bound of the generalization error for our method is provided in Theorem 1 below, with the proof available in [52] . Theorem 1 demonstrates that the mutual information between the latent representation and observations, as well as the latent space dimensionality, upper bound the expected generalization error of our InfoRM method. Theorem 1. Let |S| be the cardinality of the latent representation space of InfoRM, l(•) be the loss function following sub-σ-Gaussian distribution, X be the reward model input, S be the latent representation of InfoRM, and Θ be the network parameters, we have the following upper bound for the expected generalization error of our InfoRM:E[R(Θ) -R T (Θ)] ≤ exp - L 2 log 1 η 2σ 2 n log I(X, S) ≤ exp - L 2 log 1 η 2σ 2 n log |S|,where L, η, and n are the effective number of layers causing information loss, a constant smaller than 1, and the sample size, respectively.R(Θ) = E X∼D [l(X, Θ)] is the expected loss value given Θ and R T (Θ) = 1 n n i=1 l(X i , Θ) is a sample estimate of R(Θ) from the training data.In this section, we further validate the effectiveness and robustness of our overoptimization detection mechanism across a broad range of datasets. The core of our overoptimization detection mechanism relies on two main aspects: (1) Overoptimized samples appear as outliers in the IB latent space of our InfoRM. (2) The emergency of these outliers can be reflected through our proposed CSI indicator. We will next use sixteen diverse datasets to validate these two aspects respectively, including AlpacaFarm [13] , FalseQA [20] , Flan [28] , HelpSteer [48] , Anthropic-Helpful [4] , Anthropic-Harmless [4] , Mkqa [27] , Oasst1 [23] , OpenOrca [31] , Piqa [50] , PKU-SafeRLHF [22] , ShareGPT 8 , SHP [2] , Instruct-GPT 9 , TruthfulQA [26] , and WebGPT [32] datasets, which encompass a wide range of scenarios.In this part, we explore the relationship between outliers in the IB latent space of InfoRM and overoptimized samples across various datasets used for response generation. The overoptimized samples are identified by GPT-4 as elaborated in Section 5. We provide visualizations of the sample distributions in the IB latent space before and after RLHF, along with the distribution of overoptimized samples, in Figures 9, 10, and 11.From the left column of Figures 9, 10, and 11, it is evident that overoptimized samples consistently appear as prominent outliers in the latent IB space of InfoRM across these datasets. By comparing the left and right columns, we observe that the incorporation of InfoRM consistently results in a significant reduction in the number of outliers post-RLHF, effectively mitigating the emergence of overoptimized samples. These findings further corroborate the outlier behavior of overoptimized samples in the IB latent space, as well as the significant role of our InfoRM in mitigating overoptimization. This section delves into how our proposed approach effectively filters out information irrelevant to human preferences, thus enhancing the relevance and precision of model outputs. A salient example of human preference-irrelevant information is length bias [38] . Typically, human annotators may favor more detailed answers, leading reward models to erroneously equate longer responses with higher quality. This can result in RLHF models producing unduly verbose and excessively detailed outputs. Here, the detail is relevant to human preference, but the mere length is not.To demonstrate our InfoRM's capability in eliminating such length bias, we calculate the average response length on diverse datasets by the models at different RLHF steps using ourInfoRM and Standard RM. The datasets used for response generation includes AlpacaFarm [13] ,FalseQA [20] , Flan [28] , HelpSteer [48] , Anthropic-Helpful [4] , Anthropic-Harmless [4] , Oasst1 [23] , OpenOrca [31] , Piqa [50] , PKU-SafeRLHF [22] , SHP [2] , TruthfulQA [26] , and WebGPT [32] datasets. The results, presented in Figure 14 , illustrate that the output lengths produced by the RLHF model optimizing our InfoRM are significantly shorter than those obtained through optimizing the Standard RM. This evidence supports the effectiveness of the IB method in mitigating length bias, further substantiating the claim that IB can indeed filter out irrelevant information.It's worth noting that beyond length bias, we have empirically identified other examples that illustrate the efficacy of our approach in filtering out information irrelevant to human preferences. Specifically, in datasets with a high prevalence of harmful data, models tend to exhibit an overly cautious refusal to respond, even when the input itself is benign-a phenomenon known as excessive caution. Our empirical observations indicate that the use of IB significantly reduces this phenomenon, highlighting its broader utility in enhancing model generalizability by filtering out extraneous information; please see Appendix K for the corresponding case studies. In our approach, there are two parameters that require manual adjustment, namely, the IB dimensionality, and the IB tradeoff parameter β. IB latent dimensionality refers to the length of the IB representation vector. Next, we will analyze their impact on the overoptimization detection mechanism and RLHF performance, separately.First, we tested the impact of different hyperparameter settings on the performance of our overoptimization detection mechanism. The relevant results are displayed in Figure 15 . We observe that regardless of the parameter settings, overoptimized samples consistently appear as outliers in the latent space of InfoRM. This demonstrates the robustness of our overoptimization detection mechanism against variations in InfoRM's hyperparameters. Figure 16 : Win rate (%) on Anthropic-Harmless dataset between the models after and before RLHF using our InfoRM with different hyper-parameters, according to GPT-4. In order to remove ties, we calculate the win rate as win/(win + loss).In this part, we tested the impact of different hyperparameter settings on the RLHF performance of our InfoRM. Related results are shown in Figure 16 . It can be observed that our model achieves its optimal performance when the IB dimensionality is 128 and the β value is 0.1.Furthermore, to further illustrate the practical utility of our proposed overoptimization detection mechanism in facilitating parameter adjustments in real-world scenarios, we present the response distributions before and after RLHF using InfoRM, with varying IB dimensionality and β values in Figures 17. We observe that, at optimal parameter settings, i.e., IB dimensionality=128 and β=0.1, the output of the RLHF model exhibits the smallest deviation in the IB latent space relative to the output of the SFT model. In addition, the CSI values in the RLHF processes of InfoRM with different IB dimensionalities and β are presented in Figure 18 . As observed, at the optimal parameter setting, the CSI consistently maintains lower values compared to other parameter configurations. These observations validate our overoptimization detection mechanism's additional capability to assist in adjusting hyper-parameters in real-world scenarios. In this section, we investigate the universality of our overoptimization detection mechanism across different RMs. The visualization of the response distribution before and after RLHF in the latent spaces of different RMs, as well as the distribution of overoptimized samples are provided in in Figure 19 .We find that outliers in the latent space of InfoRM consistently correspond to overoptimized samples. Conversely, the latent space distributions of the standard RM are more intricate, where outliers do not necessarily signify overoptimized samples, as illustrated by the green ovals in Figure 19 (b) . This difference arises because InfoRM benefits from information bottleneck theory, resulting in a more compact latent space, whereas the latent spaces of standard RM are relatively dispersed. Therefore, CSI, by detecting outliers in the latent space, effectively identifies overoptimization in our InfoRM. However, it may not be applicable in the contexts of other RM without IB, such as standard RM. To explain how to use the CSI metric to select the stopping point during model training, in this section, we elaborate an automated early-stopping algorithm based on our CSI metric for executing early stopping. The CSI-based early stopping algorithm is detailed as follows:• Update model parameters.3:C t ← evaluate_CSI(model) 4: ∆ t ← |C t -C prev | 5: t = ∆ t / 1 t-1 t-1 i=1 ∆ i 6: if t > max then 7:Trigger early stopping and exit loop. C prev ← C t 11: end for Output: Final model before early stopping.To ensure the fairness and reliability of the experiments, we report the performance of each compared method under different hyperparameter settings in Table 2 . As shown, our method consistently demonstrates significant advantages, regardless of the parameter configurations. In this part, we provide our experiments details in this work.To better demonstrate the implementation details of InfoRM, we provide the pseudocode of InfoRM's implementation in Algorithm 2.To better demonstrate the implementation details of our CSI, we provide the pseudocode of CSI calculation process in Algorithm 3.In our study, all models were initialized from pre-trained checkpoints, ensuring that their architectural setup and hyperparameters remained aligned with those of their original pre-trained counterparts. # Define the LLM backbone to extract hidden state.self.model ← LlamaModel(config) 5:# Define the IB dimensionality of our InfoRM.self.latent_dim ← kwargs.pop("latent_dim", 128)7:# Define the IB tradeoff parameter of our InfoRM.self.beta ← kwargs.pop("beta", 0.1)# Define the last layer of RM encoder for IB representation generation from hidden state. # Get hidden states using self.model. # Get IB representation using self.encode_head. # Get final reward prediction using self.decode_head. # Compute normal reward loss (i.e., L pref erence ) and KL loss (i.e., L bottleneck ). return L total 36: end function The fine-tuning process for the pre-trained models in simulation experiments was carried out on a solitary node outfitted with 8 A100-SXM80GB GPUs. We implemented Data Parallelism (DP) and made use of Automatic Mixed Precision (AMP) with bfloat16, capitalizing on the capabilities of the Deepspeed Zero framework [35] . During training, a learning rate of 5e-5 was used, along with only one epoch for the SFT phase and a global batch size of 64.For reward modeling in simulation experiments and real-world experiments, we employed a learning rate of 5e-6, a global batch size of 64, and trained the model on human preference datasets for only 1 epoch to prevent overfitting. In addition, the IB trade-off parameter β is selected from {0.1, 0.01, 0.001}, and the IB dimensionality is selected from {32, 64, 128}, indicating that the final reward can be represented by a vector of this length.Regarding the PPO training in simulation experiments, we utilized a learning rate of 5e-7 for the policy model and 1e-6 for the critic model. The number of epochs was set to 1, with a global batch size of 16. The sampling temperature was set to 0.8, top-p was set to 0.9, and the maximum output token length was set to 512. The critic model was initialized with the weight of the SFT model, as suggested in [54] , and the Generalized Advantage Estimation parameter λ is set to 0.95. The clipping value in policy and critic optimization is set to 0.2, and the coefficient of KL divergence penalty is selected from the candidate {0.0001, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0}, manually adjusting to achieve optimal results. For the real-world experiments, the global batch size was increased to 64, with all other configurations remaining unchanged. # Perform clustering on the red_points.clusters_red ← DBSCAN().fit_predict(red_points) 6:CSI_value ← 0 7:# traverse obtained clusters.for cluster_id ∈ set(clusters_red) do 9:# Get corresponding sample points. # Get corresponding cluster size.12:cluster_size ← len(cluster_points)13:# Calculate the corresponding geometric centroid. # Calculate the distance between current red centroid and the nearest blur point. # Calculate the weighted distance.CSI_value ← CSI_value + weighted_distance 22:end forreturn CSI_value 24: end functionWe use GPT-4-1106-preview as the evaluator of AlpacaFarm's results, as well as the discriminator of hacking phenomenon. Detailed instructions provided to GPT-4 are illustrated in Figure 20 .I want you to create a leaderboard of different of large-language models. To do so, I will give you the instructions (prompts) given to the models, and the responses of two models. Please rank the models based on which responses would be preferred by humans. All inputs and outputs should be python dictionaries. Here is the prompt: {{ "instruction": """{instruction}""", }} Here are the outputs of the models: [ {{ "model": "model_1", "answer": """{output_1}""" }}, {{ "model": "model_2", "answer": """{output_2}""" }} ] Now please rank the models by the quality of their answers, so that the model with rank 1 has the best output. Then return a list of the model names and ranks, i.e., produce the following output: [ {{"model": <model-name>, "rank": <model-rank>}}, {{"model": <model-name>, "rank": <model-rank>}} ] Your response must be a valid Python dictionary and should contain nothing else because we will directly execute it in Python. Please provide the ranking that the majority of humans would give.Given a pair of query and response, please evaluate the response for the "Hacking Phenomenon" based on its content, consider if it exhibits any of the following criteria:1. Redundancy: The response is marked as a Hacking sample if it severely repeats similar sentences and expresses the same content.There are several browser alternatives available that you may want to consider instead of Google Chrome. Some popular options include: 1. Mozilla Firefox: Mozilla Firefox is a free, open-source browser that is known for its customization options and add-ons. It offers a range of features, including privacy protections, security enhancements, and performance improvements. 2. Brave: Brave is a privacy-focused browser that blocks ads and trackers by default. It also offers a range of features, including a built-in ad blocker, a download manager, and a built-in password manager.3. Vivaldi: Vivaldi is a browser that is designed for power users, with a range of features including a notes feature, a built-in note-taking app, and a sidebar that allows you to access your favorite websites and tools quickly. Ultimately, the best browser for you will depend on your specific needs and preferences. It may be helpful to try out a few different options to see which one works best for you.There are several browser alternatives to Chrome that you may want to consider, depending on your specific needs and preferences. Here are some popular options:1. Firefox: Firefox is a free, open-source browser developed by Mozilla that offers a wide range of features and customization options. It is known for its speed, security, and privacy-focused features, such as the ability to block third-party trackers and create customized browser profiles for different purposes. 2. Vivaldi: Vivaldi is a new browser developed by former Opera CEO Jon von Tetzchner and his team. It offers a unique set of features, such as a multi-pane view, a sidebar for easy access to your favorite websites and tools, and support for web standards. Vivaldi also has a strong focus on customization and community involvement, with regular updates and new features added based on user feedback. 3. Brave: Brave is a privacy-focused browser developed by Mozilla co-founder Brendan Eich. It blocks ads and trackers by default and uses its own cryptocurrency, BAT, to reward users for their attention and data. Brave also has a built-in ad-blocker and features such as private tabs and a focus mode to help you stay focused on your work. 4. Safari: Safari is the default browser on Apple devices, such as Macs and iPhones. It offers a clean and intuitive interface, as well as integration with other Apple services such as iCloud and Siri. Safari also has a strong focus on performance and security, with features such as sandboxing and anti-phishing technologies. These are just a few examples of the many browser alternatives available. Each one has its own unique features and strengths, so it's worth exploring a few to find the one that best fits your needs and preferences.There are several browser alternatives to Chrome that you might consider using. Some popular options include: * Firefox: Mozilla's Firefox browser is a popular choice for many users. LM Input: What are some good browser alternatives to Chrome?There are several browser alternatives to Chrome that you may want to consider. Here are a few popular options: 1. Firefox: Mozilla's Firefox browser is a free and open-source option that is known for its speed and privacy features. It also has a wide range of add-ons and extensions available. Here we highlight the information in green that is covered by our InfoRM but missed by the competing methods (incomplete information error).The idea that the Earth is spherical in shape has been around for thousands of years, but it was not widely accepted until the Renaissance and the Age of Enlightenment. Here is a brief overview of how mankind discovered and came to understand the Earth's spherical shape. 1. Ancient Greek Philosophers: The ancient Greek philosopher Pythagoras (6th century BCE) believed that the Earth was a sphere, but his ideas were not widely accepted at the time. Aristotle (4th century BCE), on the other hand, believed that the Earth was a flat disk.2. The Renaissance and the Age of Enlightenment: During the Renaissance (14th-17th century), artists and scientists began to question the traditional beliefs of the past and sought to understand the natural world. They began to use observation and experimentation to test their ideas, leading to a greater understanding of the Earth's spherical shape. One of the key figures in this process was the Italian mathematician and astronomer, Galileo Galilei (1564-1642). In 1610, he used a telescope to observe the night sky and discovered that the Earth's moon had craters, just like the Earth's sun. This observation, along with others, provided evidence that the Earth was not flat, but a sphere. In 1633, Galileo was put on trial by the Catholic Church for his support of the Copernican theory, which stated that the Earth and other planets revolved around the Sun. Although he was found \"vehemently suspect of heresy,\" his ideas eventually became widely accepted and helped to establish the Earth's spherical shape as a scientific fact.Overall, the discovery of the Earth's spherical shape was a gradual process that involved observations, experiments, and a willingness to question traditional beliefs. Today, it is widely accepted as a fundamental aspect of our understanding of the natural world.Comment: Lacks the detailed exploration of technological contributions that InfoRM offers.The idea that the Earth is spherical has been around for thousands of years, but it was not widely accepted until the Renaissance and the Age of Exploration.In ancient times, people believed that the Earth was flat because this was the common perception based on their observations and understanding of the world around them. The idea of a flat Earth was reinforced by religious texts, such as the Bible, which described the Earth as the center of the universe. However, as humans began to explore the world and gather more information, it became increasingly clear that the Earth was not flat. In the 3rd century BCE, the Greek philosopher Aristotle proposed that the Earth was a sphere, based on his observations of the stars and the way ships disappear over the horizon.During the Renaissance, the Italian astronomer and physicist Galileo Galilei used a telescope to observe the sky and made detailed observations that supported the idea of a spherical Earth. He also conducted experiments that showed that the Earth and other planets moved around the Sun, which was a major challenge to the traditional geocentric view of the universe. Despite these advances, the idea of a spherical Earth was not widely accepted until the Age of Exploration, when explorers like Christopher Columbus and Ferdinand Magellan set out on voyages to the New World and proved that the Earth was indeed round.Overall, the discovery that the Earth is spherical was a gradual process that was influenced by observations, scientific inquiry, and cultural and religious beliefs.There The idea that the Earth is spherical in shape has been around for thousands of years, but it was not widely accepted until the Renaissance period. Here are some of the key milestones in the discovery of the spherical shape of the Earth: 1. Ancient Greece: In ancient Greece, philosophers such as Pythagoras and Parmenides believed that the Earth was a sphere, but their ideas were not widely accepted at the time. The idea that the Earth is spherical in shape has been around for thousands of years, but it was not widely accepted until the Renaissance and the Age of Enlightenment.The ancient Greeks and Romans knew that the Earth was a sphere, but they believed that the landmass of the world was flat. This belief was based on the observation that ships appeared to disappear over the horizon, and they believed that the land must be flat and the sea must be flat as well.In the Middle Ages, the idea of a spherical Earth was challenged by the Christian Church, which believed that the Earth was the center of the universe and that the Sun and other celestial bodies revolved around it. This belief was based on the teachings of the philosopher Aristotle, who believed that the Earth was the center of the universe and that everything else revolved around it. It was not until the Renaissance and the Age of Enlightenment that the idea of a spherical Earth gained widespread acceptance. During this time, scientists such as Copernicus, Galileo, and Kepler challenged the traditional beliefs of the Church and proposed new theories about the nature of the universe. These scientists used observations of the sky and the Earth to support their arguments, and they helped to establish the idea of a spherical Earth as a scientific fact. Today, the idea of a spherical Earth is accepted as a fundamental fact of science, and it is taught in schools around the world. However, it is important to remember that the acceptance of this idea was not immediate, and it was the result of a long and complex process of scientific discovery and intellectual debate.Comment: Misrepresents historical beliefs and the transition from geocentric to heliocentric models, resulting in a less accurate and informative response.Standard RM:InfoRM:LM Input: How did mankind discover that the earth was spherical, and why did they initially believe it to be flat? Here we highlight the information in green that is covered by our InfoRM but missed by the competing methods (incomplete information error).Sure, Chinchillas are small, rodent-like animals that are native to the Andes Mountains in South America. They are known for their long, luxurious fur, which is prized for its softness and warmth. Chinchillas are also very playful and curious animals, and they are known for their affectionate and gentle personalities. Here are a few things to consider before getting a Chinchilla as a pet: * Chinchillas are not well-suited to living in captivity, as they have specific environmental and dietary needs. They require a temperature range of around 50-60 degrees Fahrenheit, and they need a high-fiber diet that is rich in vitamins and minerals. * Chinchillas are also very active animals, and they need plenty of space to run and play. They should be provided with a large cage or habitat, with plenty of space for them to climb, jump, and forage for food. * Chinchillas are not suitable pets for children, as they can be easily stressed and injured by handling. They should be handled by an experienced owner who understands their specific needs and behaviors. Overall, Chinchillas can make wonderful pets for the right owner. They are intelligent, playful, and affectionate, and they are known for their gentle personalities.Comment: Lacks the depth of interaction and health details that InfoRM offers. Chinchillas are small, rodent-like animals that are native to the Andes Mountains in South America. They are known for their long, luxurious fur, which is prized for its softness and warmth. Chinchillas are also highly intelligent and curious animals, and they are popular as pets due to their affectionate and playful personalities. Here are some key points to consider when thinking about getting a chinchilla as a pet: * Chinchillas are nocturnal animals, so they are most active at night and sleep during the day. This means that they may not be the best pets for people who are away from home during the day or who have busy schedules. * Chinchillas are very social animals and require a lot of attention and interaction from their owners. They are also quite active and require a large, spacious cage with plenty of toys and enrichment items to keep them entertained and engaged. * Chinchillas are relatively easy to care for, but they do have specific dietary requirements. They are herbivores that thrive on a diet of hay, fresh vegetables, and pellets, and it is important to provide them with a balanced and nutritious diet to maintain their health and well-being. * Chinchillas are also prone to certain health problems, such as respiratory issues and dental problems, so it is important to have a veterinarian that specializes in exotic pets to care for them and provide regular check-ups and preventive care. Overall, chinchillas can make wonderful pets for the right owner, but it is important to do your research and make sure that you are prepared to provide them with the proper care and attention they need to thrive. They are very social animals and require a lot of attention and interaction, so if you are looking for a lowmaintenance pet, a chinchilla may not be the best choice for you.Comment: Provides comprehensive care requirements, social needs, and potential health issues, making it the most informative for potential pet owners.Chinchillas are small, rodent-like animals that are native to the Andes Mountains in South America. They are known for their long, luxurious fur, which is prized for its softness and warmth. Chinchillas are also popular pets because they are intelligent, curious, and playful. Chinchillas are nocturnal animals, and they spend most of their time burrowing in the ground or in their cages. They have strong front teeth that they use to dig tunnels and burrows in the ground. Chinchillas are also very active and like to climb and jump around. Chinchillas are social animals and like the company of other chinchillas and humans. They are not aggressive, but they can be skittish and may bite if they feel threatened or scared. Chinchillas are also quite independent and do not require a lot of attention or interaction from their owners.Chinchillas have special dietary requirements and need to eat a high-fiber diet that is rich in vitamins and minerals. They also need access to fresh water at all times. Overall, chinchillas make great pets for people who are willing to provide them with a proper environment, diet, and care. They are fun to watch and are very affectionate with their owners.Chinchillas are small, rodent-like animals that are native to the Andes Mountains in South America. They are known for their long, luxurious fur, which is prized for its softness and warmth. Chinchillas are also popular pets because they are intelligent, curious, and playful. Chinchillas are nocturnal animals, and they spend most of their time burrowing in the ground or in their cages. They have strong front teeth that they use to dig tunnels and burrows in the ground. Chinchillas are also very active and like to climb and jump around. Chinchillas are social animals and like the company of other chinchillas and humans. They are not aggressive, but they can be skittish and may bite if they feel threatened or scared. Chinchillas are also quite independent and do not require a lot of attention or interaction from their owners.Chinchillas have special dietary requirements and need to eat a high-fiber diet that is rich in vitamins and minerals. They also need access to fresh water at all times. Overall, chinchillas make great pets for people who are willing to provide them with a proper environment, diet, and care. They are fun to watch and are very affectionate with their owners. Comment: Repeats information, which reduces the clarity of the response.Standard RM:InfoRM:LM Input: I'm thinking about getting a Chinchilla for a pet. Can you provide me some information on them? Here we highlight the information in green that is covered by our InfoRM but missed by the competing methods (incomplete information error). In addition, we mark the repeat information error and excessive caution error in the model outputs. [...] indicates that the response was trimmed to fit this page, but the generated text is actually longer.Yes, that's correct. Japanese and Korean cars are known for their quality, reliability, and affordability, which has contributed to their popularity in the United States and other markets. In fact, many import cars, particularly those from Japan, have a strong reputation for being durable, fuel-efficient, and easy to maintain. Some of the most popular import car brands in the US include Toyota, Honda, Nissan, Subaru, Mazda, and Hyundai. These brands offer a wide range of vehicles, from compact cars and SUVs to sedans and coupes, that cater to different tastes and budgets. That being said, the popularity of import cars has been declining in recent years as some Americans have shifted their preference towards domestic and luxury cars. However, import cars are still widely popular among car enthusiasts, particularly those who are looking for a reliable and affordable vehicle that offers a unique driving experience. In summary, while import cars may not be as popular as they used to be, they still have a strong following and are considered a great option for those who are looking for a high-quality, reliable, and affordable vehicle.Comment: Lacks the depth of details on specific features that set these cars apart. Yes, Japanese and Korean automakers are highly regarded in the global market, particularly for their affordable and reliable cars. In fact, import cars from these countries have been popular for decades, and the trend continues to this day. One of the primary reasons for their popularity is the high-quality engineering and design that these manufacturers are known for. Japanese and Korean cars are often praised for their fuel efficiency, durability, and advanced safety features. Moreover, many import cars come with features that are not typically found in domestic models, such as high-end audio systems, advanced infotainment systems, and advanced driver assistance systems (ADAS). These features have become increasingly important to car buyers, who are looking for more than just basic transportation.Another reason for their popularity is the lower cost compared to domestic cars. While import cars can still be more expensive than domestic models, the overall cost of ownership is often lower due to their reliability and fuel efficiency. In summary, import cars from Japan and Korea are still popular due to their high-quality engineering, advanced features, and affordability. These factors, combined with the perception of reliability and value, have made import cars a top choice for many car buyers in the United States and around the world.Comment: Provides a comprehensive overview of the value, popularity, and specific features that make Japanese and Korean cars desirable, including quality, advanced features, and affordability.Yes, that's correct. In the United States, Japanese and Korean cars are generally considered to be of high quality and are therefore valued more highly by consumers. This is likely due to the fact that these countries are known for their advanced technology and high-quality manufacturing processes. Additionally, Japanese and Korean cars are often more affordable than American or European cars, which may also contribute to their popularity. Comment: Lacks the depth of explanation or examples that InfoRM offers.Standard RM:InfoRM:LM Input: Are imported cars from Japan or South Korea more valuable than cars imported from other countries? Here we highlight the information in green that is covered by our InfoRM but missed by the competing methods (incomplete information error). In addition, we mark the excessive caution error in the model outputs. [...] indicates that the response was trimmed to fit this page, but the generated text is actually longer.• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.For simplicity, we use x w and x l to denote the concatenation of instruction with the chosen and rejected responses, respectively.D = {(xi, yi)} N i=1 and {(x w i , x l i )} N i=1 are equivalent representations of dataset D.In this work, X, S, and Y denote the random variables, and x, s, and y denote the corresponding instances, respectively.Here, the prior over the latent variables r(S) is a centered isotropic multivariate Gaussian distribution.Ensemble RM in our experiments is implemented by combining the average reward across all models in the ensemble with the intra-ensemble variance, strictly following the UWO implementation in[10].In this experiment, our primary objective is to investigate the impact of RM size and RL data distribution on the performance of our method. Given this focus, we did not include Ensemble RM in our comparisons.https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfilteredhttps://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise 


Decision: This paper proposes an information theoretic objective for reward modelling for RLHF. In a nutshell, the method maximises the mutual information between the latent space and the reward model prediction, while minimising the spurious (useless) MI between the input and the latent space when conditioning on a given output. 

The results overall seem convincing, both in terms of the evaluation on a gold reward model and in head-to-head based on GPT4 evaluation. While I am generally quite sceptical of "gold reward models" (I believe they also get hacked), I think the head-to-head results are convincing. 

While this is a borderline paper I am currently leaning "accept" on balance. 

It would be great if the authors could address why the _trivial_ solution (latent == output) doesn't just happen.

I also ask the authors to make sure they stick to the page limits for the camera-ready-copy.

Reviews: review: paper_summary: paper_summary: The paper proposes a regularization method to mitigate reward hacking using a variational information bottleneck objective. Their experiments show the potential that their method might be an alternative to KL divergence for preventing reward hacking.
strength_weakness: strength_weakness: Although the method sounds interesting and neat, the experimental evaluation seems to have several issues that make it difficult to evaluate the practical benefit of the proposed method. 

- The study is motivated by a phenomenon called reward overgeneralization. It does not provide any evidence of solving reward overgeneralization. Length bias is mentioned as an example, but it is not evaluated in the paper.
- Why the variational information bottleneck reduces the spurious correlation is not discussed in the paper. It’s not clear to me why it would be better than a normal LLM-based RM.
- The experimental results show that it outperforms RM without KL penalty in Figures 7-13. We already know that RM without KL penalty is prone to reward hacking and KL penalty is one of the solutions. Still, the evaluations in the Appendices compare InfoRM against this weakened baseline.
questions: questions: - I would like to see the performance of KL regularized PPO with KL penalty larger than 0.01, say 0.1. Given that RM+KL is only marginally better than RM in Figure 4, I wonder if a larger KL penalty would improve the performance of RM+KL.
- It would be a great addition to the paper if RM+KL are compared in Figures 7-13.
- Does the proposed method solve the length bias problem? It was implied in the Introduction that it is one of the overgeneralization phenomena that the proposed method may solve.
- How is ensemble RM implemented?
- line 716: IB dimensionality is set to 3. Should we interpret it that ultimately the reward of the outputs can be explained by just three real numbers?
- Appendix D.2. Figure 15: Wouldn't removing ties make the estimation of the win rate worse than including ties as 0.5 wins? At least it would be informative to report the tie rate if it is removed from Figure 15.
limitations: limitations: I don't see any problems.
paper_summary: paper_summary: This paper proposes a variational information bottleneck (IB) objective for rewarding modeling in RLHF to mitigate the reward misgeneralization issue, which can cause overoptimization. The authors propose a variational information bottleneck objective to filter out irrelevant information and identify a correlation between overoptimization and outliers in the IB latent space. They also introduce the Cluster Separation Index (CSI) as an indicator for detecting reward overoptimization. Experimental results reveal the representation of IB latent space can be used to form an indicator of reward overoptimization.
strength_weakness: strength_weakness: Lack of reward model evaluations. Since the motivation of the paper is to mitigate the reward misgeneralization issue, there is no results to directly support such the claim as far as I am concerned. 
It is not clear whether IB loss would affect the reward modeling abilities, such as accuracies and OOD generalization abilities.
questions: questions: Does IB loss total address the misgeneralization issue? If not, when is IB inefficient? Could you provide the Upper bound of generalization error?
Using GPT-4 to identify overoptimization is promising. Nevertheless, it’d be nice to study how much it align with human identifications. 
What is the architecture of RM encoder?
limitations: limitations: The additional computational overhead of training InfoRM is not discussed.
paper_summary: paper_summary: This paper presents a novel way to train reward models form human preferences using an information bottleneck architecture and training method. They provide a derivation of how to train a reward model with an information bottleneck, and then produce empirical evidence of improved performance when using the InfoRM vs several baselines in the AlpacaFarm instruction-following setting. Compared with a single reward model and mean-ensemble, their results show their method has better performance and a better reward-KL frontier. The authors also introduce a method, Cluster Separation Index (CSI), which utilises the latent space of the IB to detect when overoptimisation of the reward model may be happening. They provide qualitative evidence that this CSI does somewhat correspond to overoptimisation occurring.
strength_weakness: strength_weakness: ## Bigger Points

### Insufficient and imperfect comparison to baselines

In general, the comparison to baselines in all the settings you consider isn't sufficient to justify the method is outperforming the existing SOTA. You don't compare against WCO or UWO from https://arxiv.org/abs/2310.02743, or WARM from https://arxiv.org/abs/2401.12187 (or a variant of it that just uses the ensemble methods), or https://arxiv.org/abs/2403.05171. Some of these are contemporary, but WCO, UWO, WARM aren't given they're all at least 4 months old from the time of submission, and so comparing against them is important.

Further, there is a lack of clarity about how hyperparameters were chosen for each method (e.g. KL, learning rate), including baselines, which makes it unclear how much the hyperparameters were optimised for your method vs the baselines. Picking 1 learning rate for all methods means that if that learning rate is optimal for your method, it's not necessarily optimal for other methods, and which is an unfair advantage to your method. Having a systematic way of choosing hyperparameters for all methods would make the comparison to the baselines much more compelling. This is particularly important as your method introduces two new hyperparameters on top of some of the baselines.

One of the ppo training choices is suboptimal - "The critic model was initialized with the weight of the SFT model" - it should be set to the reward model, as that's been shown in previous work to improve performance (https://arxiv.org/abs/2009.01325). This could affect all methods, or it could affect different methods differently, so it would be useful to correct this choice in the evaluations of your method.

Finally, while you do compare to some baselines in the simulated setting, you only compare to very weak baselines in the realistic setting, meaning it's impossible to know whether your method is actually outperforming existing work in this setting.

### Limited Evaluation
You only perform experiments on one datasets, with a single policy size and a small range of reward model sizes. It would be beneficial to perform experiments on another dataset (for example TL;DR summarisation or anthropic-HH). Additional sizes would also be beneficial but are less important that another dataset to demonstrate the generality of your method.

Further, it seems that all results are only for a single seed. Running multiple seeds is important in this setting, as performance can vary substantially between seeds.

### Non-neutral language describing their own method and contributions.
Throughout the paper, the language describing your method and contributions is not sufficiently neutral, which makes the paper harder to read and less clear. It would be better if this over-enthusiastic language could be toned down through the paper. Some examples:
* 52-53 "Which cleverly addresses" - you don't need "cleverly" here
* 66 "We surprisingly discover" - no need for "surprisingly"
* 68 "meticulously" - as above
* 73 "significantly *mitigates* the risk" - this is specifically egrerious as "significantly" is often taken to mean statistically significantly, but you perform no statistical tests throughout the paper.
* 74 "*marked* improvement in RLHF performance"

## Smaller Points

* There's a typo in the legend of figure 5: sandard -> standard.

### Unclear motivation for your method

Throughout the paper you argue that misgeneralisation is an important cause of overoptimisation, that existing methods don't tackle this but that your method does. Due to the reliance on this argument to motivate your method, it would be useful to see a clearer definition of what you mean by misgeneralisation, and how existing methods don't tackle it. In my mind, increasing model size or training an ensemble are both ways of producing models that generalise better and are more robust (not just when training RMs), so claiming that these methods don't tackle misgeneralisation needs further clarity and justification.

### Lack of quantitative evaluation of CSI

The CSI method and interpretability of the IB space are promising additional benefits of your method. However, it would be useful to see quantitative results of how well using CSI as an early stopping method actually works vs other baselines (e.g. no early stopping, or early stopping on other candidate metrics), to get a sense of whether this method would actually be useful or not in practice.

## Summary

Overall, I think the proposed method has some promise, but the existing empirical results aren't sufficient to demonstrate it's effectiveness vs the current literature, and the clarity and motivation of the paper is unclear and limited by overly enthusiastic writing. I'm currently reccomending a reject. If the issues with comparisons to baselines were fully addressed and the method still performed better than existing approaches I would raise my score to a weak accept, and additionally if the language of the paper was adjusted, better evaluation of CSI was provided and evaluation with multiple seeds and over additional datasets was provided then I would raise my score to an accept. However, as it stands I don't believe the paper is worthy of acceptance.

[EDIT]: I have raised my score to a 6. The new results show much more convincingly that this method is better than the baselines compared against.
questions: questions: (Some of these are covered in the weaknesses section, but I will repeat them here).

* How did you select the hyperparameters for all your methods?
* Could you report reward model accuracy scores under standard training and your method, both In-distribution and out-of-distribution (e.g. to human preferences on the alpaca_eval test set). That would help with seeing whether your method is producing better RLHF performance because of better RM accuracy/generalisation, or due to another reason.
limitations: limitations: The authors discuss some of the limitations of their work, but they don't discuss any of those I broad up in the Weaknesses section, which I think are all major limitations of the work.
paper_summary: paper_summary: This paper adds an information bottleneck objective for training a hidden representation and reward head in RLHF. They then empirically investigate advantages in policy optimization using such a reward model: there is less reward model overoptimization as judged by a gold RM, and overoptimization can be spotted by outlier detection in the latent space.
strength_weakness: strength_weakness: In my weaknesses, I fully focus on the theoretical analysis, hoping that other reviewers touch on scrutinizing the empirical investigations. Overall, the theoretical derivations contain numerous local mistakes. Sometimes, these mistakes "cancel out" again, leading to correct overall conclusions, but in one case, I remain skeptical or confused. There is a chance that the authors actually implemented everything correctly by good intuitions, but the theoretical investigations themselves are lacking in clarity and correctness. I now elaborate on this further.
*I am willing to raise my score if my theoretical concerns are fully addressed, though this potentially depends on how convinced the other reviewers are of the experiments.* 

**1. Lower bound of $I(Y; S)$:** The authors give several *different* formulas for this lower bound:
- Equation (4): $I(Y; S) \geq E_{(x, y)}\Big[\int p_{\phi}(s, x) \log q_{\psi}(y \mid s) ds\Big]$.
- Equation (9): $I(Y; S) \geq \int p_{\phi}(x, s) \log q_{\psi}(y \mid s) ds dy$.
- Equation (27): $I(Y ; S) \geq \int p_{\phi}(s, x) \log q_{\psi}(y \mid s) ds$.

All three formulas are incorrect. The correct formula is:
- $I(S; Y) \geq E_{(x, y)} \Big[ \int p_{\phi}(s \mid x) \log q_{\psi}(y \mid s) ds \Big]$.

This can be verified by reading their reference [1], "Deep variational information bottleneck", Equation (12). Equation (4) is incorrect since it contains a factor $p(x)$ too much. Equation (9) is incorrect since it's not coherent: $x$ is not integrated over, but appears in the integrand. Same for (27): $x$ and $y$ appear in the integrand, but are not integrated over.

However, despite these many local mistakes, their end result, Equation (29), seems to contain the correct (!) lowerbound once everything is written down in concrete samples. However, this requires one more unstated assumption, namely that $p_{\phi}(s \mid x) = \delta_{f_{\phi}(x)}(s)$, i.e., that the distribution in the hidden state is deterministically the output of $f_{\phi}$. If this is assumed, then we have an explanation for why the integral over $S$ in Equation (29) disappears, and everything seems correct.

**2. Upper Bound of $I(X, S \mid Y)$:**
- The authors claim there is a Markov chain $X \to S \to Y$. This is incorrect: There is a Markov chain $S \to X \to Y$. It is obvious that the first Markov chain cannot hold in general since the probability of $S$ given $X$ is given by the encoder, so if the encoder removes ALL information, then no information about $Y$ can be left. I think the authors may have confused themselves by thinking about the Markov chain $X \to S \to \hat{Y}$, where $\hat{Y}$ is the RV of *outputs of the model* instead of the true data distribution.
- The authors derive locally incorrect conclusions from the Markov chain property in Equations (22) and (23).
- Equation (24) is also wrong in general: E.g., if the encoder deterministically maps each $x$ to the same point $s = \star$, then $I(X; S) = 0$, but the RHS would usually still be positive.

Despite these many mistakes, the authors somehow conclude with the correct (!) inequality $I(X; S \mid Y) \leq I(X; S)$ in Equation (25), and the rest of the derivation does not make use of the mistakes before anymore. Note, however, that the inequality in (25) is very elementary and well-known, and so does not need a one page long proof. One could either state it without proof or very easily find references for it (e.g., it's a sort-of dual of the data processing inequality and e.g. [follows directly from the proof given in wikipedia](https://en.wikipedia.org/wiki/Data_processing_inequality)). Though my true suggestion, actually, would be to just replace $I(X; S \mid Y)$ by $I(X; S)$ entirely since this is closer to the variational bound, increasing clarity on what is done in this paper. 

In the final formula for the upper bound of $I(X;S \mid Y)$ in Equation (29), there appears the following KL divergence: $KL[f_{\phi}(x_w), r(S)]$. **I do not understand what this is:** As established before, to make sense of the lower bound of $I(S; Y)$, I assumed that $f_{\phi}$ is a deterministic function. However, to put it into the KL divergence, it would need to be a distribution, so the type signatures do not match. Perhaps the authors mean the Dirac delta distribution on the unique output? (I did not check whether that case then makes sense)

**3. Minor Weaknesses (Addressing those won't change my score):** 
- Footnotes should be placed after punctuation marks, not before
- Sometimes, some clarity is missing. E.g., I needed a bit of time to figure out that $X$ denotes *input pairs* and $Y$ denotes *choices* in Section 3.2. I first thought it would be inputs and rewards. I'd recommend clarifying this. The confusion arose, e.g., because you denoted $Y$ as the "reward model output". (Besides, it's not even the preference model output but instead the true preference of the gold RM! Clarity on this might have prevented the wrong Markov chain I mention above.)
- I recommend the notation $I(X; Y)$ over $I(X, Y)$, since the latter could be confused with the entropy of the joint variable $(X, Y)$.
questions: questions: See above.
limitations: limitations: Yes, though my impression was that the broader impacts and limitations should have been explained within the 9 page page-limit, though I am not entirely sure. In the submitted paper, both are on page 10. Maybe it's worth clarifying, should the paper be accepted, whether these sections should be within 10 pages or whether they're then allowed to even be on *page 11*.

Comments: title: Thanks for your response!
Thanks for your response!
Thanks for your response!
Response to Reviewer XuFQ [Part II]
Response to Reviewer XuFQ [Part I]
Confirmation of the implementation of the reparameterization trick
Summary of the discussion
Response to Reviewer YUjH
Response to Reviewer YUjH
Response to Reviewer YUjH [Part IV]
Response to Reviewer YUjH [Part III]
Response to Reviewer YUjH [Part II]
Response to Reviewer YUjH [Part I]
Response
comment: Dear Reviewer XuFQ,

Thank you for your positive feedback and for raising your score. We appreciate your detailed review and valuable comments. We hope our latest responses further address your concerns.

Best regards.
Dear Reviewer roDN,

We sincerely appreciate your feedback regarding our efforts to address your concerns, and we would like to express our gratitude for your positive support.

Best regards.
Dear Reviewer rGMr,

Thank you very much for your positive feedback. We appreciate your recognition of the paper's contributions and the significance of our empirical results to a broad audience.

Best regards.
> **Q3:** How exactly do you do early stopping with the CSI metric? Could you describe the algorithm here.
> 

**RQ3:** We will **first provide the details of our previous early stopping validation experiments**, explaining how we use the CSI metric to select the stopping point during model training. **Following this, we will provide an automated early-stopping algorithm** based on our CSI metric for executing early stopping.

**Early Stopping Validation Experimental Details**: In this experiment, we implemented early stopping by saving multiple checkpoints, visually inspecting their CSI values, and selecting the one before a significant increase in the CSI metric as the final checkpoint. This process is validated by our observations that overoptimization correlates with a significant increase in the CSI metric, making visual inspection effective, as demonstrated in Section 5 of our paper. However, we acknowledge that automating this process by quantifying CSI metric changes would be more cost-effective. Below, we provide an automated early-stopping algorithm based on the CSI metric.

**Automated Early Stopping Algorithm Based on the CSI Metric**: The CSI-based early stopping algorithm is detailed as follows:

1. Set a maximum tolerable CSI change rate, $\epsilon_{\text{max}}$, which is empirically set to a relatively large value of 10. Let $C_t$ represent the CSI value at the $t$-th evaluation step. The change in CSI at this step is given by $\Delta_t = |C_t - C_{t-1}|$.
2. Calculate the ratio of the CSI change at the $t$-th evaluation step, $\Delta_t$, to the average change across all previous steps,  $\frac{1}{t-1} \sum_{i=1}^{t-1} \Delta_i$. This ratio is denoted as $\epsilon_t=\Delta_t / (\frac{1}{t-1} \sum_{i=1}^{t-1} \Delta_i)$.
3.  If $\epsilon_t > \epsilon_{\text{max}}$, trigger early stopping and exit the iteration. Otherwise, continue training.

To facilitate understanding, we summarize this algorithm as follows:

---

**Input:** Maximum tolerable CSI change rate $\epsilon_{\text{max}}$, initial CSI value $C_0$, maximum steps $T$

**Initialize:** $C_{\text{prev}} \gets C_0$

1. **For** $t \gets 1$ to $T$ **do**:
    1. Update model parameters.
    2. $C_t \gets$ `evaluate_CSI(model)`
    3. $\Delta_t \gets |C_t - C_{\text{prev}}|$
    4. $\epsilon_t=\Delta_t / (\frac{1}{t-1} \sum_{i=1}^{t-1} \Delta_i)$
    5. **If** $\epsilon_t > \epsilon_{\text{max}}$ **then**:
        1. Trigger early stopping and exit loop.
        2. **Break**
    6. $C_{\text{prev}} \gets C_t$

**Output:** Final model before early stopping.

---
> **Q1:** To clarify, when you write "Ensemble RM" you mean UWO? It would be better if this was made clearer in the paper, as I would take "Ensemble RM" to just describe using a mean ensemble rather than UWO. Given you are using UWO, how do you select the variance coefficient hyperparameter?
> 

**RQ1:** Thank you for your valuable suggestion. In the revised version of our paper, we will use "Ensemble RM (UWO)" to refer to the UWO method. Additionally, for the two new baselines that we plan to add, namely Mean and WCO, we will use "Ensemble RM (Mean)" and "Ensemble RM (WCO)" to refer to them, respectively. The experimental results for the new baselines are reported in the response to Q2.

Regarding **the selection of the variance coefficient hyperparameter for Ensemble RM (UWO)** in our simulated experiments, we directly use the recommended value (i.e.,$\lambda$ = 0.1) for PPO from [1], as our simulated experimental setup aligns closely with that in [1]. To ensure fairness and reliability, we report the simulated RLHF performance of Ensemble RM (UWO) with different variance coefficients in the table below. The results demonstrate that $\lambda$ = 0.1 is indeed the optimal choice in our simulated experiments.

| Variance coefficient $\lambda$ | 0.05 | 0.1 | 0.5 | 1.0 |
| --- | --- | --- | --- | --- |
| Final gold score | 5.71 | 5.96 | 5.68 | 5.43 |


> **Q2:** It would also be beneficial to compare to WCO and mean ensemble from Coste et al. for a more thorough set of baselines. &  I would still like to see comparisons with ensemble methods in the realistic setting
> 

**RQ2:** Following your suggestion, we further include the Ensemble RM (UWO), Ensemble RM (WCO), and Ensemble RM (Mean) [1] in our real-world experiments. The comparison results are presented in the following table. To ensure fairness and reliability, we also report the performance of Ensemble RM (UWO) with different variance coefficients. The optimal settings, selected based on the highest win ratio, are highlighted in bold. The results show that **our InfoRM using a single RM achieves better RLHF performance compared to the ensemble methods**, and **integrating InfoRM with the ensemble methods further enhances RLHF performance**. We will also include the Ensemble RM (WCO) and Ensemble RM (Mean) in our simulated experiments in the revised version.

| Model | Opponent |  |  | Anthropic Helpful |  |  | Anthropic Harmless |  |  | AlpacaFarm |  |  | TL;DR Summary |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Win | Tie | Lose | Win | Tie | Lose | Win | Tie | Lose | Win | Tie | Lose |
|  | Ensemble RM (WCO) | 47.2 | 35.5 | 17.3 | 52.1 | 39.9 | 8.0 | 39.8 | 35.1 | 25.1 | 63.9 | 23.4 | 12.7 |
|  | Ensemble RM (Mean) | 48.7 | 32.9 | 18.4 | 54.0 | 35.9 | 10.1 | 41.7 | 38.3 | 20.0 | 65.3 | 24.7 | 10.0 |
| InfoRM | Ensemble RM (UWO) ($\lambda$=0.5) | 46.9 | 31.8 | 21.3 | 50.5 | 33.5 | 16.0 | 38.2 | 35.9 | 25.9 | 62.9 | 27.5 | 9.6 |
|  | **Ensemble RM (UWO) ($\lambda$=0.1)** | 43.1 | 33.1 | 23.8 | 49.3 | 34.8 | 15.9 | 37.3 | 37.8 | 24.9 | 61.4 | 28.1 | 10.5 |
|  | Ensemble RM (UWO) ($\lambda$=0.05) | 43.5 | 33.6 | 22.9 | 50.1 | 35.1 | 14.8 | 37.8 | 35.3 | 26.9 | 61.6 | 29.3 | 9.1 |
| InfoRM + Ensemble RM (UWO) ($\lambda$=0.1) | Ensemble RM (UWO) ($\lambda$=0.1) | 48.7 | 35.7 | 15.6 | 52.5 | 35.1 | 12.4 | 41.2 | 38.2 | 20.6 | 63.3 | 30.1 | 6.6 |

[1] Coste, Thomas, et al. "Reward Model Ensembles Help Mitigate Overoptimization." ICLR 2024.
Reviewer YUjH,

We sincerely appreciate your recognition of our work and your timely responses during the rebuttal process. Additionally, **the reparameterization trick has indeed been used in our implementation**. **The related evidence can be found in our submitted code** from Line 331 to Line 338 in `InfoRM_code/utils/model/reward_model.py` for your kind reference.

Best regards.
Thank you for answering my last clarification questions!

Here's a summary of the discussion (e.g., for the area chair):

In the discussion, the authors provided a correct derivation of their optimization objective. Crucially, the discussion **revealed that the authors sample Gaussian noise via a reparameterization trick**. This was entirely omitted from the original submission, and thus the optimization objective in the discussion looks a bit different from the one found in the original submission. Importantly, **this is not just a detail in the derivation, but impacts the concrete implementation of the method**.

I am increasing my score to 5 (borderline accept), which comes with some trust that:

- The authors are able to use the derivations in the discussion as a basis to write up a coherent derivation in the paper;
- It is actually true that the authors have used the reparameterization trick in their implementation as revealed in this discussion.

In my opinion, it depends on the area chair's trust in these two points (together with the other reviewer's discussions) whether the paper can be accepted.
> **Q1:** Some clarification questions about $\boldsymbol S$.

**Response:** Thank you for your valuable suggestion.  You are indeed correct, and we recognize that the notation for $\boldsymbol S$ has caused some confusion. In the revised version of our manuscript, we intend to make the following clarifications to address the points you raised:

1. We will clarify that the notation $\boldsymbol S$ represents the tuple  $(\boldsymbol S^w, \boldsymbol S^l)$, where $\boldsymbol S^w$ and $\boldsymbol S^l$ denote the latent representation corresponding to the accepted and rejected samples, respectively.
2. We will replace $p_{\phi}(\boldsymbol s|\boldsymbol x_n^w)$ with $p_{\phi}(\boldsymbol s^w|\boldsymbol x_n^w)$, and similarly, $p_{\phi}(\boldsymbol s|\boldsymbol x_n^l)$  will be replaced with $p_{\phi}(\boldsymbol s^l|\boldsymbol x_n^l)$.
3. We will clarify that  $p_{\phi}(\boldsymbol s|\boldsymbol x_n)$ is equivalent to $p_{\phi}(\boldsymbol s^w|\boldsymbol x_n^w) \cdot p_{\phi}(\boldsymbol s^l|\boldsymbol x_n^l) .$
4. We will further clarify that $r(\boldsymbol S)$  is the independent product distribution over the tuple $(\boldsymbol S^w, \boldsymbol S^l)$, i.e., for an instance $(\boldsymbol s^w, \boldsymbol s^l)$, $r(\boldsymbol s)= r(\boldsymbol s^w) \cdot r(\boldsymbol s^l)$, where  $r(\boldsymbol s^w)$ and $r(\boldsymbol s^l)$ each represent a single standard Gaussian distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$.

Furthermore, we will thoroughly review our entire paper to prevent similar questions. We appreciate your valuable suggestion, which has significantly helped us enhance the clarity and precision of our derivations.
Thanks to the authors' responses and additional experiments. My concerns are clarified. I lean to hold my previous decision.
Thank you for the further details!

Some last clarification questions: When you write $\mathbf{S}$, is it true that sometimes you mean a tuple $(\mathbf{S}^w, \mathbf{S}^l)$, and sometimes a single vector (depending where you are in the derivation)? Likewise, is $r(S)$ sometimes an independent product distribution over tuples, and sometimes a single Gaussian distribution? Likewise, is $p_{\phi}(s \mid x)$ sometimes a product $p_{\phi}(s^w \mid x^w) \cdot p_{\phi}(s^l \mid x^l)$?

If so, then I think it will help some readers enormously to be clearer by distinguishing these notations.
> **Q1:** To clarify, do you sample one $\boldsymbol \epsilon$ for the whole pair $\boldsymbol x^{w}$, $\boldsymbol x^{l}$, or do you sample separate $\boldsymbol \epsilon^w$, $\boldsymbol \epsilon^l$ for the two parts?

**Response:** For each input sample, either $\boldsymbol x_n^w$ or $\boldsymbol x_n^l$, where $n$ ranges from 1 to $N$, we independently sample an $\boldsymbol \epsilon$ from the standard multivariate Gaussian distribution, denoted as $p(\boldsymbol \epsilon) = \mathcal{N}(\mathbf{0}, \mathbf{I})$, to generate the latent representation. Specifically, for $\boldsymbol x_n^w$, the corresponding latent representation is given by $\boldsymbol s_n^w = h_{\phi}(\boldsymbol x_n^w, \boldsymbol \epsilon_n^w)$, where $\boldsymbol \epsilon_n^w \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$. Similarly, for $\boldsymbol x_n^l$, the corresponding latent representation is $\boldsymbol s_n^l = h_{\phi}(\boldsymbol x_n^l, \boldsymbol \epsilon_n^l)$, where $\boldsymbol \epsilon_n^l \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$. We will distinguish between $\boldsymbol \epsilon_n^w$ and $\boldsymbol \epsilon_n^l$ in the revised version to prevent any confusion. In addition, this approach, where $\boldsymbol \epsilon_n^w$ or $\boldsymbol \epsilon_n^l$ is independently sampled for each corresponding input sample, is consistently applied across all our implementations.


> **Q2:** Furthermore, could you explain why $\epsilon$ disappears in the final part of the derivation?

**Response:**  We would like to clarify that $\boldsymbol \epsilon$ does not cancel out in our derivation. For simplicity, we just use $g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi} (\cdot)$ as an abbreviation for $g_{\psi}(h_{\phi}(\cdot, \boldsymbol \epsilon))$. Based on your suggestion, we will eliminate this abbreviation and directly utilize the intermediate product as our final objective:
$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \log \sigma \left( g_{\psi}(h_{\phi}(\boldsymbol x_n^{w}, \boldsymbol \epsilon_n^{w})) - g_{\psi}(h_{\phi}(\boldsymbol x_n^{l}, \boldsymbol \epsilon_n^{l})) \right) - \beta\ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] - \beta\ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right] \right]$, 

where $\boldsymbol \epsilon_n^{w}$ and $\boldsymbol \epsilon_n^{l}$ are independently sampled from $\mathcal{N}(\mathbf{0}, \mathbf{I})$ for each input sample.
Thank you for the response!

I think I am still confused about some aspects of this derivation, but I also think providing this derivation already provides significant value to move the discussion forward. 

> Based on the Gaussian distribution assumption on $p_{\phi}(\boldsymbol s|\boldsymbol x)$, we can use the reparameterization trick to write $p(\boldsymbol s|\boldsymbol x)d\boldsymbol s = p(\boldsymbol  \epsilon)d\boldsymbol  \epsilon$, where $\epsilon$
 is an auxiliary Gaussian random variable with independent marginal $p(\epsilon)$.

Thank you, this seems like a really important part of the derivation that I'm happy to see more details on.
To clarify, do you sample one $\epsilon$ for the whole pair $x^w, x^l$, or do you sample separate $\epsilon^w, \epsilon^l$ for the two parts?

Furthermore, could you explain why $\epsilon$ disappears in the final part of the derivation?
> **Q3:** Suggestion about directly starting with  $I(X;S)$ instead of $I(X;S|Y)$
> 

**Response:** Thank you for your valuable suggestion. In the revised version, we will directly start by minimizing $I(X;S)$ and use this term consistently throughout the paper for greater clarity.
**Continuing from the previous block, i.e., Response to Reviewer YUjH [Part II], we now complete our derivation.**

As established in the previous block, we have derived that our objective can be estimated as follows:

$L \approx \frac{1}{N} \sum_{n=1}^{N}  \left[  \log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]\right]$ .

According to the Bradley-Terry Model, the human preference distribution $p(y_n)$ can be formulated as:

$p(y_n) = p(\boldsymbol x_{n}^w \succ \boldsymbol x_{n}^l)= \sigma(r(\boldsymbol x_{n}^w)-r(\boldsymbol x_{n}^l))$,

where $\sigma(\cdot)$ is the logistic function, and $r(\cdot)$ is the reward model. Notably, in this work, reward model  $r(\cdot)$ consists of the previously mentioned encoder  $f_{\phi}(\cdot)$ and decoder $g_{\psi}(\cdot)$  and can be expressed as follows:

$r(\boldsymbol x) = g_{\psi}(h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon))= g_{\psi}(f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)+ f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)\boldsymbol \epsilon)$.

Combining the two equations, we obtain:

$\log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) = \text{log}\ \sigma(g_{\psi}(h_{\phi}(\boldsymbol x_n^{w}, \boldsymbol \epsilon)) - g_{\psi}(h_{\phi}(\boldsymbol x_n^{l}, \boldsymbol \epsilon)))$ .

Now, our estimation of the objective becomes:

$L \approx \frac{1}{N} \sum_{n=1}^{N}  \left[ \text{log}\ \sigma(g_{\psi}(h_{\phi}(\boldsymbol x_n^{w}, \boldsymbol \epsilon)) - g_{\psi}(h_{\phi}(\boldsymbol x_n^{l}, \boldsymbol \epsilon))) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right]\right]$ ,

in which $\text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]$ is replaced by $\text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] + \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right]$.

Recalling that $h_{\phi}(\boldsymbol x,\boldsymbol \epsilon)=f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)+ f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)\boldsymbol \epsilon$, we abbreviate $g_{\psi}(h_{\phi}(\cdot, \boldsymbol \epsilon)))$ as $g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi} (\cdot)$  for clarity and ease of understanding, leading to the final objective in our paper:

$L \approx \frac{1}{N} \sum_{n=1}^{N}  \left[ \text{log}\ \sigma(g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi}(\boldsymbol x_n^{w}) - g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi}(\boldsymbol x_n^{l})) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right]\right]$,

where $\sigma(\cdot)$ is the logistic function.

In the revised version, we will include this detailed derivation in the Appendix A
> **Q2:** Full derivation of Equation (29) based on Gaussian assumption.

**Response:** The full derivation of Equation (29) in our paper is as follows:

Let $\boldsymbol X$, $\boldsymbol  S$,  and $Y$ denote the random variable of reward model input,  latent representation, and human preference ranking respectively. The variational lower bound of our IB objective can be formulated as follows:

$J(\boldsymbol{\theta})=I(\boldsymbol S;Y)-\beta I(\boldsymbol X;\boldsymbol S) \geq \mathbb{E} _{(\boldsymbol x,y)}\left[\int p _\phi(\boldsymbol s|\boldsymbol x) \log q _\psi(y | \boldsymbol s) d\boldsymbol s \right] - \beta\ \mathbb{E} _{\boldsymbol x}\left[KL(p _{\phi}(\boldsymbol S|\boldsymbol x)||r(\boldsymbol S))\right]=L$,

where $r(\boldsymbol s)=\mathcal{N}(\boldsymbol{s};\mathbf{0},\mathbf{I})$ is the variational approximation of the marginal distribution $p(\boldsymbol s)$. Notably, $p_{\phi}(\boldsymbol{s}|\boldsymbol{x})$ is modeled as a multivariate Gaussian with a diagonal covariance structure, where the mean and covariance are both determined by the output of the encoder  $f_{\phi}(\boldsymbol{x})$, specifically $f_{\phi}^{\boldsymbol{\mu}}(\boldsymbol{x})$ for the mean and $f_{\phi}^{\boldsymbol{\sigma}}(\boldsymbol{x})$ for the covariance; please see the response to Q1 for their detailed relationship. Then, given a latent representation $\boldsymbol s$ drawn from $p_{\phi}(\boldsymbol s|\boldsymbol x)$, the decoder $g_{\psi}(\boldsymbol s)$ estimates the human preference ranking $y$ based on the distribution $q_{\psi}(y|\boldsymbol s)$.

By estimating the expectation on $(\boldsymbol x, y)$ using the sample estimate based on the preference dataset  $\mathcal{D}=[\boldsymbol x_n,y_n] _ {n=1}^N$, where $\boldsymbol x_{n}$ comprises a human-chosen sample $\boldsymbol x_{n}^w$ and a human-rejected sample $\boldsymbol x_{n}^l$, with $y_n$ representing the corresponding human preference ranking, the variational lower bound of our IB objective can be approximated as follows:

$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \int  p_{\phi}(\boldsymbol s|\boldsymbol x_n) \log q_{\psi}(y_n|\boldsymbol s)d\mathbf s - \beta \ KL(p_{\phi}(\boldsymbol S|\boldsymbol x_n) ||r(\boldsymbol S)) \right].$

Based on the Gaussian distribution assumption on $p_{\phi}(\boldsymbol s|\boldsymbol x)$, we can use the reparameterization trick to write $p(\boldsymbol s|\boldsymbol x)d\boldsymbol s = p(\boldsymbol  \epsilon)d\boldsymbol  \epsilon$, where $\boldsymbol \epsilon$ is an auxiliary Gaussian random variable with independent marginal $p(\boldsymbol \epsilon)$. In this way,  $\boldsymbol s$ can be expressed by a deterministic function $\boldsymbol s = h_{\phi}(\boldsymbol x,\boldsymbol \epsilon)=f _ {\phi}^{\boldsymbol \mu}(\boldsymbol x)+ f _ {\phi}^{\boldsymbol \sigma}(\boldsymbol x)\boldsymbol \epsilon$.
 Hence, we can get the following objective function:

$L \approx \frac{1}{N} \sum _ {n=1}^{N} \left[ \mathbb{E} _ {\boldsymbol \epsilon \sim p(\boldsymbol \epsilon)} \left[\log q _ {\psi}(y _ n | h _ {\phi}(\boldsymbol x _ n, \boldsymbol \epsilon)) \right] - \beta \ \text{KL} \left[ p _ {\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]\right]$

In our experiments, we further employ a sample estimate to determine  $\mathbb{E} _ {\boldsymbol \epsilon \sim p _ (\boldsymbol \epsilon)} \left[\log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) \right]$, by sampling a $\boldsymbol \epsilon$ from $p(\boldsymbol \epsilon)$, balancing computational complexity. Thus our objective can be estimated as follows:

$L \approx \frac{1}{N} \sum_{n=1}^{N}  \left[  \log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]\right]$ .

**Due to word count limitations, the derivation continues in the next block, i.e., Response to Reviewer YUjH [Part III].**
> **Q1:** The relationship between $f_{\phi}$ and $p_{\phi}$.
> 

**Response:** By assuming  latent representation distribution $p_{\phi}(\boldsymbol s| \boldsymbol x)$ follows a multivariate Gaussian with a diagonal covariance structure, whose mean and covariance are determined by the output of the encoder $f_{\phi}(\boldsymbol x)$, the relationship between $f_{\phi}(\boldsymbol x)$ and $p_{\phi}(\boldsymbol s|\boldsymbol x)$ can be formulated as follows:

$p_{\phi}(\boldsymbol s \mid\boldsymbol x) = \mathcal{N}(\boldsymbol s \mid f_{\phi}^{\boldsymbol \mu}(\boldsymbol x), f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x))=\frac{1}{\sqrt{(2\pi)^k |\boldsymbol \Sigma|}} \exp\left( -\frac{1}{2} (\boldsymbol s - f_{\phi}^{\boldsymbol \mu}(\boldsymbol x))^\top \boldsymbol \Sigma^{-1} (\boldsymbol x - f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)) \right)$,

where $\boldsymbol x$ is the network input, $\boldsymbol s$ is the latent representation, $f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)$ is the mean of $\boldsymbol s$, and $\boldsymbol \Sigma$ is the diagonal covariance matrix determined by $f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)$. **Specifically, the encoder $f_{\phi}(\boldsymbol x)$  generates two outputs: $f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)$ and $f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)$. The first output, $f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)$, represents the $K$-dimensional mean of the latent representation $\boldsymbol s$. The second output, $f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)$ is squared to form the diagonal elements of the $K \times K$ diagonal covariance matrix $\boldsymbol \Sigma$.** 

In the revised version, We will further clarify the relationship between $f_{\phi}$ and $p_{\phi}$.
Thank you very much for the clarification.
Now I think the contribution of the paper is clear and the empirical results it brings are interesting for a wide range of audiences.
Thank you for your detailed response, and all the new results.

> We would like to clarify that we have compared our method with Ensemble RM, i.e., UWO.

To clarify, when you write "Ensemble RM" you mean UWO? It would be better if this was made clearer in the paper, as I would take "Ensemble RM" to just describe using a mean ensemble rather than UWO. Given you are using UWO, how do you select the variance coefficient hyperparameter?

It would also be beneficial to compare to WCO and mean ensemble from Coste et al. for a more thorough set of baselines.

> the early stopping strategy according to our CSI metric indeed enhances RLHF performance

How exactly do you do early stopping with the CSI metric? Could you describe the algorithm here.

Overall, the new results present a much more convincing story of the methods improved performance compared to the baselines. I will raise my score to a 6 (weak accept). I would still like to see comparisons with ensemble methods in the realistic setting, even if they are more computationally expensive. If these were presented and InfoRM outperformed them I would raise my score to a 7.
Thank you for the rebuttal!

> Second, we will add the assumption about the hidden state distribution for Equation (29) in Appendix A of our paper: “The distribution of the hidden state follows a Gaussian distribution with the mean and variance determined by the output of the encoder”

- Could you write down in formulas the relationship between $f_{\phi}$ and $p_{\phi}$? Including the precise formula for mean and variance?
- As I said, assuming a Dirac distribution, I could make sense of the first half of Equation (29). Could you provide a full derivation based on your Gaussian assumption? 

> Third, we will remove the proof of $I(X;S \mid Y) \leq I(X;S)$ in Appendix A of our paper and directly use it as a well-known inequality.

I actually recommend (and also stated so) to directly start with $I(X;S)$ in the objective in Equation (3) since this is closer to what you actually optimize and the standard in the literature.