Prototypical Reward Network for Data-Efficient RLHF (2024)

Abstract: The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to scalability issues for LLMs and complex tasks. Our proposed framework Proto-RM leverages prototypical networks to enhance reward models under limited human feedback. By enabling stable and reliable structural learning from fewer samples, Proto-RM significantly enhances LLMs’ adaptability and accuracy in interpreting human preferences. Extensive experiments on various datasets demonstrate that Proto-RM significantly improves the performance of reward models and LLMs in human feedback tasks, achieving comparable and usually better results than traditional methods, while requiring significantly less data. in datalimited scenarios. This research offers a promising direction for enhancing the efficiency of reward models and optimizing the fine-tuning of language models under restricted feedback conditions.

heading: 1 Introduction
text: Reinforcement Learning from Human Feedback (RLHF) effectively integrates meticulous human judgment with the model’s capacity for large-scale data processing (Cortes et al., 2015; Bai et al., 2022a; Stiennon et al., 2020), enhancing language models’ adaptability to human communication styles and preferences (Yuan et al., 2023). By utilizing Reinforcement Learning (RL) over supervised fine-tuning, RLHF captures the complexity of human language, involving emotions, context, and subtle linguistic differences (Ouyang et al., 2022; Wang et al., 2023) thereby offering enhanced adaptability and flexibility in human interactions.
The success of RLHF hinges on the quality of the reward model (Wang et al., 2024; Lee et al., 2023;
*Corresponding Authors
Bai et al., 2022b; Gilardi et al., 2023), which guides the RLHF learning process, ensures its accuracy and efficiency (Ouyang et al., 2022), while preventing deviations from desired outcomes (Paulus et al., 2017). A deficient reward model, however, may learn a complex yet inaccurate error surface, leading the model to favor high-scoring yet erroneous solutions (Chen et al., 2019, 2023; Li, 2017; Yang et al., 2024). This overfitting can result in responses that maximize rewards but diverge from the actual objectives and human preferences (Wang et al., 2021a). Notably, effective RLHF typically requires extensive data (Sun et al., 2023; Zhang et al., 2024a,b).
This Work. To address these challenges, we integrate prototypical networks–instance-based algorithms that learn representative prototypes for similar examples to facilitate tasks like classifications or regression (Snell et al., 2017)–with the reward model for RLHF. Prototypical networks are particularly suitable for few-shot learning as they efficiently extract key features from limited samples for decision-making (Liu et al., 2020). By optimizing the embedding process in the reward model using prototypical networks, we enable the reward model to learn stable and reliable data representation structures with limited sample sizes. This method is especially suitable in enhancing the model’s learning and generalization from human feedback samples, given the limitations of sample quantity and the complexity of human preferences (Bai et al., 2022b).
To enhance the effectiveness of the reward model with limited human feedback data, we propose the Prototypical Reward Model (Proto-RM) that decreases reliance on human feedback without compromising the performance of the reward model. The fundamental principle of the reward model is to assimilate human feedback to evaluate and steer the model outputs to meet human expectations and
13871
standards. Its crucial capability lies in effectively learning and extracting vital parameter information from limited human feedback, thus guiding the model’s behavior. Our approach focus on learning a method that performs well in few-shot scenarios and is suitable for learning from human feedback samples while preserving the reward model’s network structure and enhancing the parameterization capabilities.
Our method consists of three pivotal steps. First, in Sample Encoding and Prototype Initialization, we employ a reward model to encode samples. These encodings serve as the basis for initializing prototypes with a strategically selected subset of the encoded samples. Subsequently, we analyze the relationship between these initialized prototypes and the encodings of other samples. Second, in Prototype Update and Addition, we continuously refine the sample encodings based on their distances to the prototypes. Concurrently, we update the reward model’s parameters by validating the predictions generated through the refined encodings. This iterative refinement ensures that prototypes accurately mirror the attributes of samples, enhancing the learning efficacy through human feedback samples. Finally, Reward Model Fine-tuning employs the refined prototypes and encodings to train the reward model. This training aims to accurately evaluate and guide the outputs of the language model, thereby improving the performance of LLMs during the fine-tuning stage.
Contributions. Our main contributions include:
• We propose Proto-RM, a novel prototypicalnetwork-based method to improve the reward model. This structure facilitates training with fewer human feedback samples without compromising the learning ability of the reward model in scenarios with ample samples.
• We explore a prototypical learning method for human feedback samples, effectively managing human feedback that is difficult to quantify and varies in length.
• We conduct a series of experiments to validate the effectiveness and robustness of our method ProtoRM across different dataset sizes and evaluate the performance of LLM fine-tuned by ProtoRM. Our experiments show that our method has clear advantages and achieves the effectiveness of training with extensive data, even when using limited samples.

heading: 2 Related Work

heading: 2.1 Reinforcement Learning from Human Feedback (RLHF)
text: Large Language Models (LLMs) such as GPT4 (Achiam et al., 2023), Bard (Singh et al., 2023), and LLaMA-2 (Touvron et al., 2023) have demonstrated significant capabilities in understanding human languages and preferences (Zhao et al., 2024; Jin et al., 2024a,b). The efficacy of these models primarily hinges on Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022; Casper et al., 2023), which enables LLMs to iteratively refine their text generation capabilities, thereby producing outputs that accurately reflect human values and preferences (Song et al., 2023; Wang et al., 2021b). RLHF involves three interconnected stages: feedback collection, reward modeling, and policy optimization. Initially, human annotators assess model outputs and provide feedback. This feedback is then utilized in the reward modeling process, where supervised learning is used to train a reward model that replicates human assessments (Lambert et al., 2023; Dong et al., 2019). Finally, during policy optimization, the
model is fine-tuned to produce outputs that receive positive evaluations from the reward model (Zheng et al., 2023). RLHF excels at identifying “good” behaviors than other reward specification or learning methods. However, it faces significant challenges due to the large volume of human feedback data required, which can be costly and resourceintensive (Beeching et al., 2023; Yao et al., 2023b).

heading: 2.2 Prototypical Networks
text: Prototypical Learning is a powerful approach for improving model interpretability and accuracy in few-shot classification scenarios (Liu et al., 2020; Kim et al., 2014). Researchers have extensively enhanced prototypical networks for category learning (Pan et al., 2019; Ding et al., 2020; Ji et al., 2020). The advantages of prototypical networks lie in their simplicity and intuitiveness, enabling rapid adaptation to new samples and categories without the need for extensive data or complex training processes (Fort, 2017; Yao et al., 2023a). While these networks are commonly used in classification problems with distinct category labels, their application is notably absent in the domain of non-quantitative semantic understanding and text comparison.

heading: 3 Problem Formulation
text: The primary objective is to train a reward model capable of training a policy that generates highquality texts, as evaluated by humans, using a constrained set of human-annotated data.
Input. The input of the reward model is a dataset D = { (xi, y + i , z + i , y − i , z − i ) }N i=1
with N examples, where each example consists of a common post xi ∈ X and two corresponding summaries y+, y− ∈ Y. These summaries are distinguished by human preferences: y+ is the preferred (chosen) response with annotation z+i ∈ Z, and y− is the less preferred (rejected) response with annotation z−i ∈ Z, where Z = {chosen, rejected}.
Output. The outputs consist of 2 components:
• Predictive scores s(xi,y+i ) and s(xi,y−i ) for each
example (xi, y+i , y − i ), indicating the model’s assessment of the relative quality of the summaries;
• The reward model fϕ : X×Y → E , where E is the embedding space. fϕ includes an embedding function eϕ and an aligned linear scoring process.

heading: 4 Methodology
text: The primary objective is to train a reward model that predicts which answer {y+i , y−i } is better according to human judgment, given a prompt xi.

heading: 4.1 Reward Model with Prototypical Network
text: Reward Model for RLHF. The reward model evaluates the quality of outputs generated by the language model. The model’s feedback guides finetuning so that the model outputs align with human preferences. Given the input dataset D, the RLHF reward model, parameterized by ϕ, converts text pairs into encodings in the embedding space E :
fϕ(x, y)→ e ∈ E , e = (ex, ey). (1)
Here, e is the representation of the input pair (x, y), ex and ey are the representations of the prompt and answer, respectively.
Prototypical Network. In the prototypical network, a set of prototype vectors pk is categorized into two groups: p+ and p−. The classification of each sample pair’s embedding e(xi,y∗i ), where y∗ ∈ {y+, y−}, is determined by the proportion of these two classes of prototypes within the adjacent prototypes. The embedding e(xi,y∗i ) is updated based on all the prototype vectors in their respective category, with weights assigned according to their importance. The importance of prototype pk is computed using the distance metric d(·, ·):
P(pk|(xi, y∗i )) ∝ exp(−d(e(xi,y∗i ),pk)), (2)
where d(·, ·) is usually taken as squared L2 distance. We then update the embedding for each sample according to its class. For a sample embedding related to the p∗ prototype, we update its embedding e(xi,y∗i ) using all |P| prototypes in class p∗, where |P| is the total number of prototypes of class ‘∗’. The formula for updating the embedding is expressed as:
e(xi,y∗i ) = 1
|P|
|P|∑
k=1
(P(pk|(xi, y∗i )) · pk). (3)
The updated embedding is then transformed into a score within a linear layer.

heading: 4.2 Reward Model with Prototypical Network
text: Prototype Initialization. During the initialization phase, our goal is to properly initialize two
classes of prototypes pk ∈ {p+,p−}. We randomly select n sample pairs and aggregate them according to their labels zi. Specifically, we initialize different prototypes using the sample embeddings labeled “chosen” and “rejected”. This strategy is employed to allow the model to better learn human preferences instead of mere differences in the content of the samples. We process the prompt and answer components of each text bar separately. For embeddings that initialize p0 to be the original prototypes , we perform pairwise sample alignment to ensure uniformity and fairness in compression and computation across positive and negative examples. This alignment method guarantees that the prototypes are updated consistently, reflecting a balanced representation of both prompt and answer components in the embedding space:
exi ← align(exi ,max ∥exi∥), (4)
where align(·) denotes updating the embedding exi to a new vector with the same maximum length as the longest embedding vector among all exi . Elements beyond the original length of exi are padded with zeros. Similarly, we have:
ey∗i ← align(ey∗i ,max ∥ey∗i ∥), (5) e(xi,y∗i ) = (exi , ey ∗ i ). (6)
An initial prototype constructed from n text pairs is defined as p0 = 1n ∑ e(xi,y∗i ), with a length of ∥p0∥ = (max ∥exi∥+max ∥ey∗i ∥), i = 1, 2, . . . n. This ensures that the prototype encapsulates the essential features of both prompt and answer.
We derive an initial set of K prototypes using mean pooling as the aggregate function while keeping the parameters of the reward model ϕ frozen. During initialization, we disable gradient updates of the prototype vectors, ensuring that other model parameters do not affect the initialization process and thus guaranteeing its robustness.
Prototype Update. Our goal is to represent the samples effectively and comprehensively using prototypes. However, a fixed number of prototypes may not suffice for this purpose. Too few prototypes can lose important information, while too many can affect their representativeness and increase computational costs (Snell et al., 2017; Ming et al., 2019; Jin et al., 2023). Therefore, we consider employing Infinite Mixture Prototypes (IMP) (Allen et al., 2019) to automatically generate prototypes during training. This approach allows the model to increase the number of prototypes as needed, based on the distance relationship between the prototypes and the samples. The IMP technique is commonly used in classifying graphical samples, but its application in textual information is relatively less frequent. Prototype methods excel in processing graphical samples with their visual and intuitive features, but text’s abstract and multidimensional characteristics, covering semantics, syntax, and context, complicate their use in textual data. Due to our reliable embedding and alignment of text samples using the reward model, we successfully implement IMP for effective learning from human feedback samples.
After initializing the prototypes, we use the prototypical network to better assimilate new inputs. To enhance the representativeness and diversity of the prototypes, we 1) appropriately generate new prototypes and 2) continually update existing ones.
Generating New Prototypes. We define the set of prototypes as P. To increase the representativeness and diversity of the prototypes, for each sample (xi, y∗i ) ∈ D, if the minimum distance between e(xi,y∗i ) and any prototypes in P exceeds a threshold λ, we create a new prototype based on e(xi,y∗i ). The threshold distance λ is defined as:
λ = 2σ log
( α
(1 + ρσ ) d/2
) , (7)
where σ is the cluster variance learned jointly with ϕ, ρ is the standard deviation for the base distribution from which the cluster means are sampled, and α is a hyperparameter controlling the concentration of clusters in the Chinese Restaurant Process (Wang and Blei, 2009). Our approach can balance between fitting simple data distributions with low capacity and complex distributions with high capacity.
We then compute the distance from each text bar in a text pair to every prototype pk in their class, denoted as d(e(xi,y∗i ),pk). Using the negative of these distances, we calculate the softmax to obtain the probability distribution of sample (xi, yi) belonging to prototype pj . Additionally, during the update of sample embeddings, we incorporate a proportionate dropout of the prototypes, which enhances the model’s ability to generalize and avoid overfitting to specific patterns:
P (pi = pj |(xi, y∗i )) = exp(−d(e(xi,y∗i ),pj))∑⌊ρK⌋
k=1 exp(−d(e(xi,y∗i ),pk)) ,
(8)
where ρ is the dropout ratio, K is the total number of prototypes, and ⌊·⌋ represents the floor function. Instead of random dropout, Proto-RM calculate the cosine similarity of prototypes within the same class and drop prototypes with the highest similarity. This approach ensures that the remaining prototypes are more diverse and thus more representative of the data. The new embedding e′(xi,y∗i ) is derived using the weighted average of pk with respect to the probability distribution:
e′(xi,y∗i ) = K∑
k=1
P (pi = pk|(xi, y∗i )) · pk. (9)
Annotation Prediction. We then evaluate the performance of the model and update it. We predict the annotation zi of the new embedding e′(xi,y∗i )
. The embedding transform into a score s(xi,y∗i ) through a linear layer. By comparing the scores s(xi,y+i ) with s(xi,y−i ), the model annotate the one with the higher score as “chosen”, and the one with the lower score as “rejected”. We evaluate the model’s predictions zi against real human annotations and perform backpropagation accordingly.
Loss and Backpropagation. The final step involves the computation of the overall loss, including reward loss and diversity loss to enhance the model’s performance and reduce the risk of overfitting. Inspired by Stiennon et al. (2020), we define the reward loss Lr as:
Lr =− E(xi,y+i ,y−i )∼Z [log(σ(rϕ(xi, y + i )
− rϕ(xi, y−i )))], (10)
where rϕ(xi, y∗i ) is the scalar output of the reward model for prompt xi and answer y∗i with parameter ϕ, andZ is the collection of human annotations. At the end of training, we normalize the reward model outputs such that the reference text pairs from the dataset achieve a mean score of 0.
For diversity lossLdiv, in order to ensure a sparse distribution among prototypes, we employ a hyperparameter τ to constrain their average in-between Euclidean distances. As model parameters, prototypes are involved in backpropagation through gradient descent, allowing for dynamic refinement. The sparsity constraint is implemented via a diversity loss Ldiv (Ji et al., 2022), which is guided by the average Euclidean distance between prototypes:
ψ = { Euc(Φ)− τ if Euc(Φ) ≥ τ, τ − Euc(Φ) if Euc(Φ) < τ,
(11)
Ldiv = log(ψ + 1). (12) The full objective L linearly combines Lr and Ldiv using a hyperparameter ρd:
L = Lr + ρdLdiv. (13)

heading: 5 Experiments
text: In this section, we first compare the consistency of annotations between Proto-RM and Baseline Reward Model (Baseline RM) with real human feedback on Prompt-Answer text pairs. Subsequently, we contrast the differences in text quality
Algorithm 1 Reward Model with Prototypical Networks
1: Input: D = { (xi, y + i , y − i ), (z + i , z − i ) }N i=1
, where each z+, z− ∈ Z = {chosen, rejected} 2: Output: The predicting score pair S(x, y+, y−) = (s+, s−) and the reward model fϕ 3: Initialize K Prototypes through Prototype Initialization 4: for minibatch Br ∈ D do 5: Perform Prototype Update and Addition and estimate λ according to Eq. 7 6: for (xi, y+i , y − i ) ∈ Br do 7: Converts (xi, y+i , y − i ) into encodings
e(xi,y+i ) and e(xi,y−i )
8: for y∗i ∈ y+, y− do 9: Allign e(xi,y∗i ) according to Eq. 4
10: Calculate di,k = d(e(xi,y∗i ),pk) for pk ∈ p∗, and di,k = +∞ for pk /∈ p∗ 11: Update the embedding according to Eq. 3 12: if min di,k > λ then 13: Create the K + 1-th prototype
pK+1 using e(xi,y∗i ); Increment K by 1
14: end if 15: Compute s(xi,y∗i ) though Annotation Prediction 16: end for 17: end for 18: end for
of LLM outputs after fine-tuning with Proto-RM versus Baseline RM. Following this, we explore the significance of different modules in the learning of the reward model, assessing the effectiveness of our innovative points.

heading: 5.1 Experiment Settings
text: Datasets. We train reward models using three datasets at varying data proportions. The datasets employed are as follows:

heading: 5% 979 1,657 9,692
text: • Webgpt Comparisons (Webgpt) (Nakano et al., 2021) contains pairs of model answers with human preference scores in the WebGPT project.
• Synthetic Instruct GPT-J Pairwise (Pairwise) (Alex et al., 2021) contains human feedback for reward modeling, featuring pairwise summary evaluations and Likert scale quality assessments.
• Summarize from Feedback (Summarize) (Stiennon et al., 2020) contains pairwise summaries with human annotations from the TL;DR dataset.
Implementation Settings. We use GPT-J (Wang and Komatsuzaki, 2021) as the pre-trained LLM. Our experiment applies a batch size of 8 and initialize each prototype using n = 2 examples. The sequence length is set to 550. We fix the value of α at 0.1 and the initial value of ρ at 5. We use the AdamW optimizer (Zhuang et al., 2022) and search the best learning rate within the range of [1e− 6, 1e− 5]. Other hyperparameters are set to their default values as in Allen et al. (2019). We use the trlX framework (Havrilla et al., 2023) for model implementation. All experiments are conducted for a maximum of 5 epochs with early stopping on a server with NVIDIA Tesla A100 GPU (80GB memory).

heading: 5.2 Comparison with Baseline Reward Model
text: To compare the performance of Baseline RM and Proto-RM, we train and test both reward models on three datasets by different ratios. From Table 2 we can see that, across the different data proportions on the three datasets, Proto-RM consistently surpasses Baseline-RM. On the Webgpt dataset, there is an accuracy improvement ranging from 1.48% to 2.15%; on the Pairwise dataset, the improvement spans from 0.48% to 0.59%, with Proto-RM nearly achieving perfect accuracy; and on the Summarize dataset, especially at the 20% data proportion, Proto-RM exhibits the most significant accuracy gain of 1.26%.
The line graphs Figure 3 reinforce the table’s data, showcasing that the Proto-RM model maintains a higher accuracy across epochs compared to the Baseline-RM for the 5%, 10%, and 20% of the Summarize dataset. Proto-RM not only starts at a higher accuracy but also demonstrates less variability and ends with a higher accuracy, indicating a more robust model.

heading: 5.3 RLHF Performance
text: To ensure the consistency and integrity of the evaluation, we employ GPT-4 (OpenAI, 2024) to assess all outputs from GPT-J (6B) (Wang and Komatsuzaki, 2021) across four dimensions. This methodology aligns with various studies highlighting the capabilities of LLMs to produce high-quality text evaluation that align with or surpass human judgments (Gilardi et al., 2023; Alizadeh et al., 2023; Bai et al., 2022b). Our scoring criteria encompass factual accuracy, text relevance, information completeness, and clarity of expression, are uniformly applied. Each dimension receives a score up to 10, with increments of 0.5. The overall score is derived from the average of four metrics.
Accuracy (Acc): Assesses whether the content of the answer or summary accurately reflects the information and intention of the original prompt.
Relevance (Rel): Checks whether the answer or summary is closely related to the original prompt.
Completeness (Comp): Evaluates whether the provided information is comprehensive, covering all key points and details in the prompt.
Expression (Expr): Considers whether the language expression of the answer or summary is clear and understandable.
The results in Figure 4 indicates that the LLM fine-tuned with Proto-RM outperforms the Baseline across all four aspects, showing an increase
from 0.4/10 to 0.54/10 in overall score, which is significantly higher than the Baseline. Moreover, it demonstrates a clear advantage in both Accuracy and Expression, with the highest scores reaching 0.76/10 and 0.82/10 respectively. Table 4 demonstrates the differences in the output text quality of GPT-J with no fine-tuning, fine-tuned using Baseline-RM, and fine-tuned using Proto-RM. The discrepancies highlighted also validate the efficacy of our improved reward model.

heading: 5.4 Ablation Study
text: Study of IMP. (Allen et al., 2019) We explore and compare the effects of using different numbers of prototypes with various methods for setting the prototype quantities. Specifically, we examine the outcomes of setting the prototype numbers to twice and thrice the default amount and the outcomes of gradually increasing the number of prototypes from the default to double and triple using the IMP method. Figure 6 illustrates that adopting the IMP method for prototype numbers yields better results in both accuracy and stability compared to fixed prototype numbers. The lines representing IMP methods (both IMP-Double and IMP-Triple) show higher accuracy over the epochs. Additionally, the IMP lines demonstrate a smoother progression with less fluctuation, suggesting greater stability in model performance across epochs.
Effects of Dropout. As shown in Figure 5, we find that employing a Dropout method, which proportionally drops out a part of the prototypes during the sample embedding updates, yields better results. Specifically, as the line chart illustrates, adopting a Dropout method significantly outperforms the approach of not using Dropout in terms of accuracy.
Among the Dropout approaches, the method utilizing Cosine Similarity Dropout achieves higher accuracy compared to Random Dropout and exhibits greater stability. This underscores the effectiveness of using Cosine Similarity Dropout.
Toxicity. We conduct experiments on different proportions of a toxicity dataset from (Bai et al., 2022a) over a single epoch. As shown in Table 3, our Proto-RM method outperforms the Baseline at random data proportions of 5%, 10%, and 20%.

heading: 6 Conclusion
text: Our research demonstrates the efficacy of prototypical networks in refining RLHF, especially in scenarios with limited human feedback. The enhanced reward model shows a marked improvement in aligning LLM outputs with human preferences, as evidenced by our experimental results. future works can explore the application of our method to more diverse and extensive datasets to further validate its effectiveness and adaptability.
Limitations There are a few limitations to the current framework. First, the Proto-RM performs best for open-domain pairwise human preference tasks that focus on alignment with human judgments. It remains unknown if Proto-RM yields substantial improvements in scenarios involving non=pairwise tasks or datasets that require less complex decision-making processes.
Second, we focus our study on the Englishlanguage human preference data. This source mainly represents the English-language NLP study. Our study does not include all the necessary research from other nature languages.
We examine how often NLP researchers cite older work by analyzing factors such as the mean age of citations. Our findings indicate a link between these factors and the frequency of citations. However, these links do not prove causation. More studies are needed to understand the reasons behind the citations of older papers.
Ethics Statement The Proto-RM framework can enhance the effectiveness and data efficiency of the RLHF process for LLM. However, as the framework uses pre-trained GPT-J as support and GPT-4 as an evaluator, it may inherit the ethical concerns associated with GPT-J and GPT-4, such as responding to harmful queries or exhibiting biased behaviors.
Acknowledgements This work was supported by Public Computing Cloud, Renmin University of China, Beijing Outstanding Young Scientist Program (BJJWZYJH012019100020098), Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China, and fund for building world-class universities (disciplines) of Renmin University of China. This work was partially done at Beijing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China, Public Policy and Decision-making Research Lab of RUC, Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education, MOE.

heading: A Supplementary Statistics and Plots
text: A.1 Data Efficiency In this section, we present a comparative analysis of our Proto-RM method against the Baseline results obtained from the standard RLHF on the full-size dataset. Notably, Proto-RM demonstrates equivalent effectiveness using only 20% of the data that the Baseline requires 100% to achieve. This significant reduction in data usage without compromising performance underlines the data efficiency of our method.
Here is a detailed table of the performance across various datasets:
For Webgpt dataset the Proto-RM method slightly improves the performance, with a performance increase from 60.17% to 60.56%. This improvement, while modest, indicates that our approach can enhance model effectiveness even with reduced data input.
For Pairwise dataset, the Baseline achieves a near-perfect score of 99.89%. The Proto-RM’s performance closely follows at 99.84%, which is remarkable given that it only uses a fraction of the data.
For Summarize dataset, the performance of our Proto-RM method is comparably high at 68.72%, closely trailing the Baseline’s 68.88%. Although there is a minor decrease, it falls within the margin of error and showcases that our approach can maintain high levels of effectiveness in summarizing tasks.
In all instances, the Proto-RM method validates our claim of data efficiency. Our method achieves competitive or even better results than the Baseline with significantly less data. This result shows that Proto-RM is suitable for improving data efficiency when the data cost is high or the data is limited.
A.2 Human Evaluations To avoid any form of tuning specifically tailored to GPT-4’s evaluation patterns, we present results in Table 7 and Table 8 evaluated by humans, which
may slightly vary from GPT-4’s, but also validate the effectiveness of our method in aligning with human preferences.
A.3 Time Complexity Here we analyze the time cost of Proto-RM. For all samples within each minibatch, the prototype updates introduce a time complexity of O(n), where n is the number of samples. For each sample, ProtoRM needs to identify the closest k prototypes out of allm prototypes to compute the output of the prototype network. Although the IMP method gradually increases the number of prototypes, we have set a cap on the maximum number of prototypes that can be added. This cap is defined as α times the initial number of prototypes, where α is a fixed constant. Here, computing the output of the prototype network for each sample incurs a time complexity of O(km). Thus, the introduction of prototype networks results in a time complexity of O(nkm).
This analysis indicates that including prototypical networks does not significantly elevate the overall complexity. Additionally, we provide an analysis of the time taken for training and testing on a dataset of 8,552 samples to support our assertion that Proto-RM does not considerably increase computational costs in Table 6. We believe that the added complexity is manageable and justified by the performance improvements, especially considering the high data costs in such scenarios.

heading: Future Work
text: future works can explore the application of our method to more diverse and extensive datasets to further validate its effectiveness and adaptability. Limitations There are a few limitations to the current framework. First, the Proto-RM performs best for open-domain pairwise human preference tasks that focus on alignment with human judgments. It remains unknown if Proto-RM yields substantial improvements in scenarios involving non=pairwise tasks or datasets that require less complex decision-making processes. Second, we focus our study on the Englishlanguage human preference data. This source mainly represents the English-language NLP study. Our study does not include all the necessary research from other nature languages. We examine how often NLP researchers cite older work by analyzing factors such as the mean age of citations. Our findings indicate a link between these factors and the frequency of citations. However, these links do not prove causation. More studies are needed to understand the reasons behind the citations of older papers