Distributional LLM-as-a-Judge
Authors: Luyu Chen
Zeyu Zhang
Haoran Tan
Quanyu Dai
Hao Yang
Zhenhua Dong
Xu Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
LLM-as-a-Judge
Distributional Alignment
Human Evaluation
Adversarial Training
Primary area: evaluation

Abstract: LLMs have emerged as powerful evaluators in the LLM-as-a-Judge paradigm, offering significant efficiency and flexibility compared to human judgments. However, previous methods primarily rely on single-point evaluations, overlooking the inherent diversity and uncertainty in human evaluations. This approach leads to information loss and decreases the reliability of evaluations. To address this limitation, we propose a novel training framework that explicitly aligns the LLM-generated judgment distribution with human evaluation distributions. Specifically, we propose a distributional alignment objective based on KL divergence, combined with an auxiliary cross-entropy regularization to stabilize the training process. Furthermore, due to limited human annotations, empirical human distributions are merely noisy estimates of the true underlying distribution. We therefore incorporate adversarial training to ensure a robust alignment with this true distribution, rather than overfitting to its imperfect approximation. Extensive experiments across various LLM backbones and evaluation tasks demonstrate that our framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods, with superior alignment quality, strong robustness, and competitive evaluation accuracy.

Distributional LLM-as-a-Judge
Luyu Chen Zeyu Zhang'", Haoran Tan'*, Quanyu Dai?,
Hao Yang"* Zhenhua Dong", Xu Chen Gaoling School of Artificial Intelligence, Renmin University of China Huaawii Noah's Ark Lab (luyu. chen, xu. chen) @ruc chen)@ruc.
edu. cn
Abstract
LLMs have emerged as powerful evaluators in the LLM-as-a-Judgo paradigm, offering significant efficiency and flexibility compared human judgments. How- ever, previous methods primarily rely on single-point evaluations, overlooking the inherent diversity and uncertainty human evaluations. This approach leads to information loss and decreases the reliability of evaluations. To address this limitation, we propose novel training framework that explicitly aligns the LLM- generated judgment distribution with human evaluation distributions. Specifically, we propose a distributional alignment objective based on KI. divergence, combined with an auxiliary cross-entropy regularization to stabilize the training process. Furthermore, due to limited human annotations, empirical human distributions are merely noisy estimates of the true underlying distribution. We therefore incorpo- rate adversarial training to ensure a robust alignment with this true distribution, rather than overfitting its imperfect approximation. Extensive experiments across various LLM backbones and evaluation tasks demonstrate that our framework significantly outperforms existing closed-source LLMs and conventional single- point alignment methods, with superior alignment quality, strong robustness, and competitive evaluation accuracy.
Introduction
In recent years, large language models (LLMs) have demonstrated remarkable progress across various tasks, such as natural language understanding [2, reasoning [3,5), and evaluation 66] 77. One of their most significant applications is for automatic judgment, which employs LLMs to evaluate specific targets based predefined criteria instructions [ [ This I.LM-as-a-Judye paradigm offers significant advantages in efficiency and flexibility due to its capability to efficiently handle large-scale data and adapt diverse evaluation tasks. Therefore, using L.LMs judges has emerged as promising alternative to conventional human evaluations [10].
Most previous works adopt single-point judgment with LLMs, which just outputs single result for each sample 10133) Although this paradigm is straightforward, it overlooks the inherent diversity of human evaluations. In real-world scenarios, human evaluations are rarely deterministic. Instead, they follow a distribution that encodes valuable signals like the level consensus and controversy. 15). Therefore, replacing this distributional human evaluations with single-point LLM judgment may cause information loss [15], which limits the comprehensiveness and reliability of evaluations. This limitation is particularly critical in high-stakes domains like medical diagnosis policy-making. where reliance on a single prediction inherently risky and unreliable [16].
In order to empower LLM judgment with the diversity and uncertainty of human evaluations, an intuitive approach to generate judgment distribution based on LLMs. Although LLMs can
Corresponding author:
Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
Output Text
"Ground Truth"
Input
Tasks:
LLM
Single-point Alignment
B
For this task, will be shown
Judge
two conversations:
<Conversation Av
Annotators
.Conversation B>
Output Distribution
Evaluation Distribution
Evaluste the two conversations
and choose one of the following:
(b)
Choices:
LLM Judge
Distribution Alignment
better
Conversation better
B
A
B
Figure 1: Comparison between single-point alignment and distribution alignment. (a) Single-point Alignment: In this method, LLMs are trained to generate outputs that exactly match the desired text. (b) Distribution Alignment: By using this approach, the models are trained produce judgment distributions that align with the human evaluation distributions.
inherently provide probability distributions over output tokens, previous studies have shown that they often overconfident and skewed towards a few options [17 Besides, most current LLM training approaches focus on single-poin alignment, aiming to maximize the probability of generating specific correct or desired output [8] [9] illustrated in Figure This focus inherently limits their ability to capture the diversity and uncertainty present in human evaluations, hindering effective distribution alignment. Therefore, is necessary to design an explicit distributional alignment framework to align LLMs' output with human evaluation distributions, as illustrated in Figure (ib).
To address the above challenges, design a novel framework that explicitly aligns the output distributions of LLMs with human evaluation distributions. Specifically, we propose a distributional alignment objective that leverages the Kullback-Leibler (KL) divergence (20] minimize the discrepancy between the model's predicted distribution and the empirical distribution derived from human annotations. Besides, we introduce a hybrid loss function that combines the primary KL divergence objective with an auxiliary cross-entrope loss to improve the training stability. It combines the distributional advantages of KL divergence and the stability of single-point alignment. To further mitigate the risk of overfitting caused by limited human annotations, we propose an adversarial training strategy to improve model robustness. Specifically, we apply the worst-case perturbation to the empirical distributions during optimization, encouraging the model to align with any plausible distribution within the bounded perturbation set. Our major contributions are presented follows:
Explicit Distribution Alignment Framework. We propose a novel framework to explicitly align the distribution of LLM judgment with human evaluation distributions, thereby effectively capturing the uncertainty and diversity inherent in human evaluations.
Robust Distribution Alignment Methodology. By introducing an adversarial optimization strategy that leverages distribution perturbations during training, we significantly enhance the fidelity and robustness of model alignment with real human evaluation distributions.
Extensive Experimental Validations. Experiments across diverse LLM backbones and evalua- tion tasks demonstrate that our approach consistently surpasses existing closed-soure source LLMs and substantially outperforms conventional single-point alignment methods in multiple aspects.
2
Related Work
2.1
L.LM-as-a-Judge
LLMs are increasingly used as automated evaluators (i.e., LLM-as-a-Judge) [TE 21 22) due to their efficiency, scalability, and generalization capabilities [8] 9. Previous works typically utilize LLMs to produce a sigle-point deterministic evaluation, such as binary consistency judgments 21 and Likert-scale ratings (22]. However, these approaches neglect the inherent variability observed in human evaluations, where humans often present diverse opinions, resulting in evaluation distributions [4]. Collapsing this diversity into a single decision overlooks valuable information [[5] such as disagreement, uncertainty, and subjectivity. To address this limitation, we generate probability distributions from I.IMs for evaluations. We propose an explicit alignment method to better match the distributions generated by LLMs with the actual distributions provided by human annotators.

---

Prior work, such as [23] for the NLI task, has also advocated for learning from full human judgment distributions instead of single labels. Our approach is distinguished by two primary innovations. First, we introduce novel adversarial training mechanism on the label distribution itself. This mechanism designed mitigate the annotation noise stemming from limited data, a known limitation that prior work [23] had not mechanistically solved. Second, we validate our framework's s effectiveness in the contemporary LLM-as-a-Judge paradigm, extending its application to modern evaluation tasks like quality evaluation and preference understanding.
2.2 Distributional Reward Models
To model diverse human preferences, distributional reward models in Reinforcement Learning from Human Feedback (RLHF) [9] aim to output a distribution over reward values rather than a single scalar reward (64. Existing research in this area typically employs methods that model preference score distributions using mean and variance [24 25 or adopting quantile regression techniques to achieve finer-granne preference modeling 26). These approaches often infer reward distributions from human preferences indirectly and necessitate architectural modifications that may decrease the general capabilities of LLMs (27.. Different from previous studies, proposed approach directly leverages the explicit distributions derived from human evaluations, while it can also preserve the inherent language generation capabilities of LLMs without architectural changes.
2.3 Adversarial Training
Adversarial training can enhance model robustness by exposing models to worst-case perturbations in training phase, which has been widely adopted in various fields, such computer vision (28 (29] and natural language processing [30,3]] 31 J. often formulated as a min-max optimization problem with two adversarial stages. Specifically, the maximization stage identifies the worst-case perturbation, and the minimization stage updates the model parameters to minimize loss under these perturbations [32] This iterative procedure enables the model to learn more robust and reliable decision boundaries. Common optimization algorithms employed adversarial training include the Fast Gradient Sign Method (FGSM) (33) and Projected Gradient Descent (PGD) [4 both of which have shown strong stability and effectiveness. In this study, we adopt adversarial training to enhance the robustness of LLMs, in order better align model predictions with human evaluation distributions.
Preliminary
Consider dataset D, where each sample x E D is annotated independently by N human annotators. Each annotator assigns labels from discrete set of categories C = {1, C}. We define the empirical distribution of human judgments (i.e., human evaluation distribution) for given sample as the vector p(r) e C whose i-th component is given by:
1 N
(x(x)
= I = ViEC,
N
3=1
where yj denotes the label from the j-th annotator for sample and T(.)) is the indicator function.
Correspondingly, let denote the parameters of the LLM. For an input sample æ, the model outputs normalized probability distribution over the C categories. This distribution is represented by the vector (z()), where each component qo.3 (x) indicates the predicted probability for category i. Formally, the probability vector (1) is defined
qo (x) E {q E R
4 1 O, ViEC, Σ
9:
In classical evaluation settings B5 36,, a single deterministic reference label r(z) E (0, 10 is often used, typically defined by selecting the most frequent human annotation:
1, if i arg max (x( (x(x)=
&
0, otherwise.
The primary objective of our method to optimize the model parameters so that the predicted judgement distribution (z(x) closely aligns with the human judgment distribution p(r). Our proposed

---

(a)
(b)
Step 1
Step 2
LLM Output
LLM Output
Empirical Distribution
True Undertylng Distribution
LLM Backbone
Perturbation Domain
Probability
Adversarial Learning
Hybrid Loss
Perturbation
Human Empirical
Human Worst-case
Human Annotations
Label
A
Figure 2: Overview of our proposed framework. (a) Training framework: We generate adversarial
perturbations of the empirical human distribution and optimize the hybrid loss. (b) Motivation: Illustrates the relationship between the empirical, perturbed, and underlying distributions. Robust alignment mitigates the deviation problem in the empirical human distribution.
explicit distributional alignment framework enables the model to better capture nuanced human judgments, therehy resulting in more informative and representative evaluations.
4
Methodology
4.1
Overview
To overcome the limitations of j uncertainty in human evaluations, we propose a novel training framework that explicitly aligns model-generated probability distributions with real-world human evaluation distributions. Given an input, we extract logits corresponding to the judgment token to obtain the predicted distribution. Our training involves two main steps, demonstrated Figure a). First of all, we generate worst-case perturbation around the empirical distribution to enhance robustness. Then, we compute the hybrid loss between the model prediction and the perturbed distribution, and update the model parameters accordingly. This approach mitigates the inherent limitation of empirical human distributions, which serve only as imperfect estimates of the true underlying distributions, as illustrated Figure(2J)) By aligning model predictions with all plausible distributions within the perturbation set, our method promotes more robust and faithful distributional alignment.
and better reflect the inherent diversity and
4.2 Human Distribution Aligument via Hybrid Loss
To achieve effective alignment between the model's output distribution and the human judgment distribution, we propose hybrid loss function. This loss function combines KL divergence for distribution alignment with an auxiliary cross-entropy objective for training stability. First, to explicitly encourage distributional alignment, we introduce the KL divergence loss:
1
LKT. (0)
Σ DKL. (p(x) (x)),
D
RED
where D || denotes the KI divergence. This objective promotes fine-grained alignment between the model's predicted distribution and the human evaluation distribution. Second, to improve training stability and guide learning with more direct supervision, also include the cross-entropy loss as an auxiliary regularizer:
1 Σ CF(qg(2), CCE
(5)
D
ZED
where r(x) the reference label for sample x, determined by majority-vote among the human annotators. This hybrid design mirrors the philosophy of knowledge distillation [37). where student models are trained using both soft targets (through KI. divergence from a teacher model) and hard labels (via cross-entropy with ground truth). This approach leverages the rich informational content

---

provided by the teacher's outputs and the direct guidance of true labels, improving both learning fidelity and convergence stability. Embracing this principle, our hybrid loss function blends these two objectives via weighting factor E [0. 1]:
Lyyrii (0) = a (8) (1 - (1- (().
(6)
This hybrid approach ensures stable training using cross-entropy while also achieving nuanced distributional alignment through KL divergence. As result, it effectively captures both consensus and diversity in human annotations.
4.3 Robust Alignment via Adversarial Training
In practice, due to the limited number of human annotations, we only have empirical approximations of the human judgment distribution, as illustrated in Figure2[b) Directly aligning the model output qo (2r) with these empirical approximations p(z) results in the model overfitting to sampling noise or artifacts, reducing the robustness of alignment with the true underlying distribution.
To address this challenge, we introduce adversarial training into our distribution alignment framework. Specifically, we define a perturbation set & around the empirical annotation distribution p(*) and identify the worst-case perturbed distribution p' [ar) within this set. Aligning our model with this worst- case distribution ensures robustness against any plausible perturbation within E, thereby improving alignment with the human judgment distribution. This transformation converts objective into min-max optimization problem as follows:
8¹ = arg min max [a (p'(('x)) qo + (1- (1-a) a) CE(co(z),r())))
p' (x))E
This min-max formulation is structurally similar adversarial training methods like TRADES [8]] However, the two frameworks are conceptually distinct in several key aspects. TRADES perturbs model inputs for attack robustness, with the KI. term serving as an auxiliary regularizer for output smoothness. In contrast, we perturb the target label distribution, making KI. divergence the primary objective for achieving more robust and faithful alignment with human judgments.
This adversarial training process consists of two alternating steps:
1. Adversarial Distribution Generation: For fixed model parameter @ and each sample in the batch, find the worst-case distribution p' (xx within the perturbation set that maximizes the loss. 2. Model Update: Update model parameters 0 to minimize the adversarially perturbed loss.
Through this adversarial training procedure, we explicitly model the worst-case scenarios of the true underlying human judgment distribution, thereby ensuring robust and stable alignment. By accounting for potential annotation noise and sampling artifacts, the model becomes less sensitive to empirical inaccuracies, thus enhancing its generalization performance and practical applicability.
4.4 Implementing Adversarial Training via Projected Gradient Descent
To solve the inner maximization equation in Equation 7 we adopt Projected Gradient Descent (PGD) to iteratively search for the worst-case perturbation within a constrained space.. Specifically, we seek an adversarial distribution p' o'(xr) that maximizes the KL divergence from the model prediction qo (z), while remaining close to the original human distribution p(x) and preserving the properties of a valid probability distribution.
We define the feasible perturbation set È as the intersection of two convex sets as E = A n B.- (p(x)), where A = [p' E RC E 1, p/ 2 0} is the C-dimensional probability simplex, and B,- (p(x)) an &2 ball of radius e centered at the original human distribution p(x). Based on the feasible perturbation set, we design the optimization procedure as follows. First of all,
we initialize the unperturbed distribution as p (0) -.r).. Then, we conduct the gradient ascent by updating the current iterate in the direction of the gradient of the KL. divergence as:
y- (1+1) pp" ( + V pco
[DLL [o(2))
(8)
where y denotes the step size controlling the gradient ascent magnitude. After that, we project the updated distribution back onto the feasible set & with the equation:
(c+1) (t+))
arg min ||p' 1 (y+1)
p'eg
5

---

This projection step is a convex Quadratically Constrained Quadratic Program (QCQP) [39, as it minimizes a convex quadratic objective over the intersection of two convex sets: the simplex and the E2 ball. Notably, the intersection & is guaranteed to be non-empty since the original distribution p(r) E by definition. Consequently, this projection problem well-posed and can be efficiently solved using off-the-shelf convex optimization solvers such as CVXPY [40].
Experiments
5.1 Experiment Setup
Datasets. We evaluate our framework using representative datasets [15] from three fundamental LLM-as-a-Judge applications: dataset labeling, quality evaluation, and pairwise preference prediction.
Dataset Labeling (SNLI 4T//NLLI (22).. We use the classic NIJ benchmarks to represent the dataset laheling task, where the goal is determine the logical relationship (entailment, neutral, contradiction) between two sentences. Each instance annotated by five distinct raters, providing the necessary label distribution. We randomly sample an equal number of instances from MNLI (10,000 each) to maintain a comparable data scale with SNLI.
Quality Evaluation (SummEval D. To evaluate performance on text quality assessment, we use the SummEval dataset. This benchmark contains machine-generacee summaries of news articles from the CNN/DailyMan corpus. For each summary, quality ratings are provided by a group of experts and crowdworkers on 1-5 Likert scale across four dimensions (fluency, coherence, consistency, and relevance), forming a rich distributional signal of perceived quality. We treat cach dimension as an independent evaluation instance.
Pairwise Preference Prediction (MT-Bench [43). For the task of understanding human prefer- ences, we use the MT-Bench dataset. For each dialogue, preferences between two model responses (A vs. are collected from multiple human reviewers. This yields a preference distribution (A is better, B is better, or Tie) for each comparison, directly reflecting the consensus and disagreement in human choices.
All datasets are split into training and test sets at an 8:2 ratio in our experiments. To facilitate the reproduction, we present the detailed prompts that are used all the tasks in Appendix
Baselines. We compare our proposed approach against two baseline methods. (1) Raw Model: We directly evaluate the pretrained LLMs without any task speciffc fine-tuning. This baseline aims to measure the inherent alignment between pretrained models and human judgment distributions, reflecting the model's 's original capability approximate human judgment without explicit training or adjustment. (2) Single-point A Alignment: We adopt the traditional supervised fine-tuning strat- egy [19), using only the most frequent human annotation label as the supervision target. This baseline evaluates the effectiveness of conventional single-point alignment methods.
Models. Our study evaluates both open-source (Qwen2.5-7B [44, LLaMA3.1-8D 45 and closed- source (GPT-4o, GPT-4o-mini) language models. Closed-source models, recognized for their strong performance, are utilized without additional tuning, whereas the chosen open-source models are tested both with and without further training.
Training Details. We fine-tune all selected open-source models using Low-Rank Adaptation (LoRA) [46) with a uniform hyperparameter configuration to ensure fair comparison. Specifi- cally, we employ the Adam W [47] optimizer with learning rate of 5 x 10-5 and train each model for 2 epochs. To enhance model robustness, we incorporate adversarial training, setting the pertur- bation step size to 0.05 and performing 5 gradient ascent steps per training iteration. Additionally, we conduct a hyperparameter search for two critical parameters: the weight parameter a, chosen from the set {0, 0.2, 0.4, 0.6, 0.8, 1.0), and the perturbation radius parameter selected from the set (0.0, 0.05, 0.1,0.15 0.2, 0.25). We conduct all experiments on one NVIDIA A100-40G GPU.
Evaluation Metries. We employ two metrics to measure the alignment between model-predicee and human-annotated distributions. KL Divergence is our primary metric, while Accuracy serves as a complementary measure:
(1) KI. Divergence: As our primary measure of success, this metric directly quantifies the discrepancy between the model's predicted distribution (æ(x) and the human distribution p(x). Lower KL

---

Table 1: Main results comparing raw models, single-point alignment, and our distribution alignment method across four datasets. KL indicates KL divergence, and Acc denotes top-1 accuracy. Results for tne-ttmed models (Single-point and Distribution) are averaged over 5 runs. The indicates statistically significant improvement over the single-point baseline (p- < 0.05).
SNLI
MNLI
Summeval
MT-Bench
Model
Method
KL4
Acc
KL4
Acct
KLL
Acct
KLJ
Acc?
GPT-44-mini
Raw model
2.13
87.0%
1.88
84.9%
5.23
25.6%
5.63
62.3%
GPT-4o
Raw model
1.75
85,5%
1.16
84.2%
2.82
35.2%
2.48
68.5%
Raw model
2.08
83.1%
1.77
83.5%
4.94
22.7%
3.36
62.0%
Qwen2.5
Single-point
0.60
92.7%
0.64
89.7%
0.73
45.6%
0.82
64.0%
Distrihution (Ours)
0.23'
93.3%
0.23*
89.8%
0.53"
45.9%
0.68
65,4% %
Raw model
0,90
64.9%
0.67
70.5%
3.60
29.5%
1.58
53.4%
LLaMA3.1
Single-point
0.69
92.4%
0.67
89.6%
0.67
45.7%
0.81
62.1
Distribution (Ours)
0.28
92.4%
0.24
90.0% %
0.51
47.3%
0.74
62.8%
divergence indicates closer alignment:
p(r) qa (x)
(l.(px))lls(t)) = Σ P(x) log
(10)
(2) Accuracy: We include Accuracy as secondary metric for two practical reasons. First, it serves as a valuable indicator of our model's ability to capture the majority consensus human judgments. More critically, it provides a bridge for comparison with prior work that relies solely on this traditional standard. It measures whether the model's most probable predicted label aligns with the most frequent label in the human judgment distribution:
Accuracy
Σ ZED
arg max (x)( arg max Pj (t()
(11)
D
JEC
Here, [D] denotes the total number of test samples, (x) represents the model-predicted probability for category i, and pj (Ir) denotes the human judgment distribution for category j.
Extracting Model Predictions. We extract model predictions by retrieving logits corresponding to potential judgment labels (e.g.. "entailment", "neutral", "contradiction" for NLI tasks, or 1-5 for Likert-scale ratings). These logits are converted into probabilities via softmax normalization. To handle variations in tokenization, the probabilities of synonymous tokens are aggregated into a standard label. For example, the probabilities for tokens like "contra" or "contradict" assigned to the canonical "contradiction" label. To maintain consistency across experiments, we limit the extraction to the top-5 logits because OpenAl restricts the number of logits returned. Besides, the logits beyond the fifth highest are generally negligible, with values often falling below 1e-6.
summed and
5.2 Overall Performance
We evaluate the effectiveness of our distribution alignment method across four benchmark datasets, including SNLI, MNLI, Summeval, and MT-Bench. The primary experimental results are summarized in Table Detailed results for all fine-tuned models, including mean and standard deviation, provided in Appendix Our findings highlight three key aspects as follows:
(1) Necessity of Distribution Alignment. Without specific alignment training, both open-source and closed-source models demonstrate substantial divergence between their predictions and human judgment, with KL divergence typically exceeding 2.0. This indicates that current models inherently produce judgment distributions that are misaligned with human evaluations, highlighting the necessity for additional training of distribution alignment.
(2) Superiority of Our Proposed Method. Compared to conventional single-point alignment methods, our approach consistently achieves better distribution alignment across different datasets and I.IM backbones. significantly reduces KI divergence while maintaining accuracy, demonstrating its generalization ability and effectiveness in aligning model outputs with human-labed labeled distributions.
7

---

Table 2: Ablation study of our proposed method. We analyze the contribution of adversarial training (Adv), KI. divergence loss (KL), and cross-entropy loss (CE) on MNLI and Summeval datasets.
Components
Qwen2.5
LLaMA3.1
Method
MNLI
Summeval
MNLI
Summeval
Adv
KL
CE
KLJ
Acct
KL!
Acct
KLJ
Acct
KLJ
Acct
Raw Model
1.77
83.5%
4.94
22.7%
0.67
70.5%
3.60
29.5%
Single-point
0.64
89.7%
0.73
45.6%
0.67
89.6%
0.67
45.7%
Ours (Full)
v
0.23
89.8% %
0.53
45.9%
0.24
90.0%
0.51
47.3%
Ours w/o Adv
v 0.25
89.0%
0.64
46.6%
0.32
89.6%
0.62
45.9%
Ours w/o KL
0.78
88.4%
0.75
45.8%
0.65
89.2%
0.65
45.4%
Ours w/o CE
0.23
89.0%
0.54
46.0%
0.33
88.7%
0.58
48.0%
(3) Correlation between Model Capability and Alignment Performance. We observe positive correlation between model's inherent capability and its alignment performance. More capable models, such as GPT-4o, not only achieve higher accuracy but also yield predicted distributions closer to human annotations than weaker models like GPT-4o-mini and Qwen2.5. This suggests that stronger models naturally produce judgments distributions more consistent with human evaluations.
In conclusion, our method demonstrates superior performance in distribution alignment compared to raw models and conventional single-point alignment approaches. It reduces KL divergence while maintaining accuracy across multiple datasets and various LLM backbones, presenting the effectiveness of our method in aligning model outputs with human judgment distributions.
5.3 Ablation Study
To better understand the contribution of each component in our method, conduct ablation study, whose results are summarized in T We observe that removing any single component consistently degrades alignment performance, resulting in increased KL divergence. This indicates that all three components complement each other and collectively enhance distributional alignment.
According to the results, find that KL divergence loss is the most crucial. Removing it leads to significant increase in KL divergence, which confirms its essential role in human judgment alignment by penalizing deviations from the target distribution. Besides, adversarial training also contributes to improving the performance. By introducing perturbations during training, the model can align better with human distributions even under worst-case distributional shifts, thereby improving robustness and generalization. In addition, removing cross-entropy loss results in only a slight increase in KI. divergence. Although its effect distributional alignment is limited, extensive experiments in Section 5.4 demonstrate that a small amount of auxiliary CE loss can stabilize training.
5.4 Impact of sappee-ssrameters
We further perform experiments how the weighting parameter and perturbation radius affect model alignment performance. Using Qwen2.5 as the base model, we evaluate its alignment perfor- mance across all four datasets. Lower KL divergence indicates better alignment between the model prediction and the human judgment distribution. The results are presented in Figure
Impact of Weighting Parameter a. The weighting parameter balances the KI. divergence and CE losses, thereby affecting the performance of aligmment. We observe that increasing a from 0 to approximately 0.8 consistently enhances the alignment performance across all datasets. However, removing the CE term entirely (a=1.0) leads to a noticeable performance decline. This suggests that a small portion of the CE loss is crucial for stabilizing the training process. This instability is especially acute on MT-Bench. We hypothesize this stems from the task's 's high difficulty and subjectivity, which creates more complex target distribution. For such challenging distributions, pure KL divergence objective can become unstable.
Impact of Perturbation Radius We further explore the influence of the perturbation radius in our method. Each line in Figure corresponds to specific value of perturbation radius, and we use the blue line (e 0) to represent training without adversarial perturbations. The results demonstrate that

---

SNLS
MNLI
0.70
0.75
KL. Divergence
KL. D...verggece
0.60
0.60
0.45
0.40
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.6
0.8
1.0
Alpha
Alpha (#)
SummEval
MTBench
0.96
Llle1100 -2-11-2-44
0.75
KL. Divergence
K1. Divergence
0.70
0.88
0.65
0.80
0.60
0.55
0.64
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Alpha (a)
Alpha (a)
8 0.0
0.05
0.1
0.15
0.2
0.25
Figure 3: Effect of weighting parameter and perturbation radius KL divergence across four datasets. Lower values indicate better alignment between model predictions and human distributions.
the method without adversarial training commonly performs the worst. It indicates that adversarial training can enhance the model's generalization capability, thereby improving alignment performance. As the perturbation parameter increases, the alignment performance exhibits general improvement. However, the performance gains gradually diminish with increasing perturbation magnitudes, and this trend is particularly evident the MNLI dataset.
These results have verified our earlier statements: KL divergence serves as the primary mechanism for alignment, while incorporating a minor component of cross-entropy loss can further improve the training stability. It further underscores the importance of combining both components. Besides, the integration of moderate adversarial perturbations further boosts alignment performance by increasing the model's robustness against the shifts in real-world human evaluation distributions.
5.5 Robustness Analysis
To further analyze the robustness of our method, conduct extensive experiments by adding random perturbations to the target label distributions in the test set. Specifically, our random perturbations ranged from 0.00 to 0.25. The experiments performed on all four datasets based on Qwen2.5, and we use the parameerss identified in Section 5.4 with a = 0.8 and 0.25.
As shown in Table 3 our full method consistently achieves the lowest KL divergence across all perturbation levels and datasets, outperforming both the single-point alignment baseline and the variant without adversarial training. These results suggest that incorporating adversarial training enables the model to effectively align with all plausible distributions within the perturbation set, thereby improving robustness and fidelity in distributional alignment.
Conclusion
In this paper, we propose a distribution alignment framework that explicitly aligns the model outputs with the human evaluation distribution, aiming for more nuanced evaluation results. Specifically, we employ KL divergence the main objective to minimize the discrepancy between model predictions and target distributions. Furthermore, we introduce a hybrid loss function, incorporating an auxiliary cross-entropy loss stabilize training. Finally, adversarial training utilized further enhance alignment performance by increasing the model's robustness against distributional shifts. Experiments demonstrate that our distribution alignment method outperforms existing single-point alignment approaches and exhibits strong generalization and robustness across different models and datasets.

---

Table 3: The robustness analysis of our distribution alignment method under varying label perturbation
levels (6). The KL divergences are reported across different datasets and models, with lower KL divergence indicating better performance in distribution alignment.
KL Divergence at different perturbation levels (0)
Dataset
Method
8 = 0.00
å = 0.05
= 0.10
å = 0.15
6 6= 0.20
= 0.25
Single-point
0.715
0.717
0.708
0.692
0.720
0.709
SNLI
Ours w/o Adv
0.334
0.336
0.333
0.327
0.344
0.346
Ours (Full)
0.324
0.325
0.323
0.317
0.335
0.336
Single-point
0.742
0.744
0.743
0.745
0.753
0.757
MNLI
Ours w/o Adv
0.271
0.272
0.272
0.276
0.283
0.293
Ours (Full)
0.243
0.245
0.245
0.251
0.256
0.264
Single-point
0.743
0.748
0.752
0.778
0.809
0.836
Summeval
Ours w/o Adv
0.639
0.643
0.649
0.669
0.707
0.735
Ours (Full)
0.525
0.529
0.539
0.565
0.588
0.612
Single-point
0.833
0,833
0.837
0.832
0,836
0.848
MT-Bench
Ours w/o Adv
0.831
0.830
0.834
0.829
0.832
0.846
Ours (Full)
0.675
0.676
0.678
0.675
0.677
0.689
Limitations
Although our approach can effectively align model outputs with human judgment distributions, it exhibits two notable limitations. First, the model's explainability is limited, as the generated explanations only correspond to single sampled judgment rather than interpreting the entire predicted distribution. Second, suitable training datasets remain scarce due high human annotation costs. Most existing datasets contain only single annotation instance. Therefore, improving model alignment across diverse tasks requires constructing more datasets with richer human evaluation data. In future work, we will further improve the explainability and efficiency of our proposed method.
Acknowledgments and Disclosure of Funding
This work is supported in part by National Natural Science Foundation of China (No. 62422215 and No. 62472427), Major Innovation Planning Interdisciplinary Platform for the "DoubleFirst Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world-class universities (disciplines) of Renmin University of China, the Outstanding Innovative Talents Cultivation Funded Programs 2024 of Renmin University of China, and Huawei Innovation Research Programs. We gratefully acknowledge the support from Mindspore CANN(Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
References
[1] Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv preprint arXiv:2303. 2...22223..2333. 2023. URL httt:://atxxii http://axxvvvrgr.////// htse/bbs/23000 18223
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya
https:://vee h//ww wwndsmore.ec.mmmmmmmm
10

---

Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[3] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS 22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
[4] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23, Red Hook, NY,
USA, 2023, Curran Associates Inc.
[5] Aske Plaat, Annie Wong, Suzan Verberne, Joost Brockens, Niki van Stein, and Thomas Back. Reasoning with large language models, survey, 2024. URL https://arxiv org/ /bs/
2407.
[6] Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. Evaluating and improving tool-augmented computation-intensiive math reasoning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
[7] Lianmin Zheng, Wei Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Ducheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as---jude with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY. USA, 2023. Curran Associates Inc.
[8] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: A comprehensive survey on Ilm-based evaluation methods, 2024. URL 055799
[9] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang. Wen Gao. Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. Uttpttss://arr org/ abs/
[10] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: less is more for alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
[11] Seonghyeon Ye, Doyoung Kim, Hyeonghin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. In ICLR 2024. International Conference on Learning Representations (ICLR), 2024.
[12] Lianghui Zhu, Xinggang Wang. and Xinlong Wang. Judgelm: tune- large language models are scalable judges, 2025. URL htttps://aaxii.....// org/ /bs/ 17631
[13] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking ilms with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025.
[14] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157): 1 1124-1131, 1131, 1974.
[15] Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi, Ling Liu, Sravan Babu Bodapati, and Dan Roth. Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM--a--a-judge. In The Thirteenth International Conference on Learning Representations, 2025. URL //tttrei//peeeeviee http:///ppppaeeeeeeeeeeeeerrrrrp
11

---

[16] Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quantifi- cation in machine-assisted medical decision making, Nature Machine Intelligence. 1(1):20-23, 2019.
[17] Esin DURMUS, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfeel-Doodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling, 2024.
[18] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[19] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 27:77730-27744, 2022.
[20] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86 1951.
[21] Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt as factual inconsistency evaluator for text summarization, 2023. URL
2303. 15621
[22] Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang. and Xiaojun Wan. Human-like summarization evaluation with chatgpt, 2023. URL htttss//aaxii: org/ arggabs/ abaaabaaaaaaaa /204. 02554
[23] Xiang Zhou, Yixin Nie, and Mohit Bansal. Distributed NLI: Learning to predict human opinion distributions for language reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings the Association for Computational Linguistics: ACL 2022, pages 972-987, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. http:////2222..nnnss-eeee 79/
JRLhttpss htttt::://aalaanhoologgee org/2022..cco. findings acl
[24] Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Mende Distributional preference learning: Understanding and accounting for hidden context in RLHF. In The Twelfth Inter- national Conference on Learning Representations, 2024. URL https::/ https://openreview net/ 1
[25] Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu. Aligning crowd feedback via distributional preference reward modeling. In ICML 2024 Workshop on Models of Human Feedback for Al Aligument, 2024.
[26] Nicolai Dorka. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409. 10164, 2024.
[27] Victor Wang, Michael JQ Zhang, and Eunsol Choi. Improving llm-as-a-judge inference with the judgment distribution. arXiv preprint arXiv:2503.0306 2025.
[28] Asif Hanif, Muzammal Naseer, Salman Khan, Mubarak Shah, and Fahad Shahbaz Khan. Fre- quency domain adversarial training for robust volumetric medical segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 457-467. Springer, 2023.
[29] Fu Wang, Zeyu Fu, Yanghao Zhang, and Wenjie Ruan. Self-adaptive adversarial training for robust medical segmentation. In International Conference on Medical Image Computing and Computer- Assistee Intervention, pages 725-735. Springer, 2023.
[30] Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. In International Conference on Learning Representations, 2017.
[31] Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 00.18653/v1/D177
URL https: /aattp:://aclanthhloog anthology www.ee/orgDDD71121111
12

---

[32] Zhuang Qian, Kaizhu Huang, Qiu-Feng Wang, and Xu- Yao Zhang. A survey of robust adversar- ial training in pattern recognition: Fundamental, theory, and methodologies. Pattem Recognition. 101:108889, 2022. ISSN 0031-3203. doi:
URI.
[33] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6577 42.6222, 2014.
[34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In International Conference on Learning Representations, 2018
[35] Alexander R Fabbri, Wojciech Kryšcińskk, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 2021.
[36] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511-2522, Sin- gapore, December 2023, Association for Computational Linguistics. doi: 10. 18653/61/22223
emnlp-main.15 1533.
URL URL hatp:s:///lllaahhoogo.o org/2023. 2023. emnlp-main. 153/
[37] G Hinton. Distilling the knowledge in neural network. In Deep Learning and Representation Learning Workshop in Conjunction with NIPS, 2014.
(38) Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 9131-9143, Online, November 2020. Association for Computational Linguistics. doi: 18.186 1/20000emmlp-mmiin.755.
[39] C. Albers, F. Critchley, and J. C. Gower. Quadratic minimisation problems in statistics. Multivar. Anal., 102(():698-77))) -13, March 2011. ISSN 0047-259X. doi: 10.v0166j.mmva2200.1111 018. httpsttss://///// /ttttp:://do..org//1// 10. 1016/ j.v... 12.
[40] Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddeed modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
[41] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.11853/11/D15-1000 URL www.aanoooooooooooooooo-ooo..- org/D15-1075/
[42] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10.18653/v1/N18-- 1101. URL httpps //aclanthology.oom .rg.e...orgoooo... org/N18-1//1111 N18-1101/
[43] Lianmin Zheng, Win Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chathot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
[44] An Yang. Baosong Yang, and Beichen Zhang et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[45] Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri al. The llama herd of models,
2024. URL https: httptt:t/////tiii/ ://arxiv...v. org/ abs/ /20b 21783,
13

---

[46] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL hntpps://openrevien wttp://iwiiinneeffoonumo net/forum?
id-Bkg6RiCqy7
[48] Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we lear from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
[49] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top- lop-perfrrmin reward models. In The Thirty-eight Conference on Neu- Information Processing Systems Datasets and Benchmarks Track, 2024. URL https wttpe//wwwww.etteoooord/dddooooe
[50] Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. prrffreenep Complementing ratings with prefer- ences. In The Thirteenth International Conference Learning Representations, 2025. URL
[51] Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4, 2023. URL attps://arxiv.er www.wxwwwwww...a//////aa.aaaaa/a /304 03439
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect
paper's contributions and scope?
Answer: [Yes]
Justification: This paper proposes novel training framework to explicitly align LLM- generated judgment distributions with empirical human distributions, utilizing distribu- tional aligmment objective with KL divergence, cross-entropy regularization, and adversarial training to enhance robustness. We accurately introduce this framework and its objectives in the abstract and introduction, and clearly highlight our main contributions at the end of the introduction (Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We focus on the development and validation of novel training framework for aligning LLM judgment distributions with human evaluations and do not include new theoretical results or formal proofs in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose the information necessary to reproduce our experimental results, which can be found in Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide the source code and datasets the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Possciiccciioss for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 http://ccppulllcuuuddooooo
(CodeSummsssionn Polly for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the training and test details in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct statistical significance tests for the main experiments in Section 5 as well as for the supplementary analyses in Appendix A and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information the computer resources in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp://neurrps.. http://www...ppppooocccccccccioa
Answer: [Yes]
Justification: This research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the question in Appendix
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss the question in Appendix
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The existing assets properly credited and mentioned. Please see Section 5.] for details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets are well documented the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: our study, the L.LM is used only for writing, editing, or formatting purposes and does not impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) hatt:p///earrrppccc/ooofeeeeeee/ /c//////eoeeeeeee//22////.. for what should or should not be described.
21

---

A Effectiveness under a Fixed Annotation Budget
critical question for our distributional framework whether its superior performance stems from its inherent methodology, or if it is merely an artifact of using more human annotations than single- point methods. To answer this question, this section empirically investigates the trade-off between annotating larger number of unique samples (sample breadth) versus collecting multiple judgments for each sample (annotation depth), while keeping the total number of collected annotations constant.
A.1 Experimental Setup
We designed a controlled experiment on the SNLI dataset with a fixed budget of approximately 8,000 total human annotations. We compare three distinct data allocation strategies:
Strategy (Breadth-Focused) 8,000 unique samples were used, each paired with one randomly selected human annotation. This strategy maximizes sample breadth to represent the conventional single-point alignment approach.
Strategy 2 (Balanced); 2,667 unique samples were used, each paired with three randomly selected human annotations. This strategy represents a balanced trade-off between sample breadth and annotation depth.
Strategy 3 (Depth-Focusedl. 1,600 unique samples were used, each paired with all five available human annotations. This strategy maximizes annotation depth over smaller set of unique samples.
All strategies were trained using our distributional alignment framework. Notably, for Strategy 1, our method's objective simplifies to become equivalent to traditional single-point alignment. All models were trained under identical conditions for fair comparison.
A.2 Results and Analysis
Table 4: Performance comparison under a fixed annotation budget. The balanced strategy (Strategy 2) achieves the best distributional alignment (KL Divergence) and is time-efficient.
Annotation Strategy
Time/Epoch (min)
KL Divergence (L)
Accuracy (1)
Strategy
21.9
0.32
89.1% 89.1%=o.m
Strategy Strategy
9.6
0. 0.251...0
88.9%=o.29
5.7
0.29
89.1% 19.1%_10.1%
The results, summarized in Table 4 highlight two key findings. First, the balanced approach (Strategy 2) achieves the best distributional alignment, yielding lower KI. Divergence than the other strategies. Compared to the greadth-focused strategy (Strategy 1), this underscores the importance of a distributional signal for effective alignment. Compared to the depth-focused strategy (Strategy 3), suggests that maximizing annotation depth the expense of sample variety can hurt generalization, likely due to the model overfitting to a smaller set of examples.
Second, the distributional approaches (Strategies 2 and 3) are substantially more computationally efficient. By processing smaller number of unique samples per epoch, they dramatically reduce training time. This analysis indicates that collecting moderate number of judgments per sample is more effective and efficient strategy for distributional alignment.
B Further Validation on Modern Benchmarks
To further assess the effectiveness of our framework, we conducted additional experiments on three modern benchmarks. These datasets correspond to three fundamental LLM-as-a-Judge applications but feature more challenging data, including denser annotations, more contemporary model outputs, and greater sample diversity.
B.1 Benchmark Datasets
serves as a highly robust benchmark for the Dataset Labeling task. In contrast to SNLI's 5 annotations per instance, ChaosNLI was specifically created to study the full spectrum
ChaosNLII 48
22

---

of human opinion by collecting 100 annotations for each of the 3,113 examples, providing an exceptionally dense and reliable ground-trun -ruth distribution.
HelpSteer2 44 provides a modern benchmark for the Quality Evaluation of LLM-generated
responses. Unlike SummEval, which evaluates outputs from specialized summarization models, HelpSteer2 focuses on rating the outputs of contemporary LLMs across five dimensions: help- fulness, correctness, coherence, complexity, and verbosity. Each response is rated by multiple annotators on a Likert-5 scale, and we follow our main experimental protocol by treating each dimension as an independent evaluation instance.
SelpSteer2-Preferemme (50) is a fine-grained benchmark for Pairwise Preference Prediction. Similar MT-Bench, it involves human annotators indicating their preference between two LLM responses. However, it offers a more nuanced 6-point rating scale (from -3 to +3) for the degree of preference. To maintain consistency with our experimental setup, we mapped these scores to a 3-point scale (A is better, B is better, or Tie), where scores of (-3, -2] correspond to one preference, tie, and scores of {2. 3) correspond the other preference.
scores of [-1, 1) correspond
B.2 Results and Analysis
The results on these modern benchmarks are presented in Table The findings are highly consistent with the conclusions from our main experiments presented in the paper.
Table 5: Results modern benchmarks for dataset labeling (ChaosNLI), quality evaluation (Help- Steer2), and preference prediction HlelpSteer2-Pieffeennee Our method consistently outperforms baselines in KL Divergence while achieving competitive or superior accuracy. The indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).
ChaosNLI
HelpSteer2
HelpSteer2-Pret
Model
Method
KLL
Accf
KLJ
Accf
KL4
Acct
GPT-4o-mini
Raw model
3.92
64.1% 64.1%6g..0%
4..3
42.4%=o.0x
13.8
9.2%
GPT-44
Raw model
2. 2.43.0.00
61.2% 61.2%
2. 2.09
10. 10.4%oo.00%
1.9820.00
18. 18.9% 18.9%
Raw model Single-point
60. 60.3%guu.% 3.94
3. 3.79
32. 32.0%
7.65 00
10.0% o%
Qwen2.5-7B
1 2.22
70.6%
0. 0.76
60.5%=o.11
71.4% 0.57
Distribution (Ours)
0 0.402
10..%%
0.631
60.0%
0.49 00.0
71.3% 21 1%
Raw model
0.08
57.8%2o.0%
2.50
13.3%
2.86 00
14.9% 40.0%
LLaMA3.1-88 1-BB
Single-point
1.14
65.0% 29.55%
0. 0.73
62.4%2o
62.4
0.51
71.6%
Distribution (Ours)
0. 0.43
65.7%
0.59 1n.00
62.4% 10.1%
0.47 20.00
73.8% 73.8% 20.3%
Across these benchmarks, our framework consistently outperforms the baselines in KL Divergence, often by significant margin. At the same time, maintains competitive superior accuracy. Consistent with our main experimental findings, these results provide strong additional evidence that approach is robust, generalizable, and highly effective for modern LLMM-a--aJudge applications.
C Out-of-Distribution Generalization
To assess our framework's 's generalization capabilities, we conducted an -fiitrriiitiinoo (OOD) experiment. Specifically, models were fine-tuned exclusively on the SNLI training set. Subsequently. they were evaluated directly on the unseen ChaosNLI test set, without any further training or adaptation. ChaosNLI serves as suitable OOD target due to its shared task formulation but distinct data source and significantly denser annotation distribution. The results are presented in Table 6
As shown in the table, even when faced with an unseen dataset, our framework significantly outper- forms the single-point alignment baseline in both KL Divergence and Accuracy. This result confirms that our method learns a robust and transferable representation of human disagreement.
D Analysis of Computational Efficiency
A key concern with adversarial training is that the inner PGD optimization loop could introduce significant computational overhead. However, in our framework, this overhead is minimal. Our PGD procedure computes gradients with respect to the target label distribution p(x), not the language
23

---

Table 6: OOD experiment results from training on SNLI and evaluating on the unseen ChaosNLI test set.
The indicates a statistically significant improvement over the single-pons poogle-point baseline (p < 0.05).
ChaosNLI (OOD)
Model
Method
KL
Acct
Raw model
4.0000.00
60.3%..0.0%
Qwen2.5-7B
Single-point
1.00
66.5%+..99%
Distribution (Ours)
0.31
67.8 % +0.5%
Raw model
0.6840.00
57.8%
1LaMA3.1-8B
Single-point
1.15
60.3%+0.11
Distribution (Ours)
0.46+..02
62.7 % 10.1%
model's parameters 0. During these inner steps, the model'ss output (=(x) is treated a fixed constant, thus avoiding any costly backpropagation through the language model.
To empirically quantify this overhead, we benchmarked the training efficiency on the MNLI dataset with the Qwen2.5-7B model on a single NVIDIA A100 GPU. As shown in Table 7 our method introduces a modest slowdown of approximately 21 % compared to the standard single-point baseline. We contend that this is an acceptable trade-off for the significant improvements in alignment quality and robustness.
Table 7: Training efficiency comparison on the MNLI dataset. Our method incurs a modest overhead for significant gain in alignment performance.
Method
Time Epoch (min)
Throughput (samples/sec)
Relative Slowdown
Single-point
23.4
5.70
1.0x
Distributional (Ours)
28.3
4.71
1.21x
E Detailed Main Results with Standard Deviations
This section provides the detailed experimental results for the open-source models presented in Section 5.2 We report the and standard deviation over 5 runs. Table ( presents the results for the NLI tasks, and Table 9 presents the results for the evaluation tasks.
Table 8: Detailed results (mean + std over runs) for NLI tasks (SNLI and MNLI). The indicates statistically significant improvement over the single-point baseline (p < 0.05).
SNLI
MNLI
Model
Method
KL4
Accf
KLJ
Acct
Single-point
0.60+0.01
92.7%
0.64
89.7%.1.2%
Qwen2.5
Distribution (Ours)
0.23
93.3 % 10.2%
0.23
89.8%
Single-point
0.6940.02
92.4%++
0.67
89. 89.6%.1.22%
LLaMA3.1
Distribution (Ours)
0.28*
92.4%++..13%
0.24 0.24"o.00
90.0 % +0.2%
F Prompts
Summeval. These prompts are reused from G-Eval]36) with slight modifications. Specifically,
the output label region has been explicitly specified, and the model is instructed to directly output evaluation results without providing explanations.
24

---

Table 9: Detailed results (mean + std over 5 runs) for evaluation tasks (Summeval and MT-Bench).
The * indicates statistically significant improvement over the single-point baseline (p < 0.05).
Summeval
MT-Bench
Model
Method
KLP
Acct
KLJ
Acct
Single-point
0.73
45.6%+0.6%
0.82
64.0%+1.0%
Qwen2.5
Distribution (Ours)
0.53
45.9% 1%
0.68 0.68 #0.02
65.4% 25.4%=11 +1.1%
Single-point
0.67
45.7%.4..8%
0..01
62. 62.1%%1...8
LLaMA3.1
Distribution (Ours)
0.51 10.01
47.3% 1
+0.6%
0..41..01
62.8%
Prompts Used for Summeval for Coherence Evaluation
You will be given one summary written for a news article.
Your task is to rate the summary on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Coherence (1-5) the collective quality of all sentences. We align this dimension with the DUC quality question of structure and coherence whereby "the summary should be well-structured and well-organized. The summary should not just a heap of related information, but should build from sentence to coherent body of information about topic."
Evaluation Steps:
1. Read the news article carefully and identify the main topic and key points.
2. Read the summary and compare it to the news article. Check the summary covers the main topic and key points of the news article, and if it presents them in clear and logical order.
Assign a score for coherence scale of where I the lowest and 5 is the highest based on the Evaluation Criteria.
Example:
Source Text:
[Document]
Summary:
(Summary)
Evaluation Form(scores ONLY): Make selection from "1", "2", "3",""" "4",""" Only write the answer with a single score, do not write reasons.
Coherence:
25

---

Prompts Used for Summeval for Consistency Evaluation
You will be given news article. You will then be given summary written for this article. Your task is rate the summary on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Consistency (1-5)- the factual alignment between the summary and the summarized source. factually consistent summary contains only statements that are entailed by the source document. Annotators were also asked to penalize summaries that contained hallucinated facts.
Evaluation Steps:
Read the news article carefully and identify the main facts and details it presents.
2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.
3. Assign score for consistency based the Evaluation Criteria.
Example:
Source Text:
[Document]
Summary:
(Summary)
Evaluation Form(scores ONLY): Make a selection from "1", "2", "3", "4", "5". Only write the answer with single score, do not write reasons.
Consistency:
26

---

Prompts Used for Summeval for Fluency Evaluation
You will be given summary written for news article.
Your task is rate the summary on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Fluency (1-5): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
Poor. The summary has many errors that make it hard understand or sound unnatural.
2: Below Average. The summary has several noticeable errors that significantly impact readability, though some parts can be understood with effort.
33: Fair. The summary has some errors that affect the clarity smoothness of the text, but the main points are still comprehensible.
4: Good. The summary has minor errors that do not significantly interfere with understand- ing; it reads relatively smoothly.
Excellent. The summary has few errors and is easy to read and follow, with notural- language throughout.
Example:
Summary:
[Summary)
Evaluation Form(scores ONLY): Make selection from "1", "2", "3", "4", "5". Only write the answer with single score, not write reasons.
Fluency:
27

---

Prompts Used for Summeval for Relevance Evaluation
You will be given summary written for news article.
Your task is rate the summary on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Relevance (1-5) selection of important content from the source. The summary should include only important information from the source document. Annotators were instructed to penalize summaries which contained redundancies and excess information.
Evaluation Steps:
1. Read the summary and the source document carefully. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information contains.
4. Assign relevance score from I to 5.
Example:
Source Text:
[Document]
Summary:
(Summary)
Evaluation Form(scores ONLY): Make a selection from "1", "2", "3", "4", "5". Only write the answer with single score, do not write reasons.
Relevance:
28

---

MT-Bench. These prompts are reused from MT-Bench 43 with slight modifications.
Prompts Used for MT-Bench
Human: For this task, you will be shown conversations between and Al assistant, labeled A and B. Your goal to evaluate which response (A or B) better follows the user's instructions and more helpfully answers their question.
Preffudgmentt <Conversation A> |conversation </Conversation A>
<Conversation B>
(conversation __
</Conversation B>
<Instructions>
Evaluate the two conversations and choose one of the following: If Conversation A's Al assistant better follows the user's instructions and answers their question
b: If Conversation B's AI assistant better follows the user's instructions and answers their question
tie: If both Al assistants equally good/poor in following instructions and answering the
user's question
Consider factors like helpfulness, relevance, accuracy, depth, creativity, and appropriate level of detail when making your evaluation. Do show positional bias towards A B. Response length should not unduly influence your decision. Make selection from "a", "b", "tie". Only write the answer with a single word, do not write reasons.
</Instructions>
</PrefJudgment>
Assistant:
29

---

NLI Tasks. For SNLI and MNLI datasets, prompts are reused from LogiEval [1] with slight modifications.
Prompts Used for NLI Tasks
You will be given a premise and a hypothesis. Your task is to determine whether the hypothesis logically follows from the premise. Choose only one of the following labels and output your answer with ONLY the label (one word), ensuring there are no spaces or other characters in the answer.
Possible labels:
The hypothesis follows logically from information contained the premise. It is not possible to determine whether the hypothesis is true or false without further information.
entailment:
neutral:
contradiction:
The hypothesis is logically false from the information contained in the premise.
Read the following premise and hypothesis thoroughly and select the correct answer from the three answer labels.
Premise: (premise)
Hypothesis: (hypothesis)
Make a selection from "entailment", "neutral", "contradiction". Only write the answer with a single word, do write reasons.
G Ethical Consideration
Our work explores the alignment of LLM-generated evaluation distributions with human judgment distributions, aiming to enhance the accuracy, diversity, and robustness of automatic evaluations. This has positive societal implications by potentially reducing the reliance on costly and time-consuming human evaluations, enabling scalable and fairer assessments in applications such as education, content moderation, and peer review. However, automated judgment systems also carry inherent risks. Misaligned or overconfident evaluations may lead to biased decisions, potentially reflecting and amplifying biases subtly present within the human data used for the alignment process itself. Such outcomes are particularly detrimental in high-stakes or subjective domains where fairness is paramount and the nuanced complexities of human judgment are easily replicated or may be overlooked by automated systems.
30


Reviews:
[{"review_id": "FydZRmXAZU", "reviewer": "Reviewer_Lp1a", "summary": "The authors investigate the effectiveness of training LLM-as-a-Judge models using distributions of human judgments rather than majority vote aggregations. They use a hybrid loss objective which combines KL divergence between the two distribution of labels with a deterministic label. This method shows better performance on four standard benchmarks.", "questions": "- Highly relevant paper not yet cited: https://arxiv.org/abs/2104.0867, which discusses exactly this problem in the context of NLI, which you also use. Given the identical task definition, the non-mention of this or other related works is a substantial weakness for the paper.\n- If you set a dataset size budget (number of human judgments collected), would it be more effective to sample several judgements for a single datapoint (to provide a distribution to train on) or new examples? I worry that this method is only effective due to essentially training on a much larger dataset than single-point. If correcting for this, which method performs better?", "limitations": "- Not many alternative methods are compared, related works section is very limited and missing key citations (see above)\n- Limited experimental investigation\n- Method requires high-redundancy annotations, which are costly.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Achieves best performance on several tasks as compared to previous methods\n- Introduces a new adversarial loss objective for the alignment with human distributions that improves performance\n- Discussion of hyperparameter impact is appreciated\n\nWeaknesses\n- No evaluation on tasks more similar to what LLM-as-a-Judge is often used for in practice, such as dataset labeling, quality evaluation, or pairwise preference prediction. That stated, the chosen subset of tasks (NLI, summarization, dialogue evaluation) is reasonable enough for research. The paper could be improved by expanding the tasks for which LLM-as-a-Judge is used to more closely align with common use-cases or incoporate a more diverse range of tasks.\n- Limited experimental results and minimal evaluation. There is essentially only a single result in the paper: better performance than training on single-point data or raw LLM judges.", "was_revised": true, "final_justification": "The rebuttal provided much appreciated experimental results and rigor to the paper, and improved my understanding of the significance of the results.", "rebuttal": "Dear Reviewer Lp1a,\nThank you for your rigorous and constructive review. We present our detailed responses below.\n\n> **For Weakness 1: No evaluation on tasks more similar to what LLM-as-a-Judge is often used for in practice, ... incoporate a more diverse range of tasks.**\n\nThank you for the feedback on practical relevance. We find that some of our chosen tasks correspond well to the paradigms the reviewer mentioned: SummEval is a 'quality evaluation' task, and MT-Bench addresses 'pairwise preference prediction'.\n\nTo further strengthen our evaluation, we have conducted new experiments on the recently proposed datasets, which align more closely with common use-cases. **HelpSteer2** [1] provides a state-of-the-art benchmark for 'quality evaluation' of modern LLM generations, while **HelpSteer2-Preference** [2] offers a more fine-grained benchmark for 'preference prediction' on paired LLM responses. The results are presented below, where we report **mean ± std** over multiple runs. The `*` indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).\n\n**HelpSteer2**\n\n| Model         | Method                  |     KL (↓)     |   Accuracy (↑)    |\n| :------------ | :---------------------- | :-----------------------: | :---------------: |\n| *GPT-4o-mini* | Raw model               |      $4.83_{±0.00}$       | $42.4\\\\%_{±0.0\\\\%}$ |\n| *GPT-4o*      | Raw model               |      $2.09_{±0.00}$       | $40.4\\\\%_{±0.0\\\\%}$ |\n| *Qwen2.5-7B*  | Raw model               |      $3.79_{±0.00}$       | $32.0\\\\%_{±0.0\\\\%}$ |\n|               | Single-point            |      $0.76_{±0.03}$       | $60.5\\\\%_{±0.1\\\\%}$ |\n|               | **Distribution (Ours)** | $\\mathbf{0.63}^*_{±0.01}$ | $60.0\\\\%_{±0.5\\\\%}$ |\n| *LLaMA3.1-8B* | Raw model               |      $2.50_{±0.00}$       | $13.3\\\\%_{±0.0\\\\%}$ |\n|               | Single-point            |      $0.73_{±0.02}$       | $62.4\\\\%_{±0.3\\\\%}$ |\n|               | **Distribution (Ours)** | $\\mathbf{0.59}^*_{±0.00}$ | $62.4\\\\%_{±0.1\\\\%}$ |\n\n**HelpSteer2-Preference**\n\n| Model         | Method                  |     KL (↓)     |         Accuracy (↑)         |\n| :------------ | :---------------------- | :-----------------------: | :--------------------------: |\n| GPT-4o-mini   | Raw model               |      $13.8_{±0.00}$       |       $9.2\\\\%_{±0.0\\\\%}$       |\n| GPT-4o        | Raw model               |      $4.98_{±0.00}$       |      $18.9\\\\%_{±0.0\\\\%}$       |\n| *Qwen2.5-7B*  | Raw model               |      $7.65_{±0.00}$       |      $10.0\\\\%_{±0.0\\\\%}$       |\n|               | Single-point            |      $0.57_{±0.02}$       |      $71.4\\\\%_{±0.8\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.49}^*_{±0.01}$ |      $71.3\\\\%_{±1.1\\\\%}$       |\n| *LLaMA3.1-8B* | Raw model               |      $2.86_{±0.00}$       |      $14.9\\\\%_{±0.0\\\\%}$       |\n|               | Single-point            |      $0.51_{±0.01}$       |      $71.6\\\\%_{±0.5\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.47}^*_{±0.00}$ | $\\mathbf{73.8\\\\%}^*_{±0.3\\\\%}$ |\n\nAs the results show, our framework's effectiveness is further validated on these modern and practical benchmarks. We will incorporate these extensive new results into the final version of our paper and are grateful for the suggestion.\n\n> **For Weakness 2: Limited experimental results and minimal evaluation... or raw LLM judges.**\n\nThank you for the feedback. We would like to respectfully clarify that our original evaluation was already covering Overall Performance (Table 1), Component Contributions (Table 2), Hyper-parameter Analysis (Figure 3), and Robustness to Label Noise (Table 4). We have also made three substantial additions during rebuttal:\n\n1. **Out-of-Distribution (OOD) Generalization:** We conducted a new OOD experiment, fine-tuning models on SNLI and evaluating them on the **unseen** ChaosNLI [3] dataset. The results below provide strong evidence that our framework learns a transferable representation of human disagreement.\n\n   | Model         | Method                  |     KL (↓)     |         Accuracy (↑)         |\n   | :------------ | :---------------------- | :-----------------------: | :--------------------------: |\n   | *Qwen2.5-7B*  | Raw model               |      $3.94_{±0.00}$       |      $60.3\\\\%_{±0.0\\\\%}$       |\n   |               | Single-point            |      $1.01_{±0.01}$       |      $66.5\\\\%_{±0.9\\\\%}$       |\n   |               | **Distribution (Ours)** | $\\mathbf{0.31}^*_{±0.01}$ | $\\mathbf{67.8\\\\%}^*_{±0.5\\\\%}$ |\n   | *LLaMA3.1-8B* | Raw model               |      $0.68_{±0.00}$       |      $57.8 \\\\%_{±0.0\\\\%}$      |\n   |               | Single-point            |      $1.15_{±0.05}$       |      $60.3\\\\%_{±0.1\\\\%}$       |\n   |               | **Distribution (Ours)** | $\\mathbf{0.46}^*_{±0.02}$ | $\\mathbf{62.7\\\\%}^*_{±0.1\\\\%}$ |\n\n2. **Validation on More Relevant Benchmarks:** We have validated our method on new, contemporary benchmarks, as detailed in our response to **Weakness 1**.\n   \n3. **Enhanced Statistical Rigor:** To improve the rigor of our claims, we have incorporated **significance tests into all new experimental results presented in this rebuttal**. We are committed to this standard of validation and will update the tables from our original manuscript to include these statistics.\n\n> **For Q1: Highly relevant paper not yet cited: Nie et al. (2021)... a substantial weakness for the paper.**\n\nWe sincerely thank the reviewer for pointing out this highly relevant work and apologize for the oversight in our initial submission. While we share the foundational goal with Nie et al. (2021), our work is **distinguished by a fundamentally different methodology, which we designed for the modern LLM-as-a-Judge paradigm and validated across a broader scope of tasks.** Our key distinctions are:\n\n1. **Novel Adversarial Robustness Mechanism**: our work introduces a key innovation not present in Nie et al.: **adversarial training on the label distribution**. We identified that the empirical human distribution is a noisy approximation of the true underlying distribution due to limited annotations. Our adversarial strategy is the first to mechanistically address this challenge by training the model to be robust against worst-case perturbations. This significantly enhances alignment fidelity and robustness, a critical aspect that Nie et al. acknowledge as a problem but do not solve.\n\n2. **Modern LLM-a-Judge Context with Broader Scope**: Our work is situated in the modern LLM-as-a-Judge paradigm, where we validate our framework on powerful modern LLMs across a broader range of tasks beyond NLI (e.g., quality scoring and preference understanding).\n\nWe will revise our manuscript to include a thorough discussion of Nie et al. and take this opportunity to broaden our discussion of related literature. \n\n\n> **For Q2: If you set a dataset size budget... which method performs better?**\n\nThis is an excellent and insightful question. To empirically investigate the trade-off in data collection under a fixed budget, we conducted a new experiment on the SNLI dataset. We fixed a budget of approximately 8,000 total annotations and compared three distinct strategies:\n\n- **Strategy 1:** 8,000 samples, each with 1 random annotation.\n- **Strategy 2:** 2,667 samples, each with 3 random annotations.\n- **Strategy 3:** 1,600 samples, each with all 5 annotations.\n\n| Strategy | #Samples | #Annotations | Time/Epoch (min) |     KL (↓)     |   Accuracy (↑)    |\n| :------------------ | :------: | :-----------------: | :--------------: | :-----------------------: | :---------------: |\n| Strategy 1          |  8,000   |          1          |      $21.9$      |      $0.32_{±0.01}$       | $89.1\\\\%_{±0.4\\\\%}$ |\n| Strategy 2          |  2,667   |          3          |      $9.6$       | $\\mathbf{0.25}^*_{±0.00}$ | $88.9\\\\%_{±0.2\\\\%}$ |\n| Strategy 3          |  1,600   |          5          |      $5.7$       |      $0.29_{±0.00}$       | $89.1\\\\%_{±0.1\\\\%}$ |\n\nThe results reveal an interesting **trade-off between annotation depth and sample breadth**. Strategy 2 achieves the best alignment, outperforming both Strategy 1 (which lacks a direct distributional signal) and Strategy 3 (which lacks sufficient sample for generalization). As a secondary benefit, our approach is also far more **computationally efficient**, dramatically reducing training time per epoch.\n\nWe will add this important analysis to our paper and are grateful for the reviewer's prompt, which led to this deeper and more nuanced finding.\n\n> **For Limitation 1: Not many alternative methods are compared, related works section is very limited and missing key citations**\n\nPlease see our detailed response to **Question 1** above.\n\n> **For Limitation 2: Limited experimental investigation**\n\nAs detailed in our response to **Weakness 2**, we have conducted substantial new experiments (on new SOTA datasets, OOD generalization, etc.) to form a more comprehensive evaluation.\n\n> **For Limitation 3: Method requires high-redundancy annotations, which are costly.**\n\nWe agree this is a key consideration. However, we believe the cost is justified by the fundamental importance of modeling human disagreement for trustworthy AI in high-stakes applications. In fact, our robust alignment method is exactly designed for the case with limited annotations.\n\n**We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.**\n\n**References:**\n\n[1] Wang, Zhilin, et al. \"Helpsteer2: Open-source dataset for training top-performing reward models.\" *arXiv:2406.08673* (2024).\n\n[2] Wang, Zhilin, et al. \"HelpSteer2-Preference: Complementing Ratings with Preferences.\" *arXiv:2410.01257* (2024).\n\n[3] Nie, Y., et al. \"What can we learn from collective human opinions on natural language inference data.\" *EMNLP* (2020): 9131-9143."}, {"review_id": "CY2zi1rqxo", "reviewer": "Reviewer_a38u", "summary": "This work introduces a framework for aligning an LLM-based judge's output distribution with that of human annotators. The frameworks include a hybrid KL + CE loss and adversarial training on perturbed inputs via PGD. Evaluation on 4 tasks and 2 models shows the effectiveness of the proposed framework.", "questions": "1. [3] should be cited in Section 4. The formulations are identical: a CE + KL objective solved in a PGD min–max loop, only this work shifts the KL term from input perturbations to label perturbations. \n2. Can this approach generalize to unseen datasets, e.g., with a different label space?\n3. In Figure 3, why does the KL divergence increase considerably at alpha=1.0 for MTBench, but none of the other tasks? Is there some feature of MTBench that necessitates the CE term more so than the other datasets?\n\n[3] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International conference on machine learning. PMLR, 2019.\n\nI'm willing to increase my score if the evaluation is improved.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and reads well.\n2. The approach is simple yet principled. Figures 1 and 2 are clear and aid in understanding.\n3. Evaluation reveals the method is effective, with consistent improvements in KL divergence.\n\nWeaknesses:\n1. Despite the reduction in KL divergence, the improvements in accuracy are minimal. As such, more motivation is needed to justify modeling the full distribution given the added data annotation cost.\n2. The evaluation is underwhelming. First, the margins (particularly accuracy) are slim, and only point estimates are provided. Second, the dataset choice could be improved to better fit the LLM-as-a-Judge narrative (e.g., [1] and [2], both have ~3 annotations per instance). Lastly, the method is only tested on in-distribution tasks so the generaliability of the framework is not clear.\n\n[1] Wang, Zhilin, et al. \"Helpsteer2: Open-source dataset for training top-performing reward models.\" arXiv preprint arXiv:2406.08673 (2024).\n\n[2] Wang, Zhilin, et al. \"HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages.\" arXiv preprint arXiv:2505.11475 (2025).", "was_revised": true, "final_justification": "The authors presented a series of new experiments that resolved many of my concerns and significantly improved the quality of the evaluation. While the reduction in KL divergence is impressive, I'm not convinced the required annotation cost is justified. Therefore, I'm giving a borderline score of 4 and leaving the decision to the AC.", "rebuttal": "Dear Reviewer a38u,\n\nThank you for your thoughtful feedback. We will address your concerns below.\n\n> **For Weakness 1: Despite the reduction in KL divergence, the improvements in accuracy are minimal. As such, more motivation is needed to justify modeling the full distribution given the added data annotation cost.**\n\nThe core value of our work extends beyond improving single-point accuracy. Instead, our goal is to capture the rich, underlying structure of human evaluations—such as the variance of opinions, which signals the level of consensus. This distributional information is valuable in high-stakes applications like medical diagnosis, where understanding the full spectrum of expert opinions is more useful and safer signal than a single, overconfident prediction.\n\nAccordingly, our evaluation is centered on KL Divergence as the primary metric, because it directly quantifies how well our model captures this entire information-rich distribution. Accuracy serves as a secondary metric to ensure our method captures the majority consensus and remains competitive with traditional standards.\n\nWe will expand on this motivation in the final paper.\n\n\n\n> **For Weakness 2: The evaluation is underwhelming. First, the margins (particularly accuracy) are slim, and only point estimates are provided. Second, the dataset choice could be improved to better fit the LLM-as-a-Judge narrative. Lastly, the method is only tested on in-distribution tasks so the generaliability of the framework is not clear.**\n\n**1. On slim margins and point estimates**\n\nWe have conducted new multi-run experiments (N=5) on the datasets with the narrowest margins: **SNLI and MNLI**. The results are reported in the table below, where we report **mean ± std** over multiple runs. The `*` indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).\n\n| Model         | Method                  |        SNLI KL (↓)        |         SNLI Acc (↑)         |        MNLI KL (↓)        |         MNLI Acc (↑)         |\n| :------------ | :---------------------- | :-----------------------: | :--------------------------: | :-----------------------: | :--------------------------: |\n| *Qwen2.5-7B*  | Single-point            |      $0.60_{±0.01}$       |      $92.7\\\\%_{±0.1\\\\%}$       |      $0.64_{±0.02}$       |      $89.7\\\\%_{±0.2\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.23}^*_{±0.01}$ | $\\mathbf{93.3\\\\%}^*_{±0.2\\\\%}$ | $\\mathbf{0.23}^*_{±0.00}$ |      $89.8\\\\%_{±0.2\\\\%}$       |\n| *LLaMA3.1-8B* | Single-point            |      $0.69_{±0.02}$       |      $92.4\\\\%_{±0.42\\\\%}$      |      $0.67_{±0.02}$       |      $89.6\\\\%_{±0.2\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.28}^*_{±0.01}$ |      $92.4\\\\%_{±0.13\\\\%}$      | $\\mathbf{0.24}^*_{±0.02}$ | $\\mathbf{90.0\\\\%}^*_{±0.2\\\\%}$ |\n\nThe results highlight our key achievement: yielding substantial and statistically significant improvements in KL divergence, while maintaining competitive accuracy. We will include these rigorous statistics for all datasets in the final version.\n\n**2. On the choice of datasets**\n\nWe have conducted new experiments on these modern benchmarks (**HelpSteer2** [1] for Qualitative Assessment task, and **HelpSteer2-Preference** [2] for Human Preference Understanding task). The results for both datasets are presented below.\n\n**HelpSteer2**\n\n| Model | Method | KL Divergence (↓) | Accuracy (↑) |\n| :--- | :--- | :---: | :---: |\n| *GPT-4o-mini* | Raw model | $4.83_{±0.00}$ | $42.4\\\\%_{±0.0\\\\%}$ |\n| *GPT-4o* | Raw model | $2.09_{±0.00}$ | $40.4\\\\%_{±0.0\\\\%}$ |\n| *Qwen2.5-7B* | Raw model | $3.79_{±0.00}$ | $32.0\\\\%_{±0.0\\\\%}$ |\n| | Single-point | $0.76_{±0.03}$ | $60.5\\\\%_{±0.1\\\\%}$ |\n| | **Distribution (Ours)** | $\\mathbf{0.63}^*_{±0.01}$ | $60.0\\\\%_{±0.5\\\\%}$ |\n| *LLaMA3.1-8B* | Raw model | $2.50_{±0.00}$ | $13.3\\\\%_{±0.0\\\\%}$ |\n| | Single-point | $0.73_{±0.02}$ | $62.4\\\\%_{±0.3\\\\%}$ |\n| | **Distribution (Ours)** | $\\mathbf{0.59}^*_{±0.00}$ | $62.4\\\\%_{±0.1\\\\%}$ |\n\n**HelpSteer2-Preference**\n\n| Model | Method | KL Divergence (↓) | Accuracy (↑) |\n| :--- | :--- | :---: | :---: |\n| GPT-4o-mini | Raw model | $13.8_{±0.00}$ | $9.2\\\\%_{±0.0\\\\%}$ |\n| GPT-4o | Raw model | $4.98_{±0.00}$ | $18.9\\\\%_{±0.0\\\\%}$ |\n| *Qwen2.5-7B* | Raw model | $7.65_{±0.00}$ | $10.0\\\\%_{±0.0\\\\%}$ |\n| | Single-point | $0.57_{±0.02}$ | $71.4\\\\%_{±0.8\\\\%}$ |\n| | **Distribution (Ours)** | $\\mathbf{0.49}^*_{±0.01}$ | $71.3\\\\%_{±1.1\\\\%}$ |\n| *LLaMA3.1-8B* | Raw model | $2.86_{±0.00}$ | $14.9\\\\%_{±0.0\\\\%}$ |\n| | Single-point | $0.51_{±0.01}$ | $71.6\\\\%_{±0.5\\\\%}$ |\n| | **Distribution (Ours)** | $\\mathbf{0.47}^*_{±0.00}$ | $\\mathbf{73.8\\\\%}^*_{±0.3\\\\%}$ |\n\nThe results on both modern benchmarks provide strong additional evidence for our framework's effectiveness. We will incorporate these extensive new results into the final paper and are grateful for this suggestion.\n\n**3. On out-of-distribution tasks and generalizability**\n\nWe thank the reviewer for this excellent suggestion. While our paper already demonstrated robustness to label noise (Sec 4.5), we agree that testing for out-of-distribution (OOD) generalization is a complementary and more challenging evaluation. Therefore, we conducted a new OOD experiment, fine-tuning models on SNLI and evaluating them directly on the **unseen** ChaosNLI [3] dataset. The results are presented below.\n\n| Model         | Method                  |     KL Divergence (↓)     |         Accuracy (↑)         |\n| :------------ | :---------------------- | :-----------------------: | :--------------------------: |\n| *Qwen2.5-7B*  | Raw model               |      $3.94_{±0.00}$       |      $60.3\\\\%_{±0.0\\\\%}$       |\n|               | Single-point            |      $1.01_{±0.01}$       |      $66.5\\\\%_{±0.9\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.31}^*_{±0.01}$ | $\\mathbf{67.8\\\\%}^*_{±0.5\\\\%}$ |\n| *LLaMA3.1-8B* | Raw model               |      $0.68_{±0.00}$       |      $57.8 \\\\%_{±0.0\\\\%}$      |\n|               | Single-point            |      $1.15_{±0.05}$       |      $60.3\\\\%_{±0.1\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.46}^*_{±0.02}$ | $\\mathbf{62.7\\\\%}^*_{±0.1\\\\%}$ |\n\nThese OOD results demonstrate our method's superior generalization over the baseline, confirming that our framework learns a transferable representation of human disagreement. We will integrate this crucial analysis into our final paper and are grateful for the valuable suggestion.\n\n\n\n> **For Q1: [3] should be cited in Section 4. The formulations are identical: a CE + KL objective solved in a PGD min–max loop, only this work shifts the KL term from input perturbations to label perturbations.**\n\nWe thank the reviewer for their sharp observation and the opportunity to clarify this connection.\n\nWhile there is a structural resemblance in the optimization form, the two works operate in fundamentally different domains with distinct goals. TRADES addresses robustness to external input attacks in computer vision. Our work, in contrast, addresses distributional alignment with human opinions in the paradigm of LLM-as-a-Judge.\n\nThis fundamental difference in goals is directly reflected in our loss design. In TRADES, the KL term is an **auxiliary regularizer** to enforce output smoothness against input perturbations. In our framework, the KL term is the **primary objective**, designed to directly align model outputs with the target human distribution. Our auxiliary CE loss merely serves to stabilize this primary alignment task, a design validated by our hyper-parameter analysis. \n\nWe will revise our manuscript to cite TRADES and add this important contextual discussion. Thank you again for this valuable prompt.\n\n> **For Q2: Can this approach generalize to unseen datasets, e.g., with a different label space?**\n\n As demonstrated in our new OOD experiment (presented in our response to Weakness 2.3), our framework generalizes well. The results confirm that our distribution alignment method's capabilities transfer to the unseen ChaosNLI dataset significantly better than the single-point baseline.\n\n> **For Q3: In Figure 3, why does the KL divergence increase considerably at alpha=1.0 for MTBench, but none of the other tasks? Is there some feature of MTBench that necessitates the CE term more so than the other datasets?**\n\nThis is a sharp observation. Our hypothesis is that this phenomenon is tied to the **high difficulty and subjectivity** of certain tasks. The greater disagreement among annotators leads to more complex target distributions `p(x)`. For such challenging distributions, a pure KL divergence objective (`α=1.0`) can be unstable.\n\nTo verify this hypothesis, we conducted the same analysis on **HelpSteer2-Preference**. Similar to MT-Bench, this is a benchmark for subjective human preference. We observed the exact same pattern—a **sharp increase** in KL divergence when the CE loss was completely removed:\n\n| Alpha ($\\alpha$)      | 0.0  | 0.2  | 0.4  | 0.6  | 0.8      | 1.0      |\n| --------------------- | ---- | ---- | ---- | ---- | -------- | -------- |\n| **KL Divergence (↓)** | 0.57 | 0.53 | 0.50 | 0.49 | **0.48** | **2.01** |\n\nThe consistent results from both datasets show that the auxiliary CE loss is crucial for stabilizing these difficult alignment tasks.\n\n\n\n**We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.**\n\n\n\n**References:**\n\n[1] Wang, Zhilin, et al. \"Helpsteer2: Open-source dataset for training top-performing reward models.\" *arXiv:2406.08673* (2024).\n\n[2] Wang, Zhilin, et al. \"HelpSteer2-Preference: Complementing Ratings with Preferences.\" *arXiv:2410.01257* (2024).\n\n[3] Nie, Y., et al. \"What can we learn from collective human opinions on natural language inference data.\" *EMNLP* (2020): 9131-9143."}, {"review_id": "bBFwIX35my", "reviewer": "Reviewer_6xen", "summary": "The current LLM-as-a-Judge setup relies on single-point judgments which dismisses the inherent subjectivity of certain tasks and inputs. Instead, this paper aims to have a judgment distribution as an output of this paradigm. \n\nThe paper proposes a framework that aligns the output distribution of an LLM with the human judgment distributions. This is done through a hybrid loss-function that uses both KL divergence to minimize the distance. For improved robustness, reducing the overfitting to dataset specific noise, the authors apply a worst-case perturbation to the distributions. This is done through Projected Gradient Descent. \n\nThe framework is tested on Natural Language Inference (MNLI and SNLI), Qualitative Assessment, and Human Preference Understanding, using both open-source (Qwen2.5-7B and LLaMa3.1-8B) and closed-source (GPT4o, GPT4o-mini) models. To evaluate the framework, accuracy and KL-divergence is used. The results indicate that the proposed framework outperforms other baselines in terms of KL-divergence and yields competitive results for accuracy.", "questions": "- Why do you opt for the KL loss specifically and not for example Jensen-Shannon Divergence, which is a symmetrical version of it? \n- Line 204-205: I think you accidentally mixed up closed-source and open-source in the sentence.\n- Shouldn’t perturbation take into account annotator information? For instance, if there is no high diversity in annotators, the perturbation could be higher? Or is that impractical for the real world? (I am just curious about your thoughts on this specific question)\n- Line 242 - 245: Doesn’t gpt-4o-mini do better than gpt-4o in terms of accuracy for the NLI datasets", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper addresses an important limitation of the current LLM-as-a-Judge paradigm. It is important that such a judge can reflect multiple viewpoints, especially for tasks that include a lot of subjectivity. \n- The paper is well-written and structured and has an easy to follow flow. \n- Good analyses through ablation studies, e.g., looking at the impact of both the KL and CE losses and the hyperparameters. \n\n**Weaknesses**\n- While the authors are working toward a model that can output the distribution, I find some specific design choices in the setup rather contradicting of the notion. Firstly, I wonder if it is a good idea to incorporate the aggregated majority vote in the loss function through the CE loss (I am assuming this is the hard version of the loss function and not the soft CE loss, please correct me if I am wrong). If we have a rather spread out distribution, then there is no clear single ground-truth label. Consequently, this reflects into the usage of accuracy as a metric. Though I admit that measuring correctness gets a bit though to measure in such a case but I think at least a discussion around this would strengthen the paper a lot. \n- I wonder why ChaosNLI [1] was not used for the experiments. This is an NLI dataset with 100 annotations per sample, which would be a great fit for the experiments conducted in this paper. \n- Some analysis of the method’s robustness to other unseen datasets would have been nice. E.g., the models for SNLI and MNLI could have been tested on ChaosNLI. \n- Given that some of the models show very similar accuracy scores, statistical significance tests would be really useful. \n\n---------------------------------------\n[1] Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What Can We Learn from Collective Human Opinions on Natural Language Inference Data?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.", "was_revised": true, "final_justification": "I think the paper addresses an important problem of the potential subjectivity of certain tasks with LLM-as-a-judge. \n\nThe proposed framework is rigorously tested and evaluated accordingly. I am still a bit apprehensive about using the hard CE loss for a task on subjectivity, intuitively this does not seem very straightforward to me, however, the authors backup this choice with empirical results indicating that this provides stability during training. \n\nThey cleared up my doubts about using accuracy as a metric, which is a secondary metric for verification but the focus is more on KL-divergence. \n\nThe authors added my suggested experiments, including statistical tests, measuring performance on out-of-distribution datasets, experimenting with Jensen Shannon Divergence as a loss function and testing the framework on ChaosNLI. This strengthens the findings of the authors and I think the subfield of subjective modeling can benefit from the proposed method which goes beyond mere prompting strategies.", "rebuttal": "Dear reviewer 6xen,\n\nThank you so much for your precious time and your thoughtful feedback. We will now address your concerns.\n\n> **For Weakness 1: The using of CE loss and accuracy metric may contradict our paper's goal of modeling distributions.**\n\nWe thank the reviewer for this fundamental question. These designs are not contradictory, but are deliberate and principled decisions to ensure both **training stability** and **meaningful evaluation**.\n\n**1. CE loss -- for learning stability:** \n\nThe reviewer is correct that we use a standard CE loss on the majority-vote label. However, its role is not as a primary objective, but as an **auxiliary regularizer for training stability**. As we noted in our paper (Line 129-133), this design is directly inspired by the Knowledge Distillation [2, 3] framework, where a \"hard\" label loss is used to stabilize learning from a \"soft\" target distribution. Our own results in Figure 3 empirically validate this necessity: on challenging datasets like MT-Bench, a pure KL objective (`α=1.0`) is unstable and performs worse, demonstrating the crucial role of the CE term as a stabilizing anchor.\n\n**2. Accuracy metric -- for meaningful evaluation:** \n\nSimilarly, **Accuracy is a secondary, complementary metric**, with KL divergence being our primary measure of success. We include it for two practical reasons: it serves as a valuable indicator of our model's ability to capture the majority consensus, and more critically, it provides a crucial bridge for comparison with prior work that relies solely on this measure. This allows us to demonstrate that we achieve our primary goal without sacrificing performance on this traditional standard.\n\nWe will add a more detailed discussion of this rationale to the paper. We thank the reviewer again for this valuable prompt.\n\n\n\n> **For Weakness 2: Not using the highly relevant ChaosNLI [1] dataset.**\n\nWe sincerely thank the reviewer for this important suggestion. We agree that ChaosNLI is an ideal and challenging benchmark for our distribution alignment framework. Consequently, we have conducted **a new set of experiments on the ChaosNLI dataset**. The results are presented in the table below, where we report **mean ± std** over multiple runs. The `*` indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).\n\n| Model | Method | KL Divergence (↓) | Accuracy (↑) |\n| :--- | :--- | :---: | :---: |\n| *GPT-4o-mini* | Raw model | $3.92_{±0.00}$ | $64.1\\\\%_{±0.0\\\\%}$ |\n| *GPT-4o* | Raw model | $2.43_{±0.00}$ | $61.2\\\\%_{±0.0\\\\%}$ |\n| *Qwen2.5-7B* | Raw model | $3.94_{±0.00}$ | $60.3\\\\%_{±0.0\\\\%}$ |\n| | Single-point | $1.22_{±0.02}$ | $70.6\\\\%_{±0.7\\\\%}$ |\n| | **Distribution (Ours)** | $\\mathbf{0.41}^*_{±0.02}$ | $\\mathbf{71.8\\\\%}^*_{±0.5\\\\%}$ |\n| *LLaMA3.1-8B* | Raw model | $0.68_{±0.00}$ | $57.8\\\\%_{±0.0\\\\%}$ |\n| | Single-point | $1.14_{±0.04}$ | $65.0\\\\%_{±0.5\\\\%}$ |\n| | **Distribution (Ours)** | $\\mathbf{0.43}^*_{±0.05}$ | $65.7\\\\%_{±1.2\\\\%}$ |\n\nThe results on ChaosNLI are fully consistent with our original findings: our method significantly outperforms baselines in KL divergence while maintaining competitive accuracy. We will incorporate these results into the final paper and are grateful for the valuable suggestion.\n\n\n\n> **For Weakness 3: Analyze the method’s robustness to other unseen datasets.**\n\nWe thank the reviewer for this excellent suggestion. While our paper already demonstrated robustness to label noise (Sec 4.5), we agree that testing for **out-of-distribution (OOD) generalization** is a complementary and even more challenging evaluation. Therefore, we conducted a new OOD experiment, fine-tuning models on the SNLI training set and evaluating them directly on the **unseen ChaosNLI** test set:\n\n| Model         | Method                  |     KL Divergence (↓)     |         Accuracy (↑)         |\n| :------------ | :---------------------- | :-----------------------: | :--------------------------: |\n| *Qwen2.5-7B*  | Raw model               |      $3.94_{±0.00}$       |      $60.3\\\\%_{±0.0\\\\%}$       |\n|               | Single-point            |      $1.01_{±0.01}$       |      $66.5\\\\%_{±0.9\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.31}^*_{±0.01}$ | $\\mathbf{67.8\\\\%}^*_{±0.5\\\\%}$ |\n| *LLaMA3.1-8B* | Raw model               |      $0.68_{±0.00}$       |      $57.8 \\\\%_{±0.0\\\\%}$      |\n|               | Single-point            |      $1.15_{±0.05}$       |      $60.3\\\\%_{±0.1\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.46}^*_{±0.02}$ | $\\mathbf{62.7\\\\%}^*_{±0.1\\\\%}$ |\n\nThese OOD results demonstrate our method's superior generalization over the baseline, confirming that our framework learns a transferable representation of human disagreement instead of merely overfitting to dataset-specific patterns. We will integrate this crucial analysis into our final paper and are grateful for the valuable suggestion.\n\n\n\n> **For Weakness 4: The need for statistical significance tests.**\n\nWe thank the reviewer for this crucial point on experimental rigor. We completely agree that statistical testing is essential, particularly for metrics with small margins like accuracy. We have conducted new **multi-run experiments (N=5)**, focusing specifically on the datasets with the narrowest accuracy margins: **SNLI and MNLI**.\n\n| Model         | Method                  |        SNLI KL (↓)        |         SNLI Acc (↑)         |        MNLI KL (↓)        |         MNLI Acc (↑)         |\n| :------------ | :---------------------- | :-----------------------: | :--------------------------: | :-----------------------: | :--------------------------: |\n| *Qwen2.5-7B*  | Single-point            |      $0.60_{±0.01}$       |      $92.7\\\\%_{±0.1\\\\%}$       |      $0.64_{±0.02}$       |      $89.7\\\\%_{±0.2\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.23}^*_{±0.01}$ | $\\mathbf{93.3\\\\%}^*_{±0.2\\\\%}$ | $\\mathbf{0.23}^*_{±0.00}$ |      $89.8\\\\%_{±0.2\\\\%}$       |\n| *LLaMA3.1-8B* | Single-point            |      $0.69_{±0.02}$       |      $92.4\\\\%_{±0.42\\\\%}$      |      $0.67_{±0.02}$       |      $89.6\\\\%_{±0.2\\\\%}$       |\n|               | **Distribution (Ours)** | $\\mathbf{0.28}^*_{±0.01}$ |      $92.4\\\\%_{±0.13\\\\%}$      | $\\mathbf{0.24}^*_{±0.02}$ | $\\mathbf{90.0\\\\%}^*_{±0.2\\\\%}$ |\n\nThe table highlights our method's key achievement: yielding substantial and statistically significant improvements in KL divergence, while maintaining competitive accuracy. While these multi-run experiments were limited to SNLI/MNLI due to time constraints, we will include these rigorous statistics for all datasets in the camera-ready version. Thank you again for this feedback.\n\n\n\n> **For Q1: Why not using JS loss?**\n\nWe thank the reviewer for this insightful question. Our choice of KL divergence is based on two main reasons:\n\n- First, our choice is motivated by its established success as the standard measure in paradigms like Knowledge Distillation [2, 3], which shares the core technical goal of aligning a model's output distribution with a target \"soft\" distribution. \n- Second, KL divergence generally offers more stable gradients for optimization compared to JS divergence, which can suffer from vanishing gradients. [4]\n\nTo empirically validate this choice, we conducted a new experiment comparing both training objectives on SNLI using Qwen2.5-7B. The results below confirm that using KL divergence yields stronger performance across all metrics:\n\n| Method | KL Divergence (↓) | JS Divergence (↓) | Accuracy (↑) |\n| :--- | :---: | :---: | :---: |\n| **Ours (using KL Div.)** | $\\mathbf{0.23}^*_{±0.00}$ | $\\mathbf{0.049}^*_{±0.000}$ | $\\mathbf{93.2\\\\%}^*_{±0.2\\\\%}$ |\n| Ours (using JS Div.) | $0.27_{±0.00}$ | $0.050_{±0.000}$ | $92.1\\\\%_{±0.2\\\\%}$ |\n\n\n\n> **For Q2: Typo on Line 204-205.**\n\nWe thank the reviewer for catching this error. We will correct this typo in the final version.\n\n\n\n> **For Q3: Perturbation taking into account annotator information.**\n\nThis is a fascinating and thought-provoking question. We agree with the reviewer's excellent intuition: if the annotator pool lacks diversity, the collected distribution may be biased, and a larger perturbation radius `ε` would be a principled way to account for this unobserved variance. The primary practical obstacle, however, is the lack of annotator metadata in most crowdsourced datasets, and we would like to leave this direction as an important future work.\n\n\n\n>**For Q4: Accuracy of GPT-4o-mini vs. GPT-4o on NLI tasks**\n\nThank you for this sharp observation. GPT-4o-mini indeed achieves slightly higher accuracy on the NLI datasets than GPT-4o. Our hypothesis is that on a mature and relatively well-defined task like NLI, the accuracy of powerful models can be very close.\n\n\n\n**We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.**\n\n\n\n**References:**\n\n[1] Nie, Y., et al. \"What can we learn from collective human opinions on natural language inference data.\" *EMNLP* (2020): 9131-9143.\n\n[2] Hinton, Geoffrey, et al. \"Distilling the knowledge in a neural network.\" *arXiv:1503.02531* (2015).\n\n[3] Sanh, Victor, et al. \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.\" *arXiv:1910.01108* (2019).\n\n[4] Arjovsky, Martin, et al. \"Wasserstein GAN.\" *PMLR* (2017): 214-223."}, {"review_id": "RtOmuUyQM1", "reviewer": "Reviewer_3U8S", "summary": "The paper addresses a key limitation in the LLM-as-a-Judge paradigm: human evaluations inherently involve diversity and uncertainty (reflected as distributions), but existing methods rely on single-point judgments, leading to information loss and reduced reliability. The authors propose a distribution alignment framework to align LLM-generated judgment distributions with empirical human evaluation distributions.  Extensive experiments demonstrate that that framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods, with improved alignment quality, evaluation accuracy, and robustness.", "questions": "Please refer to the comments above", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n-The paper is novel in that it appears to be the first work to explicitly align LLM judgment distributions (not just point estimates) with human evaluation distributions.\n\n- The proposed method is technically sound in that it introduces hybrid loss to balance distribution alignment and stability, as well as employing adversarial training to avoid overfitting to limited human annotations.\n\n- The method is rigorously evaluated across diverse tasks and LLM backbones, followed by ablation studies to confirm the usefulness of each component.\n\nWeaknesses:\n- The approach requires datasets with multiple human annotations per sample to obtain empirical approximations, where are not commonly seen in practice.\n\n- The approach may incur extra computational overhead in the introduced adversarial training strategy, but the overall efficiency appears undiscussed in the paper.", "was_revised": true, "final_justification": "I recommend accepting this paper, based on its novelty, technical strength, and evaluation comprehensiveness.", "rebuttal": "Dear reviewer 3U8S,\n\nThank you so much for your precious time and your positive feedback. We will now address your concerns.\n\n> **For Weakness 1: The approach requires datasets with multiple human annotations per sample to obtain empirical approximations, where are not commonly seen in practice.**\n\nThank you for this valuable comment. We acknowledge that requiring multiple human annotations per sample is indeed a necessary condition for our method. However, we would like to offer the following points to alleviate this concern:\n\n1. **Fundamental Importance:** As discussed in our paper, we believe that modeling the distribution of human judgments is fundamentally important. The natural disagreements among annotators carry valuable information that is often lost when reducing labels to a single point. This richness is particularly critical in **high-stakes domains**—such as medical diagnosis or policy-making—where capturing the full range of expert perspectives is essential. Therefore, we argue that the cost of collecting multiple human annotations is justified, given the significance of the problems we aim to address.\n\n2. **Generalization Capability of Our Framework:** A key strength of modern LLMs is learning transferable knowledge. To verify that our framework learns a transferable representation of human disagreement, rather than overfitting to dataset-specific annotation patterns, we conducted an OOD experiment (training on SNLI, testing on the unseen ChaosNLI [1] dataset).\n\n   The results are presented in the table below, where we report mean ± std over several runs and `*` denotes a statistically significant improvement (p < 0.05). The data confirms our method's strong generalization to the unseen dataset. This suggests that the learned alignment capability is transferable, **potentially reducing the need to collect extensive multi-annotated data for every new task**.\n   \n   | Model         | Method                  |     KL Divergence (↓)     |         Accuracy (↑)         |\n   | :------------ | :---------------------- | :-----------------------: | :--------------------------: |\n   | *Qwen2.5-7B*  | Raw model               |      $3.94_{±0.00}$       |      $60.3\\\\%_{±0.0\\\\%}$       |\n   |               | Single-point            |      $1.01_{±0.01}$       |      $66.5\\\\%_{±0.9\\\\%}$       |\n   |               | **Distribution (Ours)** | $\\mathbf{0.31}^*_{±0.01}$ | $\\mathbf{67.8\\\\%}^*_{±0.5\\\\%}$ |\n   | *LLaMA3.1-8B* | Raw model               |      $0.68_{±0.00}$       |      $57.8 \\\\%_{±0.0\\\\%}$      |\n   |               | Single-point            |      $1.15_{±0.05}$       |      $60.3\\\\%_{±0.1\\\\%}$       |\n   |               | **Distribution (Ours)** | $\\mathbf{0.46}^*_{±0.02}$ | $\\mathbf{62.7\\\\%}^*_{±0.1\\\\%}$ |\n\n\n3. **Robustness with Limited Annotations:** In fact, during the early conceptualization of this work, we were fully aware of this potential limitation. To address this issue, our **adversarial training strategy** was specifically designed to enhance robustness, even when only a small number of annotations are available per sample. Our original experiments (Table 1 and 2) already demonstrate that our method achieves satisfactory performance under such settings.\n\n\n\n> **For Weakness 2: The approach may incur extra computational overhead in the introduced adversarial training strategy, but the overall efficiency appears undiscussed in the paper.**\n\nWe thank the reviewer for this important practical point. The computational overhead is introduced by the inner PGD loop (K=5 steps), which finds the worst-case label distribution. Crucially, this PGD optimization computes gradients with respect to the label distribution `p(x)`, not the model parameters `θ`. During this inner loop, the model's output distribution `q_theta(x)` is treated as a fixed constant. As a result, these PGD steps are computationally inexpensive as they **do not involve any backpropagation through the main language model**.\n\nTo empirically validate this, we conducted a direct comparison of the **training efficiency**, benchmarked on the MNLI dataset with the Qwen2.5-7B model on a single NVIDIA A100 GPU. The table below shows our method incurs a modest slowdown of approximately **21%**, which we believe is a worthwhile trade-off for the significant gains in alignment quality.\n\n\n| Method                  | Time / Epoch (min) | Throughput (samples/sec) | Relative Slowdown |\n| :---------------------- | :----------------: | :----------------------: | :---------------: |\n| Single-point            |       $23.4$       |          $5.70$          |   $1.0 \\times$    |\n| **Distribution (Ours)** |       $28.3$       |          $4.71$          |   $1.21\\times$    |\n\nWe will add a discussion of this analysis and include these results in the final version of our paper.\n\n\n**We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.**\n\n**References:**\n\n[1] Nie, Y., et al. \"What can we learn from collective human opinions on natural language inference data.\" *EMNLP* (2020): 9131-9143."}]

Author rebuttal:

Meta-review: ### (a) Scientific Claims and Findings (AC Summary)

The paper identifies a critical limitation of the current “LLM-as-a-Judge” paradigm: almost all existing methods collapse human judgments—which are naturally diverse and uncertain—into a single deterministic label.  
**Core Claim**: By training an LLM to output a **full probability distribution** over human labels (rather than a single point estimate), one can obtain more faithful, robust and informative evaluations.  

**Key Technical Contributions**  
1. A **hybrid loss** that combines  
   • KL divergence between model and empirical human distributions (primary objective), and  
   • Cross-entropy against majority-vote labels (auxiliary stabilizer).  
2. **Adversarial training** on the label distribution (PGD perturbations) to improve robustness when human annotations are scarce.  
3. Extensive experiments across 6 datasets (NLI, summarization, dialogue, preference) and 4 model families (open & closed) show consistent, **statistically significant reductions in KL divergence** while maintaining competitive accuracy.  
4. **Auxiliary findings**:  
   • Out-of-distribution generalization (train on SNLI → test on ChaosNLI) is improved.  
   • Under a fixed annotation budget, it is better to collect **3 labels per example for 2.7k examples** than 1 label for 8k or 5 labels for 1.6k.  
   • Training overhead is only ≈21 %.

---

### (b) Strengths

1. **Problem novelty & relevance**: First principled attempt to align LLM judgments with human **distributions**, not just point consensus.  
2. **Theoretical grounding**: Simple yet solid loss design justified by knowledge-distillation and adversarial-robustness literature.  
3. **Empirical rigor**: Large-scale evaluation, ablations, significance tests, OOD tests, cost–benefit analysis, and new state-of-the-art benchmarks (HelpSteer2, HelpSteer2-Preference, ChaosNLI).  

---

### (c) Weaknesses & Missing Pieces

1. **Annotation cost**: Requires **multiple human judgments per example**, which is expensive and not yet standard in most crowdsourced datasets.  
2. **Accuracy lift is modest**: While KL divergence improves substantially, absolute accuracy gains are small (often < 2 %). The paper’s own framing treats accuracy as secondary; nevertheless, practitioners may question the ROI.  
3. **Limited label-space transfer**: Experiments keep the same categorical label set during OOD transfer. Generalization to **different label vocabularies** (e.g., switching from 3-way NLI to 5-way quality scores) remains untested.  
4. **Computational overhead**: Although the authors quantify it, adversarial PGD still adds ~20 % training time and implementation complexity.  

None of these weaknesses are fatal, but they bound the scope of the contribution.

---

### (d) Most Important Reasons for **Poster** Acceptance

- **Significant conceptual advance**: Moves the field from “best single answer” to “distribution of human judgments,” opening new research directions.  
- **Solid empirical validation**: Addressed **every major reviewer concern** via new experiments, statistical tests, and datasets.  
- **Community value**: Provides a reusable training recipe and benchmark suite that future work can build on.  

The work is **not a spotlight/oral** because:  
• The core technique is incremental—combining known losses with adversarial regularization.  
• The accuracy improvements are small; impact is methodological rather than dramatic.

---

### (e) Rebuttal Summary & AC Weigh-In

| **Anon ID** | **Main Concerns Raised** | **Author Response** | **AC Assessment** |
|---|---|---|---|
| **Lp1a** | 1. Tasks not practical (missing dataset labeling, pairwise preference). 2. Baseline comparisons limited. 3. Annotation-cost critique. | Added HelpSteer2 & HelpSteer2-Preference, ChaosNLI OOD, data-budget analysis, significance tests. | **Fully addressed**; new benchmarks match real use-cases; reviewer raised score. |
| **a38u** | 1. Accuracy gains small vs. KL; justify annotation cost. 2. Need more datasets & OOD. 3. Missing TRADES citation. | Provided new motivation paragraph, HelpSteer2/ChaosNLI, OOD, JS vs KL ablation, added TRADES discussion. | **Fully addressed**; statistical rigor added; reviewer raised score.. |
| **6xen** | 1. CE loss / accuracy contradicts distribution goal. 2. Why not ChaosNLI? 3. Need OOD & significance. | Clarified CE as stabilizer, added ChaosNLI, OOD, significance tests, JS divergence ablation, fixed typo. | **Fully addressed**; reviewer raised score. |
| **3U8S** | 1. Requires costly multi-annotations. 2. Computational overhead unexplored. | Provided data-budget trade-off, efficiency numbers (~21 %), robustness with few labels via adversarial training. | **Fully addressed**; concerns mitigated. |

**AC Weigh-In**: All critical technical and empirical questions were answered with **new experiments** and **theoretical clarifications**. The concerns about annotation cost and overhead were **quantified and bounded**, making them **acceptable limitations** rather than rejection reasons. The consensus among reviewers shifted upward, with final ratings 4, 4, 5, 5 (borderline → accept). Therefore, **poster acceptance** is warranted.
Decision: Accept (poster)