Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
Authors: Xingjian Zhang
Tianhong Gao
Suliang Jin
Tianhao Wang
Teng Ye
Eytan Adar
Qiaozhu Mei
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) are increasingly used as raters for evaluation tasks. However, their reliability is often limited for subjective tasks, when human judgments involve subtle reasoning beyond annotation labels. Thinking traces, the reasoning behind a judgment, are highly informative but challenging to collect and curate. We present a human-LLM collaborative framework to infer thinking traces from label-only annotations. The proposed framework uses a simple and effective rejection sampling method to reconstruct these traces at scale. These inferred thinking traces are applied to two complementary tasks: (1) fine-tuning open LLM raters; and (2) synthesizing clearer annotation guidelines for proprietary LLM raters. Across multiple datasets, our methods lead to significantly improved LLM-human agreement. Additionally, the refined annotation guidelines increase agreement among different LLM models. These results suggest that LLMs can serve as practical proxies for otherwise unrevealed human thinking traces, enabling label-only corpora to be extended into thinking–trace–augmented resources that enhance the reliability of LLM raters.

Published in Transactions on Machine Learning Research (02/2026)
Through the Judge's Eyes: Inferred Thinking Traces Im- prove Reliability of LLM Raters
Xingjian Zhang
jimmyazj@umiich.ce edu
University of Michigan, Ann Arbor, Michigon, USA
Tianhong Gao
tihgaoShuiich.ede edu
University of Michigan, Ann Arbor, Michigan, USA
Suliang Jin
suliang@umichh.ele edu
University of Michigan, Ann Arbor, Michigon, USA
Tianhao Wang
tianhaouangg@ucsd..de
University of California, San Diego, California, USA
Teng Ye
tengye edu
University of Minnesota, Twin Cities, Minneapolis, Minnesota, USA
Eytan Adar
eador@umich. edu
University of Michigan, Ann Arbor, Michigon, USA
Qiaozhu Mei
gmei@umichh.o edu
University of Michigan, Ann Arbor, Michigan, USA
Reviewed OpenReview: / net/ ptr/formm? id= 15LQ629Yps
Abstract
Large language models (LLMs) are increasingly used as raters for evaluation tasks. However, their reliability is often limited for subjective tasks, when human judgments involve subtle reasoning beyond annotation labels. Thinking traces, the reasoning behind a judgment, are highly informative but challenging collect and curate. We present human-LLM collaborative framework infer thinking traces from label-only annotations. The proposed framework uses simple and effective rejection sampling method to reconstruct these traces at scale. These inferred thinking traces are applied to two complementary tasks: (1) fine- tuning open LLM raters; and (2) synthesizing clearer annotation guidelines for proprietary LLM raters. Across multiple datasets, our methods lead to significantly improved LLM- human agreement. Additionally, the refined annotation guidelines increase agreement among different LLM models. These results suggest that LLMs can serve as practical proxies for otherwise unrevealed humann thinking traces, enabling label-only corpora to be extended into thinking- cratiing-tracaaaae tracc--aggmeneed resources that enhance the reliability of LLM raters
1 Introduction
Large language models (LLMs) are increasingly used as automated evaluators for open-ended text generation
tasks, primarily due to their remarkable efficiency and scalability (Gu et al. 2024b Zheng al. 2023). This
approach has been applied to evaluate diverse applications such code generation (Zhou et al. 2025 text
summarization [Bedemariam et al. 2025), and dialogue (Gu et al. 2024b However, the reliability of LLM
For reproducing results, our codebase can accessed

---

CChun et al. Published in Transactions on Machine Learning Research (02/2026)
Please read the prompt, the human story and the subject story (both stories might be the same). The story you will have to rate is the subject story.
Note: some stories have been abruptly cut in the middle of a sentence. Please rate them as if they ended just before the unfinished sentence.
Note: if the story is not relevant with respect to the prompt, only affects the Relevance criterion! Do not rate 1 everywhere.
Then, please rate the subject story on a scale from (worst) to 5 (best) on Complexity:
1: The story is very simple with no elaborate elements.
2: The story is somewhat simple with few elaborate elements.
3: The story has some complexity but lacks depth in certain areas.
4: The story is mostly complex with minor areas lacking depth.
5: The story is highly complex and elaborate throughout.
Figure 1: An example of the annotation codebook for evaluating complexity of short stories 2022h. Only basic instructions and vague scoring rubrics are provided.
evaluators often decreases in subjective tasks that require nuanced human judgment (Ismayilzada et al. 2024 Gómez-Rodrigue Rodríguez & Williams 2023), especially when they did not receive specialized training (Krumdick et al. 2025).
To understand what is missing in calibrating these models, we can draw an analogy from the process of training human annotators for qualitative coding tasks. To ensure alignment on a subjective task, a novice annotator lypically benefits from three key components that an expert can provide: (1) an annolation codebook with detailed instructions and rubrics; (2) set of eramples illustrating representative inputs and their corresponding labels; and critically, (3) explanations detailing the reasoning for assigning specific label to each example. Among the three components, explanations are particularly important in creating a shared understanding of the task among annotators (Artstein & Poesio 2008 O'Connor & Joffe, 2020). We hypothesize that these explanations of the expert's reasoning, which we term the thinking traces, are equally necessary calibrate LLM evaluators and improve their alignment with human judgment.
Despite their value, thinking traces are largely absent from existing annotation datasets. The primary reason is the substantial effort required: articulating and recording detailed reasoning process is significantly more time-consuning and costly than providing a single label (DeYoung et al.) 2019). Most annotation interfaces are not designed to capture this information, and incentivizing raters to produce high-quality traces is challenging (Carton et al.) 2020). As a consequence, many datascts for subjective tasks are limited to sparse and sometimes unreliable labels, lacking the rich context that thinking traces would provide.
Reasoning language models (RLMs) offer a promising opportunity to address this issue (Comanici al. 2025 Guo et al.) 2025 Jacch et al.) 2024 These models designed to generate step-by-step rcasoning, often termed "intermediate reasoning tokens", before producing final answer (Guo et al. 2025). This model- generated reasoning is intended to align with the underlying thinking traces of human experts, also known as "human priors" in Guo et al.] (2025). If this alignment is faithful, these models could serve as a proxy to reconstruct thinking traces at scale, mitigating the costly human annotation process. However, whether AT-generated reasoning truly aligns with human reasoning remains an open question, especially for subjective tasks where rcasoning can be highly muanced. While directly verifying this alignment is challenging, given the lack of ground-truth human thinking traces, a more pragmatic research question in our context is: Can these model-inferred thinking traces, even if imperfect, serve useful proxy to improve the reliability of automated LLM raters?
To lackle this question, we present a human-LLM collaborative framework to infer thinking traces from label-only human annotations, We employ a simple, yet effective, rejection sampling method to reconstruct these traces at scale. We demonstrate the utility of inferred thinking traces through two complementary applications. First, we use them as training data to fine-tune open-weight LLM raters so that they are spe- cialized for the evaluation task. Second, for models where fine-tuning is not option, we introduce a novel
2

---

Published in Transactions on Machine Learning Research (02/2026)
method for codebook refinement. Inspired by iterative refinement practices in qualitative coding (O'Connor & Joffe 2020), this approach addresses the common problem of ambiguous instructions in many annotation codebooks (Klie et al. 2023) (Figure D Our automatic pipeline leverages the inferred traces to synthe- size a new, more explicit codebook that better steers the model's judgments. Experiments show that both methods significantly improve the aligument of LLM raters with human judgments.. Furthermore, the syn- thesized codebook also improves the agreement between different LLM providers, demonstrating its utility in promoting cross-model consistency. These findings demonstrate that model-inferred thinking traces can effectively serve practical proxy for human reasoning, thus improving the reliability and consistency of automated LLM raters on subjective evaluation tasks.
2 Related Work
Reasoning Language Models. Reasoning Language Models (RLMs) represent significant evolution from standard LLMs, as they are specifically trained to solve tasks that require multiple steps of deliberation. This class of models has become the de facto standard for high performance (Comanici et al. 2025 Guo et al. 2025 Jaech et al. 2024). demonstrating superior capabilities in tasks such as math and coding, Models such as DeepSeek RI demonstrate the effectiveness of large-scale reinforcement learning in cultivating complex reasoning behaviors in these models (Guo et al., (2025), and provide public access to these reasoning tokens. Although these models are intended to align with "human priors" (Guo et al.] 2025), the validity of these reasoning tokens remains debatable question (Amirizaniani
al.) 2024 Dasgupta et al.) 2024 Mondorf & Plank 2024). Testing the validity of thinking traces is beyond the scope of this paper. However, we are interested in practical objective: leveraging RLMs tool to infer latent thinking traces from human experts, thereby improving the reliability of downstream LLM raters.
Distilling Reasoning via Rejection Sampling. A powerful paradigm for generating synthetic data is rejection sampling fine-tuning, where a model generates numerous candidate reasoning traces, and only those passing a filtering criterion are used for subsequent training. Existing work has pioneered this approach for tasks with objectively correct answers, such as mathematical reasoning, using an automatic verifier
accept only correct outputs (Guo et al.) 2025 Zelikman et al. 2022 Wadhwa al.) 2024 Tong et al. 2024 Singh et al. 2023). To handle more nuanced tasks, methods such RAFT (Dong et al 2023) replaced the binary verifier with a learned reward model, filtering for outputs that score highly on desired attributes. This trend of applying reasoning to more diverse domains is also evident in recent work on recommender
answering (Chen et al. 2024), software engineering (Cuadron et al. 2024| Penedo et al. 2025), finance (Qian et al. 2025), and cybersecurity (Yu et al. (025). Concurrently, rescarchers have begun curating general-purpont SFT datascts enriched with distilled thinking traces to train open reasoning language models (Maitern et al.) 2025 Guha et. al.) (2255). Crucially, while previous work exclusively focuses on using these curated traces for fine-tuning, we introduce novel second application:
systems [sai et al.] 2024), medical
distilling insights from the thinking traces to refine annotation codebooks automatically.
Refining Codebooks for Qualitative Coding and Annotation The refinement of codebooks is a long-standing challenge in both qualitative research and large-scale data annotation. Research in HCI and crowd-sourcing communities developed auman-centria workflows to improve task instructions, often relying on crowd workers to identify ambiguitics that were then resolved by an expert or through discussion (Chang et al.. 2017 Manam& & Quinn] 2018 K. Chaithanya Manam et al. 2019 Pradhan et al., 2022). More recently,
researchers have begun to leverage LLMs to automate this process. This includes work on human- LLM
collaboration for qualitative coding [iao et al. 2023 Halterman & Keith
2025
Wiebe et al. 2025 Tori
ct al. 2024 Meng et al., 2024). Other work has demonstrated that LLMs can automatically analyze data or
synl hesize new guidelines to improve rownstreem performance (Bibal et al. 2025 Hsu el al. 2025 Srivaslava ett al. 2025). A key distinction of these automated methods is that they rely only on an LLM's 's intrinsic capabilitics to analyze content or generate text, rather than being grounded in direct human annotation feedback. In contrast, our approach creates a collaborative pipeline with inputs from both humans and LLMs.
3

---

Published in Transactions on Machine Learning Research (02/2026)
3 Method
3.1
Inferring Latent
Thinking Traces
Our method begins with seed dataset of human-annotated examples, denoted Dnuman {( }s)} i=1' Here, represents an annotation target (e.g., story) and y; is the corresponding label (e.g., Likert rating for story "complexity") assigned by human raters based on specific codebook C.
For any non-trivial annotation lask, a humman rater engages in a cognitive process to arrive at the final label ye- We formalize this unobserved process as the human's latent thinking trace, t,. This trace, which includes applying guidelines, resolving ambiguities, and weighing evidence, is not recorded in standard annotation workflows due to the high cost and inherent difficulty of articulating such complex reasoning.
Our goal is to generate high-quality praxy for this latent trace, which we term the reconstructed thinking trace, 1/2 To achieve this, we use an RLM that can generate thinking tokens, hereafter referred to as the generator model, to perform rejection sampling. For each input item Ty we provide the generator model with the codebook C and the annotation target and prompt. to generate a step-by-step thinking trace that concludes with a final rating. This generation performed & times independently, yielding set of k candidate traco-label pairs, [(t,, (( where th,s denotes the j-th candidate trace and V4,5 denotes the j-th predicted label for z,, For a model to reach the correct human label y/, its reasoning is more likely (but not guaranteed) to be a plausible approxination of the latent human thought process. Therefore, we filter the candidate set by retaining only those traces where the model's predicted label vk., matches the ground-trun truth human label y/s This creates a set of traces for each sample Ny:
T = I E {I....,k) k} vh. = y
From this set Ty select the first sample to serve the final reconstructed thinking trace, G. This transforms our initial dataset into an augmented, reasoning-rich dataset, Dreason {(2,, tt v))) which forms the basis for the alignment techniques described next. In essence, each data point in Dreason can be considered a product of human-LLM collaboration, where the human label is used as a verifier to select an LLM-generated thinking trace. An example inferred thinking trace is shown Figure 2 and an illustration of the overall process provided in Figure
Okay, let's tackle this rating. The user wants me to evaluate the Subject Story based on the Eugagement criterion. The prompt mentions that stories create actual universes, and unfinished ones are hellish. The Human Story talks about a life controlled by creator, leading to existential dread when stories are incomplete.
Now, looking at the Subject Story. It starts with character exhausted from playing games, going home to feed his family. Then there's a sudden shift to genie game over the ocean, hot water leaking, cold sea shattering the window. The narrative seems disjointed. The protagonist has baseball bat, gets hit in the head, storm hits, and the story ends abruptly.
Engagement-wise, the story is confusing. The transitions between scenes are jarring. The mix of mundane elements (family, baseball bat) with surreal events (genie, sudden storms) might intrigue some readers but could also lose them. The abrupt ending, while fitting the prompt's theme of unfinished stories, might leave readers frustrated. The imagery is vivid but lacks coherence, making it hard to follow. The emotional build-up from hope to despair isn't clearly developed here. The reader might struggle to connect with the protagonist the plot. So, Engagement is probably low. I'd rate it a 2 becanse it has some interesting elements but is difficult to follow and loses interest quickly.
Figure 2: An example of the inferred thinking trace in evaluating the engagement level of short story.
3.2 Improving LLM Raters
The primary goal of inferred thinking traces, Dresson, to improve the reliability of LLM raters. We show our method's utility in two complementary scenarios, which cover the primary ways LLMs are used in
4

---

Published in Transactions on Machine Learning Research (02/2026)
Dataset & Annotation Codebook
Candidate Thinking Traces
Augmented Dataset
Target
Rating
So, the engagement scale, maless because scory barely enguging. While raises some questions, the execution is unelear and hard to fallow, leading to minimal reader interest.
Target
Thinking Traces
Bating
Snigtha had bought some chocolates the local sugermarket.
Snigdha had bought some chocolates at the local aupermazket, zare sight
... In summary, the story has moments that could engage, but the lock of clear plct, inconsästent characters, and a abrupt seene changes make it hard to interested...
eight
3
In mary, the story has moments that could engage, but lack a clear plot, inconsistent characters, and abrupt scene changes make it hard to stay interested. It's somewaat enguging in purts but loses the reader's attention frequently hence ratingg.
"ob that funny.... i remember, my androw" " anymonn. my name hitler, goow up in berlin."...
"oh that was fanzy.... remember, name andrew:" anymare. my hams adolf hitler, in berlin.'
... So, on the eagagement scule, a.2 muken sence because
2
... So the 4.0 rating likely coenes from the story being engagging due to its interesting premise and fast paced events, but lases points for coherence clarity certain parta, keeps the time travel elemente but has moments where the reader might get confused or lose track, henre not full 5.
engaging. While
story
raiscs some questions,
the exccution unelear and hard to follow, leading minimal reader interest.
... Pleuse rate the subject story on a seale
from I (worst) (best) engagement...
Step Sumple multiple candidate responses using RLM
Step Reject non-aligned thinking traces by raling
Figure rautin of Inferring Thinking Traces through an RLM. Multiple candidate thinking traces and labels are sampled. Only candidates that are aligned with human-annotated labels are preserved. The detailed process is provided in Section 3.1
practice: as fine-tunable models that can be specialized to specific domains; and as black- box APIs. For open-weight models, we use Dreason to directly fine-tune the L.M rater, aligning its judgments with human reasoning patterns. More broadly, for the common case of proprietary models, we instead use our dataset to automatically refine the annotation codebook. This refined codebook serves as a more effective prompt to guide the model's behavior. This offers practical, training-free way to leverage state-of-the-at LLMs.
Fine-Tuning Specialized LLM Raters. We show that the augmented dataset D Dreason provides pow- erful training resource for creating a specialized LLM rater. Unlike standard supervised fine-tuning (SFT), which only trains on the input-label pairs (I,, vi); the augmented dataset provides the model with the com- plete reasoning trace t/. We hypothesize that the thinking traces contribute to a much richer training signal, which can significantly improve the model's alignment with human feedback. The model is trained using standard SFT objective, as shown in Equation [ to prodict the full thinking trace followed by the final label. Here, ne denotes the LLM rater with weights 0. This objective teaches the model not only what label to produce, but also how to reason towards it.
LSFT =
[- log (((,)]
(1)
Refining Annotation Codebooks. We further investigate ways to take advantage of the inferred thinking traces when using proprietary LLMs whose weights are unavailable (i.e., they cannot be updated). Our approach is motivated by the observation that many codebooks contain ambiguous descriptions or lack concrete criteria, resulting in inconsistent ratings when used directly as prompts, as shown in Figure [ Our collection of reconstructed traces serves as a corpus of successful step-by-step applications of these guidelines. We can leverage these traces to improve the codebook by extracting common reasoning patterns and syuthesizing a more explicit step-by-step procedure. This improved codebook, C', can then be used to construct more effective prompts for proprietary LLMs where fine-tuning is not option
Specifically, we propose a two-stage process that targets the two main components of the codebook: the task instructions and the scoring rubric.
WWile not test this hypothesis in our work, an improved codebook could also potentially enhance the consistency and quality human raters.
5

---

Published in Transactions on Machine Learning Research (02/2026)
1. Improve task instructions: We first sample 10 thinking traces for each rating level and prompt LLM to summarize the common reasoning patterns in these traces into an explicit, step-by-step procedure for the new task instructions.
Enrich scoring rubrics: To enrich the scoring rubric, we then sample a set 50 thinking traces for each rating level and extract short critiques from each one (e.g., "The lack of coherent plot... Not all critiques are representative of a given rating level. For example, good story may still have some flaws. To find the most representative critiques, we cluster the lext embeddings of these critiques using the text-embedding----aaee semantic similarity to cach cluster's centroid. Finally, we present these representative critiques to an LLM, prompling it to synthesize a new, detailed rubric for each rating level that is grounded in concrete examples.
model and select the critiques with the highest
We provide the complete prompts for this process in Appendix D
Read the Materials Thoroughly Start by reading the prompt, the lummnn slory, and the subject story. Note that the subject story is the one being rated, not the human story
Step-by-Step Rating
Look for key elements such as characters, events, plot, themes, setting
Evaluate if the identified elements developed, interconnected, and coherent
Consider whether the story follows structured progression (linear or non-linear)...
Stories with minimal elements or undeveloped content are simpler. Stories that attempt multi-layered explo- ration of ideas, even if not fully realized, should evaluated more complex.
Select the Rating
1. The story is very simple, lacks depth, and introduces no intricate or developed elements.
The story somewhat simple, with few elements that are underdeveloped, fragmented, or disconnected.
'Te story demonstrates some complexity, introducing multiple elements, but lacks depth, development, or coherence in integrating these elements.
4. The story is mostly complex, weaving logether several developed elements with minor areas that could be expanded refined.
5. The story is highly elaborate and complex, integrating multiple layers (e-g., detailed world-building, character depth, advanced structure, thematic sophistication) in a cohesive and interconnected way.
Write the final rating Place your chosen rating...
Figure 4:
An example of the refined annotation codebook. Some content has been omitted due to space limitations. See Appendix D for complete example.
3.3 Evaluation
To empirically validate LLM raters, we use two primary criteria. The first, Agreement with Human Judg- ments, serves as the main test for both applications. The second, Inter-Rater Reliability, is an assessment designed to measure the specific impact of our codebook refinement. All evaluations are performed on held-out test sets randomly sampled from the original dataset.
Agreement with Human Judgments This criterion measures the degree to which an LLM rater's outputs agree with human judgments. We apply this evaluation to the outputs of both our applications. For this analysis, we treat a held-out set of human ratings as the gold standard and compare the LLM-generated labels against them. To quantify this agreement, use several standard metrics for labels in the Likert

---

Published in Transactions on Machine Learning Research (02/2026)
scale. Our primary focus is Kendall's Tau (r), as ranking correlation is the most used measure in the literature [Liu et al. 2023 2024 Thakur et al. 2025 Gu et al. 2024a). For a more complete assessment,
we also report the Mean Squared Error (MSE) for average error magnitude and the Intraclass Correlation
Coefficient (ICC3) for consistency, based on a two-way mixed-efoocts model (Koo & Li 2016).
Inter-Rater Reliability. For codebook refinement, another robust indicator success is whether enables independent raters to arrive at the same conclusions consistently. High inter-rater reliability is a sign of clear and well-defined annotation process. To measure this, we the improved codebook C' to prompt M different proprietary LLMs to rate the same set of items. We then calculate the reliability across these M independent LLM raters using ICC3 (Koo & Li 2016).
Statistical Significance Test. To assess statistical significance, we use the following statistical tests. For fine-tuned LLM raters, we apply a paired f-test on MSE, and employ onc-sided bootstrap hypothesis tests to evaluate whether improvements in and ICC3 are statistically significant the direction of superiorily. For the refined codebook analysis, we conduct one-sided paired t-tests on each metric, comparing the performance of four LLM raters before and after refinement, thereby directly testing whether the refinement leads to consistent improvements across raters. To assess the statistical significance of agreement between LLM raters, we conduct a one-sided bootstrap hypothesis test to eva evaluate whether the improvement in TCC3 is significant in the direction of superiority.
4 Experiment
4.1 Datasets
We evaluated our framework across five diverse annotation lasks, all of which require deliberate reasoning rather than simple intuition to assign a reliable score. Another key selection criterion is the public availability of the annotation codebook used for each task. These tasks cover two distinct types of human feodback on generated text: single holistic quality scores and fine-grained scores across multiple dimensions. For consis- tency in our evaluation methodology, we focused datasets that Likert scale. However, our framework is readily applicable to other formats, such forced-choice rankings. The five tasks are summarized below:
1. Evaluating Short Stories. We use the HANNA dataset (Chhun et al. 2022 annotation dataset for evaluating geneine-generateed stories. The stories were rated human annotators recruited through Amazon Mechanical Turk, restricted to native English speakers with a Master's Qualification. The complezity dimension provides a measure of structural and elemental richness, while engagement captures readers' emotional involvement, reflecting a more subjective perception.
2. Evaluating Student-Writte a Kaggle competition on automated essay scoring @Crosslcy et al. 2024 Each essay is assigned a single holistic score by expert human graders.
Essays.
For this task, we utilize dataset of student essays from
3. Evaluating News Summaries. We use the SummEval benchmark (Fabri et al. 2020 which provides human evaluations of machine-generated summaries of news articles. Each summary is annotated by three expert annotators with prior experience in writing research paper summaries for academic conferences. We focus on the consistency and fluency, which respectively evaluate factual correctness and linguistic quality, two key aspects of summary reliability and readability.
4. Evaluating Translations. The WMT 2020 dataset (Freitag et al. 2021) provides expert human evaluations of machine translation outputs under the Scalar Quality Metric framework. We focus on the Chinese-to- Chinese-to-English translation task, where each translation is given a holistic quality score by three professional translators native in the target language.
5. Evaluating Chatbot Responses. We use the HelpSteer2 dataset (aag et al. 2024), which contains human feedback on Al-generated conversational responses. The annotations were collected
We observe high correlation between Spearman's p and Kendall's 7 the results. We only report r here for simplicity.
7

---

Published in Transactions on Machine Learning Research (02/2026)
from trained human raters the Scale AI platform, where each response evaluated by at least three annotators across five quality dimensions. Additional annotators were assigned when initial ratings showed high disagreement to ensure reliability. We focus on helpfulness, which measures informativeness and relevance of the response, and correctness, which evaluates the factual accuracy.
More statistics and pre-processing details for each dataset are provided in Appendix B
4.2
Data Processing
Data Utilization Rate (%)
70
60
50
Short Story (Complexity)
Chatbot Response (Correctness)
40
News Summary (Consistence
Studen: Written Essay
Number Generations k
Figure 5: Trade-off belween Data Utilization Rate and Number of Samples. Data utilization rate denotes the percentage of data points that have matched thinking traces within the l generations.
Filter Gold Standards The evaluation an LLM rater requires comparing its judgments with a reliable gold standard derived from human consensus. For datasets that already provide single aggregated human score per sample, such as Essay and HelpSteer2, we adopt this value directly our gold standard. For ratings from multiple annotators, such as HANNA, SummEval, and WMT-2020, construct the gold standard by first filtering for quality. We discard contentious examples where the standard deviation of human scores exceeds 1.0, and then define the gold standard as the median of the remaining ratings.
datasets that provide
Infer Thinking Traces As outlined in Scction 3 we use rejection sampling to infer thinking traces from the annotation datasets. The original annotation codebook from each source serves directly as the prompt.. For our generator models, we selected DeepSeek-RJ R1 and gpt-oss-120b traces. We prompt each model to generate sixteen (k 16) thinking traces for every sample and keep the first trace that aligns with the human annotator's final rating. Here, the selection of k is pragmatic trade- off between computational budget and the coverage of matched valid thinking traces, shown in Figure A sampling temperature of 1.0 is used during inference to encourage generative diversity.
they output complete thinking
4.3 Fine- Tuning Specialized LLM Rater
This experiment demonstrates how inferred thinking traces can be leveraged to train more effective auto- mated evaluators via reasoning-enhancedd supervised fine-tuning.
We use DeepSeek-R110022--peee 2025). The model is then fine-tuned on each individual task separately. During fine-tuning, the model is trained not only to predict the final rating but also to generale the entire thought process. This approach encourages the model to learn the underlying reasoning process that leads to a specific judgment. To prevent overfitting, we use carly stopping with a patience of 100 steps on a held-out validation set, selecting the model checkpoint that achieves the highest Kendall's correlation with human ratings. The performance of this final model is then evaluated by comparing its ratings against the human gold standard the test set..
and Deepseet-R--Distiillllaaas
base model (Guo al,
8

---

Published in Transactions on Machine Learning Research (02/2026)
Task
Original Model
SFT (DeepSeek Trace)
SFT (gpt-oss Trace)
Metric
ICC₂
MSE
ICC₂
MSE
ICCa
MSE
Short: Story (Complexity)
0.177
0.231
2.016
0.463*
0.604*
0.838*
0.460*
0.581
0.944
Short. Story (Engagement)
0.275
0.289
1.939
0.048
0.364
1.364
0.179
0.371
1.831
Student- Written Essay
0.202
0.233
1.947
0.322*
0.267
1.149*
0.240
0.244
1.387*
News Summary (Consistency)
0.141
0.186
2.904
0.276
0.319
3.476
0.243
0.331
2.970
News Summary (Fluency)
0.275
0.378
1.938
0.319
0.402
1.160*
0.306
0.498
1.200*
Translation
0.171
0.240
2.778
0,260*
0.348*
2.492
0.256*
0.314*
2,439*
Chatbot Response (Correctuess)
0.162
0.288
2.049
0,297*
0,403*
1.815"
0.263*
0.433*
1.897
Chatbot Response (Helpfulness)
0.176
0.274
1.969
0.264
0.422
1.723"
0.295*
0.466*
1.575*
Average
0.197
0.265
2.193
0.281
0.391
1.752
0.280
0.405
1.780
Table 1: Effect of Reasoning-Enhanee SFT on Human-LLM Rater Agrecment (DeepSee Qwen3-8B). For the metrics, higher is better for Kendall's and ICC3, while lower is beller for MSE. An asterisk (*) denotes a statistically significant improvement over the baseline (p < 0.05).
Results and Analysis The results in Table show that reasoning-enhanced SFT significantly improves the performance of the LLM rate Across all tasks, the models fine-tuned on inferred traces from both DeepSeek-RA and opt-oss-120b achieve significant gains over the original model. On average, reasoning- enhanced SFT improves Kendall's т from 0.197 to 0.281, a relative increase of 42.6%. Notably, the final performance of models trained traces from DeepSeek-R1 and gpt-oss-120b is highly comparable. Both sets of inferred traces, despite originating from different models, serve effective training data to improve the base rater. This demonstrates the universality of our framework over heterogeneous RLMs.
An outlier is the Short Story (Engagement) task, where both SFT models show a degradation in performance, particularly in Kendall's т. We hypothesize that this anomaly may stem from a potential lack of sufficient high-quality data points in the seed dataset for this specific dimension, making it difficult to learn consistent reasoning pattern.
Overall, despite some task-specific variations, the evidence supports that fine-tuning on inferred thinking traces is an effective method for enhancing the alignment of LLM raters with human judgments.
4.4 Refining Annotation Codebook
Task
Original Codebook
Refined (DeepSeek Trace)
Refined (gpt-oss Trace)
Metric
ICC₂
MSE
ICC;
MSE
ICC,
MSE
Short Story (Complexity)
0.428
0.497
1.116
0.512
0.571
0.634*
0.401
0.504
0.999
Short Story (Eugagement)
0.303
0.514
2.044
0.380
0.559
1.525
0.391
0.564
1.392
Student- Student-Written Essay
0.421
0.413
0.945
0.491
0.481
0.895
0.459*
0.433
0.957
News Summary (Consistency)
0.166
0.150
2.861
0.188
0.178
2.591
0.177
0.181
2.668
News Summary (Fluency)
0.142
(.1822
3.121
0.192
0.197
2.203
0.190
0.248
2.855
Translation
0.347
0.419
2.392
0.368
0.448*
1.942*
0.356
0.452
1,953*
Chatbot Response (Correctness)
0.354
0.451
1.920
0.370
0.485
1.724*
0.378
0.484
1.804
Chatbot Response (Helpfulness)
0.366
0.475
1.801
0.390
0.511*
1.642*
0.376
0.484
1.585*
Average
0.316
0.388
2.025
0.361
0.429
1.645
0.341
0.419
1.777
Table 2: Effect of Codebook Refinement on Human-LLM Rater Agreement. Each value in the table an average of the corresponding metrics calculated for each LLM rater.
Although SFT is effective, it is limited to LLMs for which weights are available to update. In more common cases, we have access to the state-of-the-art model through black-box APIs only. To steer their behavior, we need synthesize more effective prompts in the same way that we refine the annotation codebook for human raters. Following the procedure described in Sec we generate two refined annotation code- books for cach task using traces from DeepSeek-NL and gpt- pet-oss--20b. respectively. We test the effective-
4Due space limit, we put the results DeepSeex-R1-Diie Diitei------ttt---a-eaaa Appendix

---

Published in Transactions on Machine Learning Research (02/2026)
Task
Original Codebook
Refined (DeepSeek Trace)
Refined (gpt-oss Trace)
Short Story (Complexity)
0.613
0,700
0.771*
Short Story (Engagement)
0.883
0.810
0.823
Shudent-Writter Fssay
0.560
0.602
0.554
News Summary (Consistency)
0.405
0.501
0.417
News Summary (Fluency)
0,352
0.395
0.370
Translation
0.633
0.672*
0.689*
Chatbot Response (Correctness)
0.578
0.649*
0.628
Chathot Response (Helpfulness)
0.615
0.636*
0.604
Average
0.580
0.621
0.607
Table 3: Effect of Codebook Relinement on Rater Agreement among LLM Raters (ICCg).
Effect Codebook Refinement on Rater Agreement among LLM Raters
Original Codebook
Refined IDeepSeek Trace)
Refined (gpt-oss Trace)
Inter Reliabilit (ICC, Score)
0.2
Story (Complexity)
Short Story (Engagement)
Student-Written Essay
News Summary (Conslstency)
News Summary (Fluency) Fransaation
Chatbot Response (Correctness)
Chatbot Response (Helpfuimess)
Figure 6: Effect of Codebook Refinement on Rater Agreement among LLM Raters (IOC3),
ness of (hese refined codebooks on four powerful LLMs from different model providers: Claude-4-Sonnet, Hamini-2.5-Flasse GPT-5, and DeepSeek 1. We adopt a temperature of 1.0 and sampling probability mass cutoff of 0.95 for inference.
Improved Agreement with Human Judgments. Table shows the improved average agreement between four LLM raters and human judgments. On average, the codebooks refined by DeepSeek-RI traces improve human-LLM Kendall's 7 from 0.316 to 0.361 (a 14.2% improvement), and the codebooks refined by gpt-oss-1200 -12b traces improve 7 to 0.341 (a 7.9% improvement). showing a consistent improvement. Additionally, the ratings obtained by prompting more advanced LLMs with the refined codebook outperform those by fine-tuned smaller LLM raters (in Table [ This highlights the effectiveness and practicality of refining the annotation codebook.
Improved Inter-Rater Reliability. Crucially, the refined codebooks also improve inter-rater re- liability among different LLM raters. Table g shows that the average inter-rater agreement (ICCs) increases from 0.580 to 0.621 with the thinking traces inferred by DeepSeek-R1. The benefits most pronounced on tasks with relative objective criteria, such correctness of Chatbot Response and Thans- lation. Conversely, on highly subjective tasks like evaluating the engagement of Short Story, the refined codebooks can sometimes increase disagreementt between LLM raters. We hypothesize that this decrease is not necessarily a negative indicator: the original codebook may have induced high but superficial agreement by promoting a shared systematic bias among the LLM raters. By providing more nuanced guidance, the refined codebook might be reducing this shared bias, potentially leading the models to better reflect the natural and sometimes divergent distribution of human judgments.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Claude (Before): ... The subject story unrelated the prompt. It mostly describes mundane routine with some disjointed details... Then shifts abruptly to snow and vague dialogue about "gift" and "danger," but these are underdeveloped. ... The narrative lacks structure, coherence, or meaningful plot development. Elements that could add depth are too vague. The story reads like disconnected fragments with no complexity, <rating>1<<rating>
Claude (After): The story includes some elements: narrator routine, home setting, dialogue, and at- mosphere. ... But development is weak: abrupt jumps, unexplained characters, disconnected dialogue, no plot or character growth, abrupt ending. ... Though introduces multiple ideas (routine, relationships, mysterious "gift"), " they are fragmented and underdeveloped. ... Falls into the category of "few elements, underdeveloped, fragmented, disconnected." <rating>2<, /rating>
DeepSeek (Before): ... The subject story is different from the prompt, lacks magical elements. ... Contains surreal and disjointed aspects (time reversing, "gift," " ambiguous dialogue) that add some complexity. ... But overall lacks cohesion and depth, feels fragmented. <rating>3</rraing""
DeepSeek (After): ... Identifies elements: narrator, teacher, child; mundane events; home/snow setting; vague themes ("gift," " "danger"). Notes minimal depth, low coherence, abrupt time shifls, unresolved dialogue. Structure linear but disjointed. ... Introduress some elements but underdeveloped, no advanced techniques. Story is somewhat simple with disconnected elements. <rating>></rating>
Figure 7: Example thinking traces from two LLM raters on evaluating the complexity of a short story, before and after codebook refinement.
Qualitative Analysis of Annotation before and after Refinement. clear illustration of how the refined codebook improves rater alignment. Initially, using the original codebook to evaluate story (ground truth rating: 2), the two LLM raters produce divergent and incorrect scores. This disagreement stems from two core flaws in the original guide.
The example in Figure provides
First, it lacks a step-by-step procedure, leading to inconsistent evaluation paths. Claude, for example, immodiately makes a top-down judgment about the story's s "lack of structure, coherence, or meaningful plot development." In contrast, DeepSeek takes a bottom-up approach, first identifying "surreal and disjointed aspects" which views as a source of "complexity." The refined codebook solves this by introducing a clear, standardized process. As seen in the new traces, both raters now begin by systematically identifying the story's elements- such as "narrator routine, home setting, dialogue" (Claude) and "narrator, teacher, child; mundane events" (DeepSeek)-bellee before making a final judgment.
Second, the original scoring rubric is too abstract, forcing the models to weigh competing factors differently. The refined rubric corrects this by grounding its descriptions in concrete examples. This allows both raters to converge on the same nuanced conclusion. Claude's final justification now states that the story "falls into the category of 'fow clements, undcrdovcloped, fragmented, disconnceted," while DeepSeek similarly concludes it is "somewhat simple with disconnected elements." They both correctly assign a rating of 2, demonstrating that the refined codebook aligns LLM raters by standardizing the entire evaluation process.
4.5 Ablation Study
Ablation on Thinking Traces in SFT To disentangle the contribution of reasoning traces from the general benefits of SFT, we trained the same base model using only the final ratings from all available training data across all tasks. Table that the model trained exclusively on ratings suffered a performance drop across multiple tasks compared to the trace-augmented model, resulting in overall performance slightly worse than the baseline. This suggests that the model's inability utilize intermediate reasoning steps during inference hinders its evaluation capability. Thercfore, these results confirm that the generated thinking traces are an essential component for grounding the model's final judgments.
Ablation on Refinement Components To understand the individual contributions of our two-stage refinement process, we conduct ablation study using thinking traces from DeepSeek R1 three diverse
11

---

Published in Transactions on Machine Learning Research (02/2026)
Task
Original Model
SFT w/ traces
SFT w/o traces
Metric
ICC₂
MSE
ICCa
MSE
ICC₂
MSE
Short Story (Complexity)
0.177
0.231
2.016
0.463*
0.604*
0.838*
0.329
0.312
1.478
Short Story (Engagement)
0.275
0.289
1.939
0.048
0.364
1.364
0.166
0.182
3.101
Student-Written Essay
0.202
0.233
1.947
0.322
0.267
1.149*
0.283*
0.277*
2.036
News Summary (Consistency)
0.141
0.186
2.904
0.276
0.319
3.476
0.193
0.236
2.666
News Summary (Fluency)
0.275
0.378
1.938
0.319
0.402
1.160*
0.167
0.381
1.565
Translation
0.171
0.240
2.778
0.260
0.348
2.492
0.217
0.272
2.763
Chatbot Response (Correctness)
0.162
0.288
2.049
0.297
0,403
1.815*
0.103
0,140
2.742
Chatbot Response (Helpfulness)
0.176
0.274
1.969
0.264
0.422
1.723*
0.072
0.120
2,430
Average
0.197
0.265
2.193
0.281
0.391
1.752
0.191
0.240
2.348
Table 4: Ablation Study on the Effect of Thinking Traces in SFT.
Task
Original
Refined (Instruction)
Refined (Rubric)
Refined (Both)
Metric
ICC3
MSE
ICCs
MSE
ICCs
MSE
ICC₂
MSE
Story
0.303
0.514
2.044
0.372
0.551
1.867
0.380
0.537
1.701
0.380
0.559
1.525
Chatbot 0.354
0.451
1.920
0.374
0.480
1.827
0.359
0.453
1.836
0.370
0.485
1.724
Summary 0.166
0.150
2.861
0.168
0.145
2.405
0.185
0.195
2.602
0.188
0.178
2.591
Table 5: Ablation study on the effect of refinement codebook component. Story: Short Story (Engagement). Chatbot: Chatbot Response (Correctness). Summary: News Summary (Consisteney)..
tasks: Short Story (Engagement)) Chatbot Response (Correctness), and News Summary (Consistencyy.. To isolate the effect of each component, we create two separale versions of each codebook: one with only the task instructions refined and another with only the scoring rubric refined, and then assess Human-LLM rater alignment. Results in Table 5 show that both instruction refinement and rubric refinement enhance performance, though their relative impact depends on the specific weaknesses of the original codebook. For the Chatbot Response task, instruction refinement. yields larger gains, increasing Kendall's F from 0.354 to 0.374, compared to 0.359 from rubric refinement. proves more effective for News Summary, raising т from 0.166 to 0.185, whereas instruction refinement provides only marginal increase to 0.168. For the Short Story task, both refinements deliver comparably strong improvements. This analysis reveals that the optimal refinement strategy depends the original codebook's ' deficiencies: rubric refinement is most beneficial when the original lacks detailed rating scale (as with News Summary), while instruction refinement is critical when it lacks a step-by-step guideline, even with a detailed rubric (as with Chatbot Response). These findings validate our two-stage approach, highlighting its robustness in addressing distinct types of flaws in annotation guidelines.
contrast, rubric refinement
Task
Original Codebook
Refined (Thinking Tokens)
Refined (Post Hoc)
Metric
r
ICCa
MSE
ICCs
MSE
ICCa
MSE
Story
0.303
0.514
2.044
0.380
0.559
1.525
0.330
0.533
1.743
Chatbot
0.354
0.451
1.920
0.370
0.485
1.724
0.376
0.484
2.275
Summary
0.166
0.150
2.861
0.188
0.178
2.591
0.172
0.160
2.533
Table 6: Ablation Study on using post hoc explanation.
Ablation Using Post Hoc Explanations. A key prerequisite reconstructing effective thinking traces is full access to the RLM thinking tokens. However, this may not always be possible, especially for state-of-the-art model like gpt-5 and Gemini-2, 2amini-..5-Pro, which only expose a brief summary of their internal thinking. Therefore, we investigate whether our framework can be adapted for models that do not expose their internal thinking process, such as most proprietary LLMs. To address this, we test the usage of post hoc explanations as an alternative for pro-decision thinking traces. Data collection involves providing a generator model with the annotation target 2; and the gold standard human rating Vi, and prompting it to generate multiple post hoc candidate explanations for how one could arrive at that rating. These candidates are
12

---

Published in Transactions on Machine Learning Research (02/2026)
then filtered for quality: we remove all explicit rating information from the generated text and use another LLM to verify if the remaining reasoning still leads to the correct rating Vi- Due to multiple rounds of LLM calls, this method is more costly. The results in Table 6 show that this collection pipeline remains a highly effective approach. The codebook refined using post hoc explanations significantly outperforms the original codebook on both lasks. While its performance slightly lower than using native thinking traces for the Story task, it highly comparable for the News Summary (consistency) task. This finding demonstrates the robustness of our framework, suggesting that can be successfully extended to LLMs without explicit thinking tokens.
5 Discussion
We list several limitations and potential directions that could be addressed in future research:
Sampling Strategy Our rejection sampling approach is effective for inferring thinking traces when the generator model is capable enough to overlap with human judgment. However, its efficacy may be reduced in tasks with large alignment gaps. The model might struggle to generate trace that matches the human's final label, even when multiple candidate responses are sampled. To mitigate this, future research could explore sampling strategies that enable a broader coverage of possible thinking traces in the space. In addition, our current method infers thinking traces for an aggregated rating from multiple raters. Future work could instead infer thinking traces for each individual annotator, which may enrich the diversity of collected reasoning patterns and provide deeper insights into subjective variation across raters.
Relation with Automated Prompt Optimization While beyond the scope of this paper, codebook refinement, process can also be viewed as complemnentary approach to Automated Prompl Optimization (APO) techniques (Pryzant et al. 2023). Many APO methods iteratively refine the prompts, starting from simple seed that is hand-crafted (Ramnath et al.) 2025). Our method, in contrast, provides a semantically rich and data-driven starting point derived from inferred human thinking traces. This refined codebook could serve as a high-quality initial prompt, which can then be passed to established APO algorithms for further fine-tuning. This two-stage approach could bridge the gap between human-centric guideline design and automated optimization, leading to prompts that are both effective and grounded in human cognitive patterns.
Involving Human Annotators Another promising direction is to create a lightweight form of human- AI collaboration for future data annotation. This approach achicves a practical tradcoff between the high complexity of asking humans to write thinking traces from scratch and the need for an accurate thinking trace. Future workflows could first use model to generate several candidate thinking traces given human rating. A human rater would then simply select the trace that best reflects their own reasoning process. This bedlect-and-vallddtee method is significantly less demanding for annotators, but still yields the high-fidelity data needed to further strengthen the reliability of downstrrem LLM raters. Furthermore, inferred thinking traces have immediate applications for human-centered tasks. They can serve as valuable training resources to accelerate the onboarding of new human annotators, and the refined annotation codebooks could be used not only to guide LLMs but also to improve inter-rater reliability among human teams a hypothesis that we can test in future work.
6 Conclusion
Our work introduces scalable framework to infer the latent thinking traces of human annotators using reasoning language models. We demonstrate that these inferred traces serve as high-fidelily proxies for the reasoning paths of human judges across two complementary case studies. First, their integration into reasonddg-enhanned fine-tuning significantly improves the alignment of wein-wiight LLM raters to human ratings. Second, these traces can be used automatically refine annotation guidelines, which in turn boosts agreement among multiple LLM raters, well their agreement with human judgments. Our findings
13

---

Published in Transactions on Machine Learning Research (02/2026)
highlight the potential to unlock vast, label-only datasets, transforming them into transparent, reasoning- rich resources for building more reliable and interpretable LLM--based evaluators.
Acknowledgments
This work was in part supported by a Propelling Original Data Science (PODS) grant sponsored by Michigan Institute for Data & AT in Society (MTDAS) and Microsoft.
References
Maryam Amirizaniani, Flias Martin, Maryna Sivachenko, Afra Mashhadi, and Chirag Shah. Do LLMs exhibit human-like reasoning? evaluating theory of mind in LLMs for open-ended responses. ar [cs. CLJ, June
2024.
Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-599 -56, 2008.
Rewina Bedemariam, Natalie Perez, Sreyoshi Bhaduri, Satya Kapoor, Alex Gil, Elizabeth Conjar, Tkkei Itoku, David Theil, Aman Chadha, and Naumaan Nayyar. Potential and perils of large language models as judges of unstructured textual data. arXiv preprint arXix:2501.081 2025.
Adrien Bibal, Nathaniel Gerlek, Goran Muric, Elizabeth Boschee, Steven Fincke, Mike Ross, and Steven N Minton. Automating annotation guideline improvements using LLMs: A case study. pp. 129-144, 2025.
Samuel Carton, Anirudh Rathore, and Chenhao Tan. Evaluating and characterizing human rationales. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 9294-9307, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 11118653//11202.emmmbb// 747. URL http:://c..anlhooogg.ooogg
emnlp-main. 747/
Joseph Chee Chang, Saleema Amershi, and Ece Kamar. Revolt: Collaborative crowdsourcing for labeling machine learning datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, May 2017. ACM.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with Ilms, 2024. URL https://arxiv org/
hbs/2412.18922
Cyril Chhun, Pierre Colombo, Chloé Clavel, and Fabian M Suchanek. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. arXiv preprint XXi::2208.11646. 2022.
Cheorghe Comanici, Eric Bieber, Mike Schaekermann, Tee Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Scott Crossley, Perpetual Baffour, Jules King, Lauryn Burleigh, Walter Reade, and Maggie Demkin. Learning agency lab automated essay scoring 2.0. https://kaggle. wwt.///ooo//oooocnnnss.... 2024. Kaggle.
Alejandro Cuadron, Dacheng Li, Xingyao Wang, Siyuan Zhuang, Yichuan Wang, Luis G. Schroeder, Tian Xia, Aditya Desai, Ton Stoica, Graham Neubig, and Joseph E. Gonzalez. Swe-bench- periffed-o--ressonnnn-hhhhh
(revision edcal3e), 2024. URL htttps:/huugiinffacee httpaggiinaac..co/aaaaaa..oo. co/datasets/ AlexCuadron/SWBP Verra high-resule
Ishita Dasgupta, Andrew K. Lampinen, Stephanic C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. Language models show human-liks content
effects reasoning tasks, 2024. URL https://aaxt:: wtg//bb//20000705555555
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and By- ron C Wallace. Eraser: A benchmark to evaluate rationalized nip models. ar preprint or orv: 10t:1111.00 1.03299. 2019.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward rAnked FincTuning for generative foundation model alignment. arXiv [es.LG], April 2023.
Alexander Fabbri, Wojeicch Kryšcińsski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint or orx:::0007.166 12626,, 2020.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Ex- perts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460-1474, 2021.
Carlos Gódoz-Rodríguez and Paul Williams. A confederacy of models: A comprchensive evaluation of Ilms on creative writing. aixiv preprint arXiv:2310.08A4 2023.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuchao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on LLM--as-ajjudge. arXiv fes. CLJ, November 2024a.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey llm-as-a-judge. arXiv preprint 2111:24111.1554, 2024b.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Faleon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https://arxi:: org/ aalaaaaa/2250 2506. 04178
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. Xixi preprint arXim:2501.12 12948, 2025.
Andrew Halterman and Katherine A Keith. Codebook LLMs: Evaluating LLMs as measurement tools for political science concepts. arXiv CLJ, January 2025.
Fashuo Hsu, Martin Ugbala, Krishna Kumar Kookal, Zouaidi Kawtar, Nicholas I, Rider, Muhammad F Walji, and Kirk Roberts. Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction. arXio [cs. CLJ, April 2025.
Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. Evaluating creative short story generation in humans and large language models. arXiv preprint arXiv:2411.02316 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ei-Kishky, Aiden Low, Alec Helyar, Alek- sander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card, arXiv preprint 2rXiv:2412.16799, 2024.
V K. Chaithanya Manam, Dwarakanath Jampani, Mariam Zaim, Meng-Han WI, and Alexander J. Quinn. TaskMate: A mechanism to improve the quality of instructions in crowdsourcing, In Companion Proceed- ings of The 2019 World Wide Web Conference, New York, NY, USA, May 2019. ACM.
Jan- Can-Christoph Klic, Richard Eckart de Castilho, and Iryna Gurevych. Analyzing dataset annotation quality management in the wild. ar Xiv ClJ,, July 2023.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Terry K Koo and Mae Y Li. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chöropractic medicine, 15(2):155-16; 2016.
Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: Limi- tations of LLM-as-a-judge without human grounding. arXiv [cs. CLJ, March 2025.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: NLG evalua- tion using GPT-4 with better human aligmment. arXiv fes. (es,C March 2023.
Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulié, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators. arXiv fes. CLJ, March 2024.
V K Manam and Alexander Quinn. Winglt: Efficient refinement of unclear task instructions. Proceedings of the AAAI Conference on Human Computation and Croudsourcing, 6:108-116, June 2018
Justus Mattern, Sami Jaghouar, Manveer Basra, Jannik Straube, Matthew Di Ferrante, Felix Gabriel, Jack Min Ong, Vincent Weisser, and Johannes Hagemann. Synthetic-l: Two million collaboratively ai/blog/
generated reasoning traces from deepseek-rl, 2025. URL
syathetic-1-release
Han Meng, Yitian Yang, Yunan Li, Jungup Lee, and Yi-Chieh Lee. Exploring the potential of human-LLM synergy in advancing qualitative analysis: A case study on mental-illness stigma. arXiv [cs.HC), May 2024.
Philipp Mondorf and Barbara Plank. Comparing inferential strategies of humans and large language models in deductive reasoning. arXiv CLJ. February 2024.
Cónal O'Connor and Hélène Joffe. Intercoder reliability in qualitative rescarch: Debates and practical guidelines. Social and Personality Psychology Compass, 14(2)=e1251, 2020.
Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarin, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. 2025.
https: hhaps://hugginggace. http:///wwggggaeeeecoco/aaaaa//s
Vivek Krishna Pradhan, Mike Schaekermann, and Matthew Lease. In search of ambiguity: A three-stage workflow design to clarify annotation guidelines for crowd workers. Front. Artif. Intell., 5:828187, May 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with' gradient descent" and beam search. arXiv preprint arXiv:2005.00// 2023.
Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Jimin Huang, and Qianqian Xic. Finol: On the transferability of reasoning enhanced llms to finance, 2025. URL
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, et al. A systematic survey of automatic prompt optimization techniques. arXiv preprint Xri::2250.. 16923, 2025.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell I. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. arXiv LCs. December 2023.
Saurabh Srivastava, Sweta Pati, and Ziyu Yao. Instruction-tuning LLMs for event extraction with annotation guidelines. arXiv fcs. CL,, February 2025.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. arXiv fes. C..), August 2025.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. DART-math: Dificuutt- rejection tuning for mathematical problem-solving. arXie [cs. CLJ, June 2024.
Maya Grace Torii, Takahito Murakami, and Yoichi Ochiai. Expanding horizons in HCI research through LLM-driven qualitative analysis. arXiv jes. HC), January 2024.
Alicia Y Tsai, Adam Kraft, Long Jin, Chenwei Cai, Anahita Hosscini, Thibai Xu, Zemin Zhang, Lichan Hong, Ed H Chi, and Xinyang Yi. Leveraging LLM reasoning enhances personalized recommender systems. arXiv [es.IR], July 2024.
Somin Wadhwa, Silvio Amir, and Byron C Wallace. Invostigating mysterics of CoT-augmented distillation. arXiv fes. CL), June 2024.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Nar- simhan Sreedhar, and Oleksii Kuchaiev. Helpsteer Open-source dataset for training lop-performing reward models. Advances in Neural Information Processing Systems, 37:1474-1501, 2024.
Joel P Wiebe, Rubaina Khan, Samantha Burns, and James D Slotta. Qualitative research in the age of LLMs: A numan-in-the-looy approach to hybrid thematic analysis. In Proceedings of the International Conference of the Learning Sciences, pp- 1 1123 1131. International Society of the Learning Sciences, June 2025.
Ziang Xiao, Xingdi Yuan, Q Vera Liao, Rania Abdelghani, and Pierre-Yver Oudeyer. Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. arXiv fes. CL), April 2023.
Yao-Ching Yu, Tsun-Han Chiang, Cheng- Weeng-Wei Tsai, Chien-Ming Huang, and Wen-Kwang Tsao. Primus: A pi- oncering collection of open-source datasets for cybersccurity Ilm training. arXiv preprint arXiv:2502.1111 11191, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. STaR: Bootstrapping reasoning with reasoning. arXiv LG), pp. 15476 15488, March 2022.
Lianmin Zheng, Wei Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595 46623, 2023.
Xin Zhou, Kisub Kim, Ting Zhang, Martin Weyssow, Luis F Gomes, Guang Yang, and David Lo. An 1lm- as-judge metric for bridging the gap with human evaluation in se tasks. arXiv preprint arXiv:2505.2085, 2025.
A Additional Tables
Generalizability across Base Architectures To demonstrate that our framework is not limited to specific model families, we conducted experiments using DeepSSek-RR---istill-lllaa
as the base model. Table 7 reports the Human-LLM, agreement metrics before and after applying our reasoning-enhanne enhanced SFT. The results show consistent improvements across the majority of tasks, confirming that our method effectively generalizes to Llama-based architectures.
Sensitivity to Trace Quantity We further investigated the robustncss of our codebook refinement stage regarding the number of thinking traces used. Table S presents an ablation study on the Short Slory (Engagement) task, varying the number of traces for instruction optimization (Ninet) and rubric generation (Nrubrie). We observe that performance improves as more traces are utilized, platcauing around our selected configuration (10/50). Crucially, even minimal trace settings (5/20) outperform the original codebook, indicating that the method is data-efficient and not overly sensitive to hyperparameter tuning.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Task
Original Model
SFT (DeepSeek Trace)
Metric
ICCa
MSE
T
ICC₂
MSE
Short Story (Complexity)
0.324
0.419
1.552
0.315
0.419
1.248
Short Story (Engagement)
0.305
0.312
1.199
0.292
0.336
1.108
Student-Written Essay
0,075
0.087
4.475
0.237*
0.260*
1,553*
News Summary (Consistency)
0.059
0.0418
1.782
0.128
0.146
4.018
News Summary (Fluency)
0.159
0.203
1.792
0.213
0.359
1,203*
Translation
0.174
0.143
2.145
0,236*
0.173
2.028*
Chatbot Response (Correctness)
0.063
0.072
2.319
0.154*
0.200*
2.046*
Chatbot Response (Helpfulness)
0.053
0.070
2.434
0.155*
0.164*
2.228*
Average
0.151
0.168
2.212
0.216
0.257
1.929
Table 7: Effect of Reasoning-Euhen Enhanced SFT on Human-LLM Rater Agreement (DeepSeek-R1-Distill) Llama-8B). For the metrics, higher is better for ICC3 and Kendall's т, while lower better for MSE, An asterisk (+) denotes statistically significant improvement over the baseline (p 0.05).
Sctup (Ninst / Nrubric)
ICC
MSE
Original Codebook
0.303
0.514
2.044
5 / 20
0.370
0.548
1.622
10 50 (Selected)
0.380
0.559
1.525
20 60
0.396
0.549
1.737
30 70
0.363
0.537
1.857
Table 8: Ablation study on the sensitivity of codebook refinement to the number of thinking traces used. We vary the number of traces used for instruction improvement (Ninst) and rubric improvement (Nrubrie) on the Short Story (Engagement) task. All tested configurations yield better performance than the baseline.
Effectivenesss on Generalist Models To address concerns regarding the necessity of specialized reason- ing models, we evaluated our framework on GPT-4o, a strong generalist model without native 'thinking" output. Table D reports the SFT performance, showing that despite GPT-4o lacking native reasoning traces, our alignment method significantly improves its agreement with human raters (Average improves from 0.197 to 0.237). This trend holds for codebook refinement well; shown in Table 10 utilizing traces from GPT-4o yields consistently better evaluation guidelines than the original codebook (Average increasing from 0.316 to 0.339). These results confirm that the benefits of our framework extend to general-purpose LLMs.
Task
Original Model
SFT (DeepSeek)
SF' (gpt-oss)
SFT (GPT-4o)
Metric
ICC,
MSE
ICC,
MSE
ICC,
MSE
ICC,
MSE
Story (C)
0.177
2.016
0.463*
0.604
0.838
0.460
0.581
0.944
0.326
0.457%
1.354
Story (E)
0.275
0.289
1.939
0.048
0.364
1.364
0.179
0.371
1.831
0.118
0.421
1.910
Essay
0.202
0.233
1.917
0.322*
0.267
1.149*
0.240
0.241
1.387
0.259
0.291
1.198
Summary (C)
0.141
0.186
2.904
0.276
0.319
3.476
0.243
0.331
2.970
0.191
0.302
2.388
Summary (F)
0.275
0.378
1.938
0.319
0.402
1.160*
0.306
0.498
1.200*
0.3/11
0.389
1.627
Translation
0.171
0.256*
2.404
Chatbot (C)
0.162
0.288
2,049
0.297*
0.403*
1.815*
0.263*
0.433
1.897
0.187
0.311
2.085
Chatbot (H)
0.176
1.909
0.264
0.422
1.723*
0.295
0.460
1.575*
0.252
0.430
1.701%
Average
0.197
0.265
2.193
0.281
0.391
1.752
0.280
0.405
1.780
0.237
0.360
1.833
Table 9: Effect of Reasoning-Enhancee SFT on Human-LLM Rater Agreement (DeepSeek-R1-Distti Qwen3-8B) including GPT-40 (non-reasoning LLM).
18

---

Published in Transactions on Machine Learning Research (02/2026)
Task
Original Codebook
Refined (DeepSeek)
Refined (gpt-oss)
Refined (GPT-40)
Metric
ICCg
MSE
ICCg
MSE
ICC₂
MSE
ICCa
MSE
Story
0.428
1.116
0.512
0.571
0.634
0.401
Story (E)
0.303
0.514
2.044
0.380
0.539
1.525
0.391
0.564
1.392
0.359
0.542
1.956
Essay
0.421
0.413
0.945
0.491*
0.481
0.895
0.459*
0.433
0.957
0.490*
0.492*
0.892
Summary (C)
0.166
0.150
2.861
0.188
0.178
2.591
0.177
0.181
2.668
0.169
0.159
2.901
Summary (F)
0.142
0.182
3.121
0.192
0.197
2:203
0.190
0.248
2.853
0.180
0.169
2.863
Translation
0.317
0.368
0,448
1.942"
0.356
0.452
1,953
0.362
0.455
1.882*
Chatbot (C)
0.354
0.451
1.920
0.370
0.485
1.724"
0.378
0.484
1.804
0.379
0.481
1.798
Chathot (H)
0.475
0.390
0.511
1.642"
0.376
0.484
Average
0.316
0.388
2.025
0.361
0.429
1.645
0.341
0.419
1.777
0.339
0.410
1.889
Table 10: Effect of Codebook Refinement on Human-LLM LLM Rater Agreement. Each value in the table is an average of the corresponding metrics calculated for each LLM rater.
B Dataset Details
HANNA The HANNA dataset (Human-ANnotateed Narratives for ASG evaluation) (Chhun al., 2022 is benchmark of 1056 stories generated from 96 prompts. Each story annotated by 3 native English speakers from Amazon Mechanical Turk with Masters Qualifications along six defined evaluation criteria- engagement, complexity, surprise, relevance, coherence, and empathy, resulting in 19008 annotations, Among these, 10245 annotations exhibit a standard deviation lower than 1.0 across raters, which we regard as effective and reliable samples.
Essay The Essay dataset consists of student essays from Kaggle competition on automated essay scor- (Crossley et al. 2024). The 17307 student essays were annotated with holistic quality score by expert raters. We regard all of them as effective samples.
SummEval The SummEval dataset is large-scale benchmark designed to evaluate summarization mod- elsbFabbri et al. 2020). The summaries were generated by 23 sunmarization models trained on CNN ONN//aily- Mail news dataset. Each summary is evaluated by crowd-sourced annotators from Amazon Mechanical Turk and three expert annotators, whore two had written academic papers on summarization for conferences, and one had completed a senior thesis on the topic. For our analysís, we focus on the expert annolations and retain only the effective samples, defined as those with a standard deviation below 1.0 across expert ratings. This filtering yields 5645 reliable samples out of 6400 total.
WMT-2020 The WMT20 Metrics Shared Task dataset a benchmark for evaluating machine translation systems |rritag al. 2021 contains system outputs across multiple language pairs (e.g., Chinese English, English to German). Each translation was evaluated on both MQM (Multidimensional Quality Metrics) and SQM (Scalar Quality Metrics) by human annolators. In this work, we focus on the Chi- nese English language pair and use the SQM (Scalar Quality Metric) annotations, where each translation is assigned three professional translators native in the target language of overall quality. The dataset contains 19,950 translation pairs, of which 6,934 exhibit a standard deviation below 1.0 across raters. We regard these as reliable samples and use them for training and evaluation.
HelpSteer2 The HelpSteer2 dataset is preference dataset for training reward models that align large language models with human preferences Wang et al.] 2024). It contains 10,681 prompts, cach paired with two responses, and every response is annolated on five dimensions: correctness, helpfulness, coherence, com- plexity, and verbosity. To ensure annotation reliability, each response is initially rated by three annotators, with two additional annotations collected when the disagroement among the first three annotations is high. In total, the dataset comprises 21,362 annotated responses, all of which we treat as effective samples. For our experiments, focus on the dimensions of correct and helpfulness.
Wann
19

---

Published in Transactions on Machine Learning Research (02/2026)
C SFT Details
We trained all models using LoRA and chose AdamW as the optimizer. The hyperparamcters are: learning rale 5e-5 (warmup 100 steps); batch size 128, weight decay 0.01, LoRA
32, LoRA 16.
Prompts
Instruction refinement
<instruction
You will be given list of analysis on scoring the {dimension) of {generation_tyen task to extract a concrete and concise step-by-step instruction from the analysis that could be easily followed by an annotator without any training, Based on your extraction, improve the codebook>... htr//orgialloodebbelee
Your
Only improve the instruction part (e.g. thinking path to follow), do not change the rubric part (e.g. criteria). Your final annotation guidelines should be put into coddbbok>..<//codcboot tag. </instruction>
original annotation guidelines <<orighal__oodebookk..o
codebook>>
[original_codebook]
</original_codeean codebook>
<analysis>
fanalysis}
</analysis>
Critique extraction from CoT
<instruction>>
You will be given an analysis on scoring the {dimension} of a (generation inpeerationttpel.. Your task is list all the critique from the analysis that help evaluate the [dimension] of the igeneration_typel Your response should be list of sentences, each a new line, without bullet points. Your response should not include any other text. </instruction>
<analysis> {cot} </analysis>
20

---

Published in Transactions on Machine Learning Research (02/2026)
Rubric refinement.
<instruction
You will be shown a list of diverse critiques on the {dimension} of {generation (etteraattootttp))... Each of specific rating level. Your task is to summarize the critiques into a
the critique is corresponding single rubric, based on the pattern of the different rating levels. Based on the new critcria, improve the original annolation guidelines. Only modily the crilerion part. Your final annolation guidelines should be put into sooocbool...<<<oddeeooll tag. </instruction>
<original_ccoeeoook> codebook>
foriginal_codeboon
</original codebook>
<critiques>
(critiques}
</critiques>
E Complete Examples of Annotation Codebooks
A complete list of codebooks before and after refinement provided in our anonymous codebase: https:
//anonymous 4open.
Here show complete refined ver-
sion of Figure 4
21

---

Published in Transactions on Machine Learning Research (02/2026)
Read the Materials Thoroughly Start by reading the prompt, the human story, and the subject story. Note that the subject story is the one being rated, not the human story. First, read the prompt to understand the context. Read the Human Story to get a sense of a complete narrative for comparison, relevant. Read the Subject Story thoroughly, as this the story you need to evaluate.
Step-by-Step Rating:
Look for key elements such characters, events, plot, themes, setting ...
Evaluate if the identified elements are developed, interconnected, and coherent
Consider whether the story follows structured progression (linear non-linear).
Stories with minimal elements or undeveloped content are simpler. Stories that attempt multi-layered exploration of ideas, even not fully realized, should be evaluated as more complex.
Select the Rating
1. The story very simple, lacks depth, and introduces no intricate or developed elements.
2. The story is somewhat simple, with few elements that are underdeveloped,, fragmented, or discon- nected.
3. The story demonstrates some complexity, introducing multiple elements, but lacks depth, develop- ment, or coherence in integrating these elements.
The story is mostly complex, weaving together several developed elements with minor areas that could be expanded refined.
5. The story is highly elaborate and complex, integrating multiple layers (c.g., detailed world- horlddbuilding, character depth, advanced structure, thematic sophistication) in a cohesive and interconnected way.
Consider overall structure and presentation
Are there abrupt shifts, confusing elements, or fragmented storytelling that make harder engage with the story?
Does the story have vivid details or emotional resonance that enhance engagement. despite structural issues?
Handle specific cases
If the story cut off mid-sentence, rate as if it ended just before the unfinished sentence.
If the story is not relevant to the prompt, focus only on how engaging it is ss-irrelevance affects the Relevance criterion, not Engagement.
Keep your focus
Rate solely based on the Engagement criterion, even if the story lacks relevance to the prompt.
Do not let your assessment be influenced by the Human Story except as potential comparison of narrative structure or clarity.
22


Reviews:
[{"review_id": "rLu1rcJ4TO", "reviewer": "Reviewer_RgCH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors try to improve the reliability of LLM judgments by analyzing the thinking processes of reasoning models. Although the authors performed simulations on different datasets, more simulations or clarifications are still needed.\n\n__Strengths__\n\n1. The authors employed various datasets, and also carried out some ablation studies.\n\n2. The framework proposed in the article is intuitive and easy to understand.\n\n__Weaknesses__\n\n1. The framework in the paper is based on the rejection sampling method, but there is a lack of discussion regarding the rejection ratio.\n\n2. The paper mainly used DeepSeek-R1 and gpt-oss-120b, and the authors could consider adding experimental results of non-reasoning LLMs as a comparison.\n\n3. Temperature could also affect the efficiency and conclusions, especially since the authors used the same temperature parameter for different tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "The goal of this paper is to improve the results of RLMs, but how do they perform on these tasks compared to other LLMs? This has not been verified through experiments in the paper.\n\nThe idea of this paper is intuitive, but more experiments may be needed to demonstrate the generalizability of its conclusions, such as considering more parameters and prompts.", "audience_yn": "Yes", "audience_explanation": "As the authors mentioned, LLMs are now widely used in evaluation tasks, so it is important to improve their reliability.", "requested_changes": "1. The authors need to add more experiments and explanations, referring to the weaknesses mentioned earlier.\n\n2. The main results of the paper are presented in similar table formats, and the authors could consider using other forms to present and compare the results more effectively.\n\n3. In addition to indicating cases of \"statistically significant improvement over the baseline\" in the tables, the authors could also highlight cases where performance has gotten worse.\n\n4. It would be better to modify Figure 3, as the relationship between \"Candidate Thinking Traces\" and the left and right columns is unclear (for example, arrows could be added between the blocks).", "broader_impact_concerns": "This paper involves very few ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BCMkCADLfA", "reviewer": "Reviewer_sFuU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks at improving LLM-as-a-judge evaluation setups by utilising reasoning traces of reasoning-based LLMs for SFT. The authors investigate whether these silver-standard reasoning traces, used as a proxy for human reasoning traces, can improve LLM judges in their reliability. The authors present two setups - one is based on SFT using the reasoning traces for an improved judge, and the other method is using the reasoning traces for improving the annotation codebook. The authors show empirical improvements in inter-annotator agreement metrics for both these methods in this work. However, they do not conduct some important experiments that are crucial for clearly supporting their methods. They also do not provide enough description of related works, especially related to the use of reasoning traces in improving LLM performance across many tasks. I also believe the presentation of the paper may be improved, as there are many points where the Figures being referenced in the text are not close to where they are referenced.\nI also do not believe the proposed method solves the issue at hand. The main reason for the use of LLM as a judge is the high cost of human evaluators. The paper looks at improving LLM evaluations, but requires significant human evaluation as a starting point for SFT to be effective. Further, an added cost of training a task-specific LLM evaluator is added to the LLM as a judge setup, making it less lucrative and difficult to reproduce for future researchers.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The authors present results on fine-tuning the base LM on the reasoning traces for improved reliability in the evaluations. However, it is not clear from these experiments whether the improved results are due to the inclusion of reasoning traces in the training setup or due to training itself. The authors need to disentangle this contribution of the SFT process, for example by training on just the human labels without the traces.\n2. The various results tables show that there is not a statistically significant improvement in performance for close to half of the tested setups. This undermines the framework's application in untested domains.\n3. The authors claim - \"By providing more nuanced guidance, the refined codebook reduces this shared bias, causing the models to better reflect the natural and sometimes divergent distribution of human judgments\". There is no evidence to support this claim. I hypothesise the opposite to be true as more detailed guidelines lead to an increased shared bias towards the aspects mentioned in the guidelines.\n4. All results are shown on a single base model. It is not clear from these experiments if the setup is generalizable and can be applied to other choices of LLM judges.\n5. The authors make many arbitrary choices (such as the selection of k = 16 generations, picking the 1st accepted reasoning trace, choice of a single 8B model for SFT). The reasoning behind these choices is not given, and it is important to understand the impact that different choices may have had.", "audience_yn": "Yes", "audience_explanation": "LLMs are increasingly used as proxies for human evaluators on many machine learning tasks. Improvements to LLM as judge systems are of great interest to the ML research community.", "requested_changes": "1. Ablation on training using human labels only without reasoning traces - critical to securing your recommendation\n2. Experiments on other choices of base LLM - critical to securing your recommendation\n3. Evidence for the claim mentioned in point 3 above or a change in how it is framed- critical to securing your recommendation\n4. experiments showing the impact of the choice of k = 16, how the reasoning trace is selected - strengthen the work", "broader_impact_concerns": "NA", "additional_comments": "1. What happens in cases where none of the generated reasoning traces match with the human label? Further, could you provide an experiment analysing what happens if only those cases where the reasoning consistently matches with the human label, for example, atleast 50% (or any other choice) of the reasoning traces match with the human label.\n2. The authors claim - For a model to reach the correct human label yi, its reasoning is more likely (but not guaranteed) to be a plausible approximation of the latent human thought process - it does not really need to be an approximation, for the tasks studied, it can be totally different to the interpretation of the human judge as well but still lead to the same outcome. This is evidenced by the fact that there are n = 0 to 16 valid reasoning traces that are produced by the LLM. Hence, how can each of them be an approximation if all of them are different?", "extra_scores": {}, "extra_text": {}}, {"review_id": "gEEMnemTl6", "reviewer": "Reviewer_okWo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary\nThe paper presents a framework that learns from inferred reasoning trace to improve the LLM as an automatic rater on downstream tasks. Key features include: 1) using rejection sampling to extract the chain of thought (CoT) tokens from models on LLM rating tasks; 2) the CoT can be seen as a proxy of the thought process of human raters when the predicted label agrees with human and thus can enhance LLM performance through SFT; 3) using the LLM summarize the inferred reasoning trace to rewrite the original code book (rating instruction and rubrics) to further improve the LLM rater performance.\n\n# Strength\n- The paper tackles an important task of using LLM as an automatic evaluator on open ended text generation tasks where the evaluation can be difficult and requires significant human efforts.\n- The refinement of annotation code book is novel: it uses the inferred trace to rewrite the annotation code books and effectively improve the LLM performance. This method is also applicable to proprietary models which are black box LLM APIs.\n- Extensive experiment results show the effectiveness of the proposed method.\n\n# Weakness\n- The method in Sec 3.1 is a standard method of distillation of reasoning via rejection sampling as mentioned in Section 2. The paper follows the existing method where LLM is used to sample a number of chain of thoughts and the ones that reach the correct final results are kept at the end for SFT. Therefore, the novelty of this method is weak.\n- The codebook refinement process is empirical and there is no guarantee that the rewritten instruction and rubrics based on inferred thinking trace (not the actual human thinking) can improve the performance.\n- Sec 3.2 Refining Annotation Codebooks: the choice of sampling 10 / 50 traces seems arbitrary.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The proposed method is evaluated with experiments of SFT on thinking trace and codebook refinement using DeepSeek R1 and GPT OSS trace on various text generation tasks. The results show that both SFT on thinking trace and codebook refinement improve the LLM rater on metrics such as ICC_3, \\tau, and MSE, which suggests better agreement with human rater and inter-rater reliability. Evaluation of codebook refinement for proprietary models such as Claude also suggests the proposed method is applicable to non-open-weights models.", "audience_yn": "Yes", "audience_explanation": "The paper tackles the task of improving the LLM performance as an automatic rater of downstream tasks. The proposed method can effectively improve the alignment of LLM automatic rating with human rating. The proposed method is easy to adopt and is applicable to both open-weight and proprietary models. This work can potentially be useful for broader LLM research on open ended text generation tasks where the labeling requires significant human efforts.", "requested_changes": "See weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)