Text2Grad: Reinforcement Learning from Natural Language Feedback
Authors: Hanyang Wang
Lu Wang
Chaoyun Zhang
Tianjun Mao
Si Qin
Qingwei Lin
Saravan Rajmohan
Dongmei Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Natural Language Feedback
Fine-Grained Policy Optimization
Reinforcement Learning for Language Models
Primary area: reinforcement learning
TL;DR: Text2Grad converts free-form natural-language critiques into span-level gradients, enabling fine-grained and actionable alignment of language models beyond scalar-reward RLHF.

Abstract: Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow, opaque learning. Recent work augments RL with textual critiques through prompting or reflection, improving interpretability but leaving model parameters untouched. We introduce **Text2Grad**, a reinforcement-learning paradigm that *turns free-form textual feedback into span-level gradients*. Given human (or programmatic) critiques, Text2Grad aligns each feedback phrase with the relevant token spans, converts these alignments into differentiable reward signals, and performs gradient updates that directly refine the offending portions of the model's policy. This yields precise, feedback-conditioned adjustments instead of global nudges. Text2Grad is realized through three components: (1) a high-quality feedback–annotation pipeline that pairs critiques with token spans; (2) a fine-grained reward model that predicts span-level reward on answers while generating explanatory critiques; and (3) a span-level policy optimizer that back-propagates *natural-language gradients*. Across summarization, code generation, and question answering, Text2Grad consistently surpasses scalar-reward RL and prompt-only baselines, providing both higher task metrics and richer interpretability. Our results suggest that natural-language feedback can serve not only as explanations, but also as actionable training signals for fine-grained alignment. The code for our method is available at *[https://github.com/microsoft/Text2Grad](https://github.com/microsoft/Text2Grad)*.

Published as a conference paper at ICLR 2026
TEXT2GRAD: REINFORCEMENT LEARNING FROM NATURAL LANGUAGE FEEDBACK
Hanyang Wang Si Qin
La Wanget
Chaoyun Zhang
Tianjun Mao Dongmei Zhang
Qingwei Lin?
Saravan Rajmohan2 Fudan University
Universiity of Chicago
Miicosoftt
ABSTRACT
Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow, opaque learning. Recent work augments RL with textual critiques through prompting or reflection, improving interpretability but leaving model parameters untouched. We introduce TEXT2GRAD. reinforcement-leaniie form textual feedback into span-level gradients. Given human (or programmatic) critiques, TEXT2GRAD aligns each feedback phrase with the relevant token spans, converts these alignments into differentiable reward signals, and performs gradient updates that directly refine the offending portions of the model's policy. This yields precise,
paradigm that turns free-
adjustments instead of global nudges. TEXT2GRAD is realized through three components: (1) high-quality feedback-annotation pipeline that pairs critiques with token spans; (2) a fine-grained reward model that predicts span-level reward on answers while generating explanatory critiques; and (3) span-level policy optimizer that back-propagates natural-lange natural-language gradients. Across summarization, code generation, and question answering. TEXT2GRAD consistently surpasses scalar-reward RL and prompt-only baselines, providing both higher task metrics and richer interpretability. Our results suggest that natural- language feedback
only as explanations, but also actionable training signals for fine-grained alignment. The code for our method is available at http::/ggtuab..com//iios////////
INTRODUCTION
Free-form natural language feedback is
abundant in real-world applications (Zhang)
Query:
Please Sumerstrre tollwinggoott juzt sent the completed uneditd
ett al.] 2024a). Users leave suggestions in
(Ina first sime suthor witha son pageugg www.gruguuuuuhueueeooooooo tred finding idoeie Trajectory:
reviews, developers comment on code pull requests, and customers critique responses from virtual assistants. Unlike scalar ratings or preference scores, this form of feedback inherently rich and expressive. It not only pinpoints what is correct or incorrect in an output but also explains why, providing ac- tionable guidance for improvement.
Sooker Rewards
Intormaion sbout ther warkiz that Span tewelteondk
FC Gen baub
PPO
Traditional
PPO
Updett
TEXT2ORAD
Policy Update
Despite its ubiquity and usefulness, most learning paradigms fail to fully leverage hu-
Figure I: Comparison of PPO and TEXT2GRAD
man feedback. Reinforcement learning from human feedback (RLHF) has become the dominant method for aligning large language models (LLMs) with human preferences (Stiennon al. 2020
Ouyang et al.] 2022) Bai et al. [2022] Rafailov et al.] (2023) Shao et al.) 2024h. RLHF typically reduces preference comparisons to scalar rewards and optimizes policies via PPO (Schulman et al. 2017 or
DPO (Raaillov et al. 2023). While effective for improving helpfulness and safety, this scalarization discards fine- -rainrdd token-level signals about what was right or wrong and where-leadee to
First author. Email: tanyangw@uchican edu. edu. 1 *Corrsponning author. Email: w1 uê0i crosoft.com.

---

Published as a conference paper at ICLR 2026
imprecise credit assignment, slower convergence, and reduced interpretability (Casper al. 2023
Wu et al. 2023;
Raschka 2024).
An alternative line of research maintains feedback in its natural language form. Methods such as ReAct (Yao et al. 2023) and Reflexion (Shinn et al. 2023) prompt the model to reflect on its outputs
(Zhang al. 2024b), generate critiques, and use them to self-correct in subsequent steps (Zhang et al.) 2025). These approaches are inspired by how humans operate in open-ended tasks through reasoning. explanation, and dialogue, rather than relying on numerical rewards (Wei et al.) [022] Nakano et al.
2021; Zhang el al., 2024c). Natural language feedback in this context improves transparency and sometimes leads to better task performance. However, because these methods leave model parameters
frozen, the feedback is not internalized, requiring repeated corrections and rendering it ephemeral
(Fernandes al. 2023
Pan et al. 2024 Sharma et al. 2024).
In this paper, we propose TEXT2GRAD, novel framework that transforms free-form textual feedback into actionable gradients for policy optimization. As shown Figure unlike prior work that compresses feedback into scalar rewards or applies textual critiques only inference time, TEXT2GRAD brings feedback into the training loop. Given a human or programmatic critique, our method aligns feedback clauses with relevant output token spans, converts these alignments into span-level reward signals. and computes a natural language gradient. This gradient is then used to perform policy updates that precisely adjust the parts of the model responsible for the error. The result is more targeted, efficient, and interpretable learning.
TEXT2GRAD is built complete pipeline for learning from text. First, we construct high-quality annotation pipeline that uses GPT-4o to label model outputs with textual feedback and span-level
critiques, following recent work on automated feedback generation (Lee al. 2023 Liang al.) 2024). Second, we train a unified reward model inspired by generative reward modeling (Mahan al. 2024) that jointly generates natural language critiques and structured span-level reward maps in single autoregressive sequence. Third, we apply span-level policy optimization using a variant of PPO that integrates these fine-grained reward signals, drawing on advances in token-aware credit assignment (Chen et al. 2024a) and text-based gradients (Yuksekgonul et al. 2024).
We evaluate TEXT2GRAY across various domains, including summarization (Scheurer et al. 2023 code generation (Xu et al. (025), and open-domain question answering (Cui et 20233.. Our results demonstrate that by converting language into gradients, TEXT2GRA AD not only achieves superior performance but also offers improved interpretability and sample efficiency, establishing natural language feedback as a powerful direct training signal. These results suggest that natural language feedback can be more than an interpretability tool: It can be converted into principled gradients to train more capable and aligned models. Overall, this paper makes the following contributions.
We formulate the problem of learning from natural language feedback via gradient- based optimiza- tion, and present TEXT2GRAD as the first complete framework to address it.
We develop a scalable annotation pipeline and a unified reward model that together produce span-level rewards and explanatory critiques, yielding interpretable, span-level supervision.
We show that TEXT2GRAD outperforms strong www--rewwwdd-eeeeeeee and prompt-based baselines in summarization, code generation, and question-answert benchmarks.
TEXT2GRAD demonstrates that natural language feedback, when properly aligned and grounded, can serve direct training signal rather than just auxiliary guidance, opening new path for building language models that learn from human- like supervision.
2 RELATED WORK
RLHF with scalar rewards Reinforcement learning from human feedback replaces supervised
labels with a reward model trained on pairwise human preferences (Christiano et al.) 2017 Ouyang ct al. 2022). The reward is a single scalar, and policy optimization methods such as PPO and DPO
update the language model toward higher scores (Schulman et al., 2017 Rafailov et al. 2023). This recipe has advanced instruction following, safety, and summarization; a 1.3B InstructGPT model aligned in this way outperformed 175B GPT 3 on adherence and toxicity (Ouyang et al., 2022 Bai et al. 2022; (Stiennon et al., 2020). Subsequent work studies reward hacking and data noise (Wang
et al.
2024
Lambert 2025: Sun al. 2023). Despite these successes, scalar rewards collapse
2

---

Published as a conference paper at ICLR 2026
multidimensional critiques into single number, obscure the location of an error, and necessitate careful regularization, such as Kullback-Leibler penalties, to remain stable (Raschka) 2024: Wu et al. 2023). Even Process Reward Models (PRMs) (Lightman et al., 2023), which offer finer credit assignment, still rely on scalar signals and lack the explanatory power of natural language feedback. Recent work addresses credit assignment through span-level optimization: MA-RLHF improves efficiency via macro-action abstraction (Chai et al.) 2024), SCAR decomposes scalar rewards using Shapley-value allocation (Cao al. 2025), and Beyond Sparse Rewards generates intermediate numeric rewards via auxiliary language models (Cao et al. 2024).
Natural language feedback inference time complementary line of research keeps feedback in natural language but applies it only while the model is running. ReAct interleaves the chain of thought reasoning with tool use to refine answers in question answering and text games (Yao et al., 2023). Reflexion stores self-generated critiques between attempts and improves coding and decision tasks (Shinn et al. 2023). Language Feedback Training incorporates human-written refinements during supervised fine-tuning (Ouyang al. (002)). Surveys categorize the many emerging feedback formats (Fernandes et al. | Liang et al. 2024). These methods lift interpretability and sometimes quality, yet the model weights stay frozen, lessons
retained and error corrections must be rediscovered each time (Pan et al.] (024) Shaarma et al.] 2024). Learning from Natural Language Feedback (Chen al. (024bb uses free-form feedback produce refined sequences and then imitation learns on those refinements under a supervised/KL view; the learning signal remains sequence-level targets derived from edited outputs.
TEXT2GRAD draws inspiration from both threads, yet differs crucially, by training a reward model that generates interpretable textual critiques. uniquely leveraging natural language gradients in token-level PPO to drive fast, interpretable policy improvements.
3 METHOD
This section details TEXT2GRAD, novel framework for Reinforcement Learning from Natural Language Feedback. We define the Natural Language Gradient (NL-Gradient) and then describe our three-stage pipeline: (1) dual-feedback annotation, (2) generative reward modeling, and (3) NL-Gradient policy optimization that enables fine- tme-grained learning from textual critiques.
3.1 NATURAL LANGUAGE GRADIENT: DEFINITION AND MOTIVATION
Traditional policy gradient methods optimize an expected scalar return J(Ø) = E (- [R(v)] where R(y) is a sequence-level reward, which masks token-level contributions and hinders inter- pretability. To address this, we introduce the NL-Gradient, which transforms textual critiques into token-level gradient signals.
Definition 1 (Natural
Language Gradient) Given a generated sequence = (yI...., yr) and its textual critique c, let {ōe)t1 be token-level pseudo-rewards derived by aligning e to y. The NL-Gradient is defined as:
V (C Σd de V log (o v.y)))
Note: "NL--Gradeet" refers to converting language feedback into gradient-based supervision, not literal differentiation through text. We align critiques to spans, map spans to discrete token-level pseudo-rewards, and use these to weight the standard policy gradient. Natural language conditions what gets updated and where. Here, 6, encodes the critique's 's local intensity token Ve, enabling: (1) Fine-Grained Guidance: Pseudo-rewards & highlight specific tokens needing improvement. (2) Interpretability: Each update step is grounded in human-readable feedback. (3) Transferability: The model learns a mapping from text to gradient signals, facilitating generalization between tasks. Our approach is compatible with both RLAIF and RLHF paradigms; human feedback experiments (Appendix L.] demonstrate direct applicability to real human critiques.
3.2 OVERVIEW OF TEXT2GRAD
The core objective of TEXT2GRAD is to construct an NL-Gradient that directly drives policy updates. This requires solving two key challenges: (1) translating free-form textual critiques into structured
3

---

Published as a conference paper at ICLR 2026
Query:
Please Summarize the following post: just sent the completed, unedited novel
LLM Generate
Response:
I'm time author with a 200 page unpublished novel. need help finding a good editor.
Textual Feedback
The summary la to... The summmar, howeverr, insocurately sortmay the author as Mirct author and simpunis their need to just tinding looo editon
Step
Step 2: Reward Model Training
Feedbbckkk Labeling
without mentioning The status of note ia stated correctly.
Span-level Reward
GPT-40
Reward Model
time 300 page unpubllshed need help finding a
good editot.
Critcquea
Affirmtionn
Poor Spars
Good Spars
Query
Author seeks reputable editors with novel from theft.
Textual Feedback: The summaay
...
Token-Level Reward Signals:
Palicy LM
Sampled Trajectory
Reward Model
seeks reputable editors
Step 3:
UPDATE
novel from theft
Policy Optimization
NL-Gradient
Figure 2: An overview of TEXT2GRAD. Yellow highlights critique phrases pointing out errors; Blue highlights affirming phrases identifying correct aspects.
token-level numerical feedback, and (2) leveraging these numerical signals to compute token-level advantages and update the policy. This establishes principled bridge from linguistic reasoning to differentiable credit assignment, operating at a fundamentally different granularity than scalar RLHF methods. The framework generalizes across tasks without modification, requiring only token- weighting wrapper on top of PPO. To address these challenges, shown in Figure [ TEXT2GRAP comprises three steps: Dual-Feedbacc Reward Annotation, which uses GPT-44 to produce high- quality paired critiques and scores; Reward Model Training, which trains unified model to jointly produce explanatory critiques and structured span-level reward maps; and NL-Gradient Policy Optimizaaion, which leverages per-token advantages and applies NL-Gradien PPO updates. Together, these phases realize andto-end NL-Gradient descent for LLMs.
3.3 REWARD LABELING
Effective NL-Gradient optimization requires dense, interpretable feedback that can be precisely mapped to token-level learning signals. We introduce dual-feedback annotation framework that jointly generates (1) free-form natural language critiques and (2) structured span-level reward labels. This design enables task-agnostic supervision while directly supporting the construction of token-level pseudo-rewards for fine-grained policy updates.
Bual-Feedback Annotation Given prompt and generated response y (y11.... yr). we aim to annotate each sample with a natural language critique c, describing strengths or weaknesses of the response in free text, and structured span-level reward map A(y), where each span is assigned neut regative].
label from
In practice, we prompt strong LLM (e.g., GPT-40) to output both feedback modalities. For example, in a summarization task, the model may generate a textual critique such as: "The summary omits key information about the character's 's concern that the manuscript may be rejected." followed by structured ISON object assigning sentiment values to spans in the summary:
"Good spans": ["200 page ur.ublished novel"],
"Poor spans": ["first time author", "finding a good editor")
Critically, our annotation prompt explicitly requires spans to be grounded in and directly supported by the critique, ensuring semantic alignment. We annotate only the most informative signals leaving neutral ral implicit, reducing overhead without loss of utility,
spans
Seesoning-Augmentte Annotation In the absence of human feedback, we employ Chain-of- Thought (CoT) prompting to elicit high-fidelity, self-justified annotations from GPT-4o. Given
4

---

Published as a conference paper at ICLR 2026
response y, the model: (1) Performs step-by-step quality reasoning; (2) Produces a critique grounded in that reasoning; (3) Derives a span-level reward map A(y) Sk + lg. where each labeled span Sq with label f ipositive,negataa
must be explicitly anchored to evidence in c.
Formally, the reward labeler outputs: RLIM (z, y) = Acy)) where A(y) : 8% + la maps each span Sy to label & {posits irosit. ve, negat ineeatiive..... explicitly justified by the critique This protocol enforces strict alignment between critique and annotation. Spans are labeled only where supported by prior reasoning, yielding semantically grounded, interpretable supervision without human references. Full prompts are provided in Appendix B
Token-Level Reward Mapping Although feedback is annotated at the span level. policy optimiza- tion requires token-level rewards. We convert each labeled span into token-aligned supervision by assigning a uniform pseudo-reward 6, E {-1,0, +1} to each token:
+1, if sk and A(y) [sx] positive,
6. =
-1, if E sy and A(y) = negative,
0, otherwise.
To reduce labeling cost while retaining informativeness, we adopt a class- ceaas-ppioiitieed strategy: only positive and neqative spans are explicitly labeled, while neutral spans are left unannotated and default to 6, = O. This yields a token-level reward vector = (ō1...., .5....,))..... which supports token-wise advantage estimation and construction of the NL-Gradieen (see Section 3.5). Our method does not impose fixed span lengths; spans are generated dynamically based on response content. Analysis on SLFSK (Tables 18 and 19) shows that performance depends on span selection quality rather than coverage: CoT-guided annotation labels 30% of tokens with precise signals while maintaining 93-96% accuracy across all span lengths, whereas dense per-token labeling at 70% introduces noise from stylistic or irrelevant tokens. This component enables scalable, interpretable, and task- geskagenrral supervision from natural language feedback.
3.4 REWARD MODEL LEARNING
To enable NL-GGrddien optimization, we train a reward model R. that jointly generates natural language critiques and structured span-level feedback in a single autoregressive pass. Instead of predicting scalar scores, we frame reward modeling language evaluations and span-leve labels as output sequences.
text generation task-producing both natural
Model Objective. Given a prompt x and model response y/ = (y1..... yr), the reward model outputs sequence
[c; A(y),, where e is a critique and A(y) is a map labeling spans in as pos t ivve or negat iitaiiie. ve. We model this as conditional language generation: (z(2 xr v) [| (2x Zec, x. v), and optimize via maximum likelihood with cross-entropy loss: CR(Ø)= - E. (2...)))D llogP((= x,
This formulation provides three advantages: (1) flexibility across tasks via textual supervision; (2) fine-grained gradient flow through tokenized outputs; and (3) interpretable feedback combining explanation and token-level reward in one model. Each training instance is serialized as [x; y; a]. and the model is fine-tume tnnettuned using teacher forcing under a standard causal LM objective. This unified, text-based approach simplifies the pipeline while enabling both structured and natural language feedback to drive token-level learning in TEXT2GRAD.
3.5 NL-GRADIENT POLICY OPTIMIZATION
Traditional RL methods rely on sequence-level scalar rewards, which obscure token-level credit assignment and limit precision. This is especially problematic in tasks like summarization and code generation, where only specific parts of the output may be incorrect. To address this, TEXT2GRAM AD uses dense token-level pseudo-rewards {8,) derived from structured textual feedback to enable fine- grained advantage estimation: A, = 2i (
5TD where 8.TH rí total.4 +VVi (z, Mc(+1) (2, Ver), where y is the discount factor, À is the GAE parameter, Vip is the value function, and ri
A 8, + r combines the token-level pseudo-reward ó, with the KL penalty term rKL
Given response y, we query the trained reward model R& generate natural language critique and span-level reward map, which is parsed into token-wise rewards {8, )T These used to construct
5

---

Published as a conference paper at ICLR 2026
the NL-Gradient: 9NL
Σi= ot Ve log (v 2, V<r), where To is the policy parameterized by 8, providing localized learning signals aligned with feedback.
We then compute token-level advantages using GAE and integrate them into the PPO objective:
LPPO (0) E, [min (peAs; clip(ee; e, e)A.)] 3 (Te((- 2,, ve,))))
where Pr (v I. y.ttt)/ (p<ct)/aLL Toa (ye x. B-()) is the importance ratio, H is the entropy bonus, is the entropy coefficient, and is the clipping threshold that stabilizes updates by constraining large policy shifts. By transforming natural language feedback into token-level gradients, TEXT2GRAD enables interpretable, precise, and efficient policy optimization.
3.6 THEORETICAL ANALYSIS: DISCRIMINATIVE POWER OF TOKEN- REWARDS
Our analysis shows that token-level rewards derived from textual feedback lead to sharper and more discriminative advantage estimates than sedd-f--eeeence rewards. Under our formulation, the advantage timestep is computed as A/ -1)}+))) ôд, where óş are pseudo-rewards aligned to tokens via natural language critiques. In contrast, end-of-sequence rewards yield A (rA)T=& Σk= dg. discounting all feedback uniformly. The difference in temporal credit assignment is given by AA AAA E'= (TA))=A)))A))))) typical settings where ? ss 0.95, a token-level reward at step = T 20 is weighted nearly 0.95 20 = 2.8 times more than it would be under end-of- -equuencceeceeeeeeeeeeeeeeeeeeee supervision shoowiision-soowingg that natural language-guided token-level feedback is nearly 3x more effective for early credit assignment. This yields more informative gradients and improves the policy' ability localize and correct errors in long-form outputs. The full derivation and comparison are provided in Appendix A
which amplifies early feedback differences. For
4 EXPERIMENTS
We evaluate TEXT2GRAD on summarization, code generation, and question answering test its ability to transform natural language feedback into fine-grained policy updates. Our experiments demonstrate that TEXT2GRAD outperforms scalar-reward baselines such as PPO, with improved sample efficiency, faster convergence, and better accuracy.
4.1 DATASETS OVERVIEW
SLFSK (Scheurer et al. summaries, and feedback. We use all 5,000 samples for SFT, reward modeling, and policy training, with 500 for evaluation. KodCode (Xu et al. 2025): A code generation benchmark with 447K question-sollution-ts. triplets across 12 domains. We sample 9K GPT-4o completions train the reward model, and use BodCode-Light-Rl
2023): : A summarization dataset with 5,000 Reddit posts, human-written
Ok for policy optimization. UltraFeedback (Cui et al. 2023): A QA dataset with 64K prompts and 256K completions from 17 models. Following Huang et al. 2024), we split the data into 30% SFT, 50% reward modeling, and 20% RL.
4.2
REWARD MODEL EVALUATION
core component of TEXT2GRAD is the unified reward model, trained to emulate the evaluative reasoning of advanced LLMs (i.e., GPT-4o) by producing structured, token-level feedback.
Experimental Setup We fine-tune L. 3. 1-11--88-sss -n-BB-Innt.uuct (Grattafiori al., 2024) serve as the reward model across all tasks. It is trained to output both a natural language critique and span-level reward signal, using supervision generated by GPT-4o. To ensure high-quality labels, we use CoT prompting strategy (Wei et al. 2022 Ding et al., 2024) in which GPT-4o first reasons through the correctness of a model response, then articulates strengths and weaknesses, and finally highlights token spans as pos i i or negati i ve. This structured annotation improves feedback precision and interpretabilityy enabling richer training signals than scalar-only supervision.
Main Results Table I presents token-level precision and recall for feedback identification, along with levnl win rates in pairwise comparisons (with without CoT reasoning) and human- alignment accuracy. To compute token-level metrics, we map each annotated span to its constituent tokens: tokens within pos i ve spans are labeled +1, negat i ve spans 1,, and all others 0

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative evaluation of reward models with and without CoT prompting, measured by
span-level precision/recall, preference win-rate (W:T.L), and human annotation accuracy.
Positive
Token
Negative Token
Dataset
Win-Rate (W:T:L)
Human-anno Acc.
Prec.
Rec.
Prec.
Rec.
SLFSK
0.58
0.63
0.58
0.43
62:9:29
86%
No CoT
0.63
0.46
0,53
0.40
UlrrFoedbackk
0.66
0.43
0,46
0.22
53:9:38
82%
No CoT
0.61
0.59
0,40
0.35
KodCode
0.64
0.68
0.84
0.71
72:7:21
94%
No CoT
0.62
0.61
0.75
0.78
(neutral); model predictions are evaluated against this derived ground truth. Span-level recall is measured through Exact/Partial Match metries (Appendix [ Our annotation pipelinee ensures high fidelity with unmatched-span rates below 2.5% across all datasets (Table 15 confirming that reward signals are grounded in actual model outputs.
Across all datasets, the CoT-based reward model consistently outperforms the ablated variant, achiev- ing a 62% win rate SLF5K and 86 % alignment with human annotations. Although the precision for positive spans slightly decreases (58% vs. 63%), the recall improves significantly (63% vs. 46%), indicating better coverage and reduced overfitting surface-level cues. The moderate negative-token 43% on SLF5K) reflects label imbalance: approximately 63-70% of
recall (22% on UltraFeeddbar tokens are neutral, so only a minority receive non-zero rewards. From policy-learning perspective. precision is more critical than recall, as false-signed rewards directly corrupt gradients, while missing correct tokens merely reduces update density. Our precision-firs design, combined with high human alignment (>82%), produces stable advantages and substantial policy improvements despite moderate recall. Similar trends hold on UltraFeedback and KodCode, with robust performance in the code domain (KodCode win rate: 72%). Critically, our pipeline enforces strict critique-span alignment: every labeled span must be justified by prior CoT reasoning (Appendix [B., and post-processing ensures spans exact response quotes (unmatched rate <2.5%, Table 15). This produces high consistency
human alignment), enabling scalable and high-fidelity feedback without signal loss. Appendix K further shows that the training time overhead modest compared PPO, primarily due to single reward model forward pass per trajectory. Detailed human evaluation results are provided in Appendix M
Collectively, these results demonstrate that structured natural language reasoning, coupled with precise span-level grounding, enables accurate, discriminative, and data-efficient reward mod- cling forming a robust foundation for token-level policy learning in TEXT2GRAD. The pairwise- comparison prompt provided in Appendix E Additional metries are reported in Appendix C
4.3 SLF5K (SCHEURER ET AL.. 2023): SUMMARIZATION
We evaluate TEXT2GRAD on the SI.F5K dataset (Scheurer et al. 2023), which involves generating summaries of Reddit posts that closely align with human- written references. This task provides natural language feedback and span-level annotations, making it well-suited for evaluating the effectivenesss of token-level reward modeling. Additional hyperparameters are provided in Appendix D
Experimental Setup We use Llama3. . 1 8BB IBB-nsttutt (Grattafiori et al. 2024) as the base policy model. It is first fine-tuned using supervised learning on SLF5K to control output length and content coverage, and subsequently optimized using our NL-Gradient method. We compare TEXT2GRAD against several baselines: (1) PPO (Schulman et al. (017) trained with scalar rewards.
(2) DPO (Rafailov al.) 2023) for preference optimization, (3) PRM-PPO (Lightman et al. 2023) combining preference modeling with PPO, (4) supervised fine-tuning (SFT), and (5) SFT enhanced with reward-guided reflection (Shinn al.) [023 Madaan et al. (023). Appendix (oaiixxNNeette details how PRM spans were defined for each domain, and includes the GPT-3.5 and GPT-4o outputs as reference points.
Evaluation metrics include ROUGE (Lin 2004), BLEU (Papineni et al. 2002), BERTScore (Zhang
et al. 2019). and L.L-Mas-aaJudee Judge (Gu et al. 2024).
Main Results As shown in Table TEXT2GRAN achieves SOTA performance on all metrics, outperforming scalar-reward and reflection-basel baselines. It surpasses PPO by +25.3% BLEU
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on the SLFSK summarization dataset. The policy model is 10mma-3.1-8B--.. Insttruutt. Bold indicates best results; underline indicates second best.
Method
R-1
R-2
R-L
BLEU
BERTScore
Proprietary Models
ChatGPT-3.5
0.155
0.059
0.108
0.020
0.844
GPT-4o
0.296
0.066
0.203
0.030
0.886
Open- Baselines (8B)
SFT
0.285
0.078
0.195
0.032
0.875
SFT Reflection
0.329
0.087
0.225
0.041
0.888
DPO
0.327
0,101
0.224
0.039
0.885
PPO
0.365
0.132
0.262
0.075
0.893
PRM-PPO
0.341
0.130
0.254
0.069
0,889
ILF (Scheurer
2023
0.349
0.134
0.259
0.073
0.892
Ours
TEXT2GRAD (w/o CоT)
0.380
0.140
0.275
0,085
0.898
TEXT2GRAD
0.400
0.155
0.291
0.094
0.902
Fitdl Medel Win
Thew
Second Model 1ite
SIFSK
3:70
0.13
PNO
39.8%
12.8%
37.8%
Test2Gral EDRK
0.12
5.00
SUT
39.2%
8.2%
32.4%
cce-
RPO
20
40
80
100
Training Samples (Thomsands)
Percentage (%)
(a) Reward curve for SLFSK dataset.
(b) GPT4 Judge Comparison across Methods.
Figure 3: Combined figure for SLF5K dataset analysis.
and +6.7 ROUGE-L, exceeds DPO and PRM--PPO by +6.7 and +3.7 ROUGE-L respectively, and improves over SFT+Reflection by +3.3 ROUGE-L, confirming that gradient- -haseedt-baaddddeeee internalization of feedback yields stronger gains than inference-time correction. To validate our span-based design, compare against dense token-level labeling. Despite maximal supervision, dense labeling performs substantially worse (ROUGE-L: 0.196 vs. 0.291), as it labels ~0% of tokens predominantly function words rather than semantic spans, introducing noise into advantage estimates. Our span- based approach achieves superior performance while maintaining high grounding fidelity (unmatched rate <2.5%) and reducing annotation costs by 85-90% (Appendix I. Qualitatively, GPT-4-as- a-Judge preferences (Figure 3b) show 12% win-rate gain over PPO, indicating more coherent and informative outputs. Quantitatively, Figure ]a reveals TEXT2GRAD converges 22 % faster, demonstrating that token-level gradients accelerate learning while allowing interpretable updates. The table also shows that removing reasoning degrades performance.
4.4 KODCODE XU ET AL. 2025): CODE GENERATION
We evaluate TEXT2GRAD on the KodCode dataset (Xu et al. 2025). ) which focuses on generating correct Python solutions across 12 diverse problem domains. This task highlights the importance of span-level feedback in structured text generation, where subtle errors can invalidate the entire output.
Experimental Setup We adopt Llama 1 1-88------- Instruct (Grattafiori et al. 2024) as the policy model. For reward model training. we sample 10,000 dataset, using GPT-40 outputs as high-quality references and GPT-3.5 completions challenging negatives to form pairwise comparisons. Annotations include textual critiques and span-level labels. We benchmark TEXT2GRAD against PPO (Schulman et al. 2017) and strong baselines, evaluating
pairs from the supervised
8

---

Table B shows TEXT2GRAD outperforms all baselines-bor pre-trained and aee-tuned-across all benchmarks. Against PPO, it gains +5.8 on MBPP+ and +3.6 on HumanEval+, demonstrating superior robustness to adversarial test cases. also surpasses DPO and PRM-PPO by 5.1 and 5.8 average points, respectively. Critically, the ablated variant (TEXT2GRAD w/o CoT) underperforms by 6.9 points on average, confirming that structured natural language feedback- not just span labels-is essential for effective token-level credit assignment. These results validate that TEXT2GRAD precisely localizes and corrects subtle coding enrors, yielding programs that Published as a conference paper at ICLR 2026
via pass@1 1 aceuracy HumanEval (Chen et al. 2021 MBPP AAsttin al. 2021). and their
robustness-enhanced variants, HumanEval+ and MBPP+ Yu al. 2024).
Table 3: Code generation benchmarks (pass @ %).. Bold:
Main Results
best; underline: second best.
HumanEval
MBPP
Method
Size
Avg.
Base
Plus
Base
Plus
Proprietary & Pre-trained Models
-lama-3.2-Instree
3B
48.2
43.9
61.9
51.3
51.3
Llama-3. I-Instruct
8B
64.0
58.5
66.7
55.0
61.1
CodeLlama
51.8
43.9
56.3
55.3
Gemini Pro
63.4
55.5
72.9
57.9
62.4
Fine-tumed (Ilama-33 1--in-33.-8B--mmee 1-8--uutttttt)
DPO
8B
65.2
56.7
66.1
56.1
61.0
PPO
64.6
61.0
68.5
55.8
62.5
PRM-PPO
8B
61.5
59.8
65.1
54.9
60.3
ILF
63.4
60.4
68.5
57.1
62.3
Ours
TEXT2GRAD (w/o CoT)
8B
63.8
57.3
62.2
53.4
59.2
TEXT2GRAD
8B
67.7
61.6
73.3
61.6
66.1
generalize reliably under stress.
4.5
ULTRAFEEDBACK (CUI ET AL.
2023): OPEN-DOMAIN QUESTION ANSWERING
To evaluate TEXT2GRAD on general- purpose alignment, we test it on Ultraeeddbak (Cui al. 2023), diverse QA benchmark spanning multiple domains and difficulty levels. This task assesses generalization to open-ended prompts, factual accuracy, and conversational coherence.
Experimental Setup We use LLlam3-8B-IIn' ruct as the policy backbone. Evaluation metrics include: (1) AlpacaEval 2.0 (Dubois et al.] 2024) for instruction alignment, (2) ARC- Challenge (Clark et al., 2018) for reasoning, and (3) MT-Bench (Zheng et al. (023) for multi-turn dialogue quality. UlrraFeedback responses long and do not contain explicit intermediate reasoning steps, making PRM-style step annotation ill-defined and costly (6-8x higher token budget). We therefore omit PRM-PPO; see Appendix N details.
Table 4: UltraFeedback QA benchmarks. Bold: best; underline: second best.
Main Results As shown in Table 4 TEXT2GRAD consistently improves over both the base model and PPO across all metrics. On AlpacaEvval 2.0, TEXT2GRAD achieves a 12.1-point gain over the base model and 2.3-point improvement over PPO, indicating stronger instruction align- ment and preference satisfaction. On ARC- Challenge, TEXT2GRAD shows improved reasoning over base and PPO, while MT- Bench results show better multi-turn dia- logue performance.
Method
AlpacaEval ARC-C MT-Bench
Proprietary Models
GPT4
30.2
96.4
7.93
GPT-3.5
22.7
85.2
6.91
Open-Source (Llamm3-8B-Instraan
Base
22.6
80.5
6.87
DPO
32.6
81.0
7.01
PPO
32.4
82.7
7.43
ILF
30.1
80.9
7.08
study isolates the effect of CoT reasoning the annotation pipeline. Specifically, we remove CoT and provide feedback only as span-level scores without natural-langun language explanations. Training without CoT yields consistent drops across all metrics, with the largest decrease on AlpacaEval. This suggests that natural- language explanations produce more actionable token-level supervision for NI-GGradeen optimization.
Our ablation
Ours
TEXT2GRAD (w/o CoT)
28.6
83.1
7.49
TEXT2GRAD
34.7
84.4
7.58

---

Published as a conference paper at ICLR 2026
4.6
CASE STUDY
Policy
Scalor Reward Model
PPO
Sampling
are arenn
match pwwerad 1o:
1231
81
Inllcy
0000 count - Cloc COUIE
BILL wwwnssponnoooo soMm ci00s COMBE
stack MA
manch paroeafaatie Saa buildl
HLGmndentPPD
Reward/Mddal
dcl cor, Io ln
for cher
Belencs 23
reture belonce
peturrs clast
Wi lscell
code
only
pault SESt belancata. A. 123, hww.annnneea 2 sir
Figure 4: A case study from the code generation scenario comparing PPO vs. TEXT2GRAD.
Figure 4 shows how TEXT2GRAD corrects a faulty implementation of match parcchppreeeseeee while standard PPO fails. The policy LM first produces buggy patch. A scalar reward model gives PPO a single negative score (-2), leaving the optimizer without guidance on where the error is located. After several updates, it still ignores the two concatconcttnnnioonn checks required by hidden tests.
TEXT2GRAD proceeds differently. The natural language reward model highlights the exact faulty span for char in 1st [0]
and explains that the code "fails to check 1st [0] + 1st [1] and lissst [1] + 1st [0]." This critique is aligned with the offending tokens and converted into negative rewards for that span and positive rewards for the rest. single NL-Gradient update rewrites
only the highlighted lines. The resulting function passes all unit tests. This example underscores the advantages of TEXT2GRAD. Additional qualitative results appear in Appendix
4.7 CROSS-MODEL GENERALIZATION
To validate that TEXT2GRAD'S 's gains transfer across model families, we evaluated on Mistral-7B: Instruct-v0.2 across code generation, open-domain QA, and summarization. As shown in Table 5 TEXT2GRAV AD consistently outperforms baselines across all tasks: on code generation, average pass@1 I improves from 42.9 (DPO) to 45.3; on QA, AlpacaEval increases from 26.17 to 29.40; on summarization, ROUGE-L reaches 0.24. These results confirm the method's generality.
Table 5: Cross-model model evaluation Mistral- 7B-II--rrtt-ttOO.1 Bold: best per metric.
Code (pass@ %)
QA
Summarization
Method
HE
HE+
MBPP
MBPP
AlpacaFval
MT-Bench
R-I.
BLEU
BERT
Base
42.1
36.0
44.7
37.0
17.11
6.60
0.200
0.024
0.66
PPO
45.7
38.4
47.1
38.3
19.62
6.55
0.230
0.035
0.70
DPO
47.6
39.6
46.7
37.6
26.17
6.30
0.210
0.030
0.69
TEXT2GRAD
50.0
40.9
49.6
40.6
29.40
6.78
0.240
0.041
0.72
5 CONCLUSION
We presented TEXT2GRAD, a new framework for learning from natural language feedback by con- verting free-form textual critiques into span-level reward signals and actionable gradients. Unlike traditional RLHF approaches that rely on scalar rewards or inference-time prompting strategies, TEXT2GRAD directly incorporates feedback into the training process through token-aware policy updates. This enables precise credit assignment and more interpretable learning dynamics. Ex- perimental results across summarization, code generation, and question answering demonstrate that TEXT2GRAD consistently outperforms scalar-reward PPO and prompt-based baselines in both alignment quality and sample efficiency. Cross-model evaluation on Mistral- 77-Innsrrcct-O.. (Sec- tion 4.7p further validates that these gains transfer across model families and architectures. Overall, TEXT2GRAD opens a new direction for fine- araegrr feedback driven optimization of LLMs, moving beyond scalar supervision toward more human-like, interpretable, and effective learning.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Chaoyun Zhang, Zicheng Ma, Yuhao Wu, Shilin He, Si Qin, Minghua Ma, Xiaoting Qin, Yu Kang. Yuyi Liang, Xiaoyu Gou, et al. Allhands: Ask me anything large-scale verbatim feedback via large language models. arXiv preprint arXiv22403.15555 2024a.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0568 2022.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, (153728-53741) 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1iXXiv:707.000 1707.063477 2017.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 152 2023.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008-59033, 2023.
Sebastian Raschka. Build a Large Language Model (From Scratch). Simon and Schuster, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-86533 8652, 2023.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, al. Ufo: ui-focused agent for windows interaction. arXiv preprint arXiv:2402.07933, 2024b.
Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos. arXiv preprint arXiv:2504. 14603, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in 2022.
neural information processing systems,
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint 12.02221 2021.
11

---

Published as a conference paper at ICLR 2026
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu. Qingwei Lin, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.1827, 2024c.
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José GC de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, et al. Bridging the gap: A survey on integrating (human) feedback for natural language generation. Transactions of the Association for Computational Lingwistics, 11:1643-1665 1668, 2023.
Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt, Feedback loops with language models drive in-context reward hacking. arXiv preprint arXiv:2402.06627, 2024.
Archit Sharma, Sedrick Scott Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. A critical evaluation of ai feedback for aligning large language models. Advances in Neural Information Processing Systems, 37:29166 090916666990000 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 188):16o2400116 2024.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprini arXiv:2410. /2832, 2024.
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. arXiv preprint oxXiv22401.06088 2024a.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496
2024.
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint 1.Xi::2303.16755.. 2023.
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arXiv:2503.02955 2025.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeebbcc:: Boosting language models with high-quality feedback. 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06088 2024.
Nathan Lambert. Reinforcement learning from human feedback. arXiv preprint arXiv:2504.1 12501, 2025.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 2rXivv2309.144 14525, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, nya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, and Hua Wu. Ma-rlhf: Rein- forcement learning from human feedback with macro actions. arXiv preprint arXo::2410.02743, 2024.
Meng Cao, Shuyuan Zhang, Xiao-Wo Wen Chang, and Doina Precup. Scar: Shapley credit assignment for more efficient rlhf. arXiv preprint vrXiv:2505.20417 2025.
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Beyond sparse rewards: Enhancing reinforcement learning with language model critique in text generation. arXiv preprint arXiv:2401.07388) 2024.
Angelica Chen, Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. Learning from natural language feedback. Transac- fions on machine learning research, 2024b.
Chenghua Huang, Zhizhen Fan, Lu Wang, Fangkai Yang, Pu Zhao, Zeqi Lin, Qingwei Lin, Dongmei Zhang. Saravan Rajmohan, and Qi Zhang. Self-evolved reward learning for llms. arXiv preprint arXiv:2411.00418, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv e-prints, pages arXiv-2407, 2024.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. In Findings of the Association for Computational Linguistics ACL 2024, pages 1638-1662, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- self-feed Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 81, 2004.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 3-3111 2002.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating 2019.
text generation with bert. arXiv preprint
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint 1.Xiv2241111554 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint 1XXx:22008.0732.. 2021.
Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Humaneval pro and mbpp pro: Eval- uating large language models self-invoking code generation. arXiv preprint arXiv:2412.21199 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controoled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.00 0444504 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803 05457, 2018.
13

---

Published as a conference paper at ICLR 2026
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-44622) 2023.
A DISCRIMINATIVE POWER OF TOKEN-LEVEL REWARDS
key design choice in our method is to provide dense, token-level feedback rather than sparse, end-of-sequence rewards. Intuitively, localized reward signals allow the policy to attribute credit blame more precisely to specific parts of the output. In this section, we formalize this intuition and show how token-level rewards lead sharper and more discriminative advantage estimates, thereby improving policy learning.
Background. In reinforcement learning, policy updates are guided by the advantage function, which measures how much better (or worse) an action is compared the policy's 's expected value. Using Generalized Advantage Estimation (GAE), the advantage at timestep is computed from the -efforal- (TD) emros::
A, = (TA) 8e42 where = +V(+4)(())))) - V(se);
and V is the value function, y is the discount factor, and is the GAE parameter.
Comparing Token-Level vs. End-of-Sequence Reward Settings. We define two settings for reward assignment:
Setting A: Loken-Level Rewards. Each token may receive its own feedback:
token.A
A 0 for many & E [1,
total,A Total reward: ri re
Aoken.A
KT. + ri
Setting B: End-of-SSquence Reward. Only the final token is rewarded:
Boken.B 0 for all <
B f 0; total reward: rt total.B tokennn + r
Let and be two trajectories, where is qualitatively better than T2. Define Ar. r ( rt Loken,A (72), and assume all KL terms and value functions are held constant for simplicity (the general case follows similarly).
Advantage Difference Across Trajectories.
The advantage difference under each setting is:
AAA (na))- Ark.
AA
Ark-
Even if ET=, Ark is the same in both cases (i.e., the same total reward difference), AAA AAP whenever any Ark > 0 for l < T, because:
(~)))-1>> (TA)T-1, for all & < T.
This means the earlier reward difference occurs in the sequence, the more strongly it is emphasized in Setting A relative to Setting B.
Amplification of Early Signal. To quantify this difference, define the amplification factor:
k= alk, T) = (xA)- = (TA)-(0-AR) (T-A)
14

---

Published as a conference paper at ICLR 2026
For typical value of yy 0.95 and gap of T k 20 steps (i.e., the difference occurs 20 tokens before the final token), we have:
a T) 0.95 20 2.8,
meaning that in Setting A, the advantage function weights early reward differences nearly 3x more than in Setting B.
This analysis confirms that token-level feedback improves the discriminative power of the advantage signal: even if the total reward difference is the same, Setting A assigns more importance to earlier deviations in quality. This sharper signal allows the policy to learn localized corrections-e.2 improving grammar or factual consistency in specific parts of summary- rather than attributing success or failure to the entire sequence. As result, our method enables faster convergence and better fine-tuning especially open-ended tasks where quality varies across tokens.
B GPT-40 CHAIN-OF-THOUGHT ANNOTATION PROMPTS
This section presents the detailed prompt templates used for generating dual-fcedback annotations across three experimental datasets. Each prompt is designed to elicit both natural language critiques and structured span-level feedback through CoT reasoning.
B.1 SLF5K DATASET
The following prompt template is used for generating annotations the SLF5K summarization dataset:
Listing 1: SLE5K GPT-4o Annotation Prompt
Please critique the following summary of post and provide feedback in the specified JSCN format:
*riginna Post:*> (post)
Generated Summary:**
ggenerated_sunmaryy
10
11
12
+*Definitions:+<
14
**good_spana spans**: 0-2 phrases from the summary that greatly improve its quality by accurately and concisely capturing the original post's s core meaning or key details, as explained in lexxtual_feebbade none apply.
Empty if
**poor_spans*x: 0-2 phrases from the summary that noticeably harm its quality due to inaccuracy, redundancy, poor wording, or being less important and replaceable with more critical content, as explained in cenetualffeebbckk Empty if none apply.
16
17
18
19
I*Instructions:*
1. Identify the summary's most essential strengths that reflect the original post accurately and its most critical weaknesses that misrepresent or confuse it.
15

---

Published as a conference paper at ICLR 2026
2. Select 0-2 of the most significant phrases for ggood_spans' and 'poor_spans" keeping them conci se and impactful, with brief justifications. Include none if no phrases stand out.
22
3. Ensure 'good_spans' and 'poor_spans' are directly supported by the analysis in ttettual feedbackk
25
25
26
Chaiin of Thought:**
First, carefully analyze both the original post and the generated summary:
1. What are the key points of the original post?
2. Which of these key points are accurately captured the summary?
50
3. What important information is missing from the summary?
4. Are there any inaccuracies or misrepresentations in the summary?
5. which specific fic phrases in the summary represent its strongest elements?
6. which specific phrases in the summary represent its weakest elements?
Based on this analysis, formulate your textual feedback and identify the good and poor spans.
36
37
39
Outpptt Format:**
Provide concise, paragraph critique and the GOOD/POOR spans in this JSON structure: ''json
42
"textuaal_feedackk: "Your critique here summarizing key strengths and weaknesses in one paragraph. ", "good_spans"; ("phrasel", "phrase2"), // 0-2 concise
phrases from the generated summary, tied to eextual_feedback. or [] if none "poor_spans" ("phrasel", "phrase2"] // 0-2 concise phrases from the generated summary, tied to textual_feedback, [] if none
46
as
Focus on precision: include only the most impactful phrases of the generated summary, avoiding excessive or minor
details.
B.2 ULTRAFEEDBACK DATASET
The following prompt template is used for generating annotations on the UltraFeedback question- answering dataset:
Listing 2: UltraFeedback GPT-40 Annotation Prompt
16

---

Published as a conference paper at ICLR 2026
<CritiquePrompt>
<instructionssCCritiqqqe response to user input and provide feedback in JSON f Innaattttins
<bvaluationCriteries
<Criterion name="Accuracc""ooes it correctly address the input?</Criteriin>
<Criterion same stay on topic?</Criterio>>
<Criterion name="Clarrtyy">t it easy to understand22/Crtterrlon.
10
<Criterion mame="Complereeness">eee it cover the
input's core needs?</Criterion>
11
</BvaluationCriteriaa
13
SpanGuidellines> <GoodSpans>
1s
<Description>Phrase from the response that best capture its strengths (e.g., accurate, relevant, clear). Select only the most essential and impact phrases, directly tied ttextual feedbacke </Description>
< /GoodSanss>>>
<
18
<PoorSpans>
DescripttionPhaasss from the response that best highlight its weaknesses (e.g., inaccurate, irrelevant, vague). Select only the most essential and impact phrases, directly tied to 'textual feedback' </Description> < </PoorSpans>
20
<Requi rement>Spans must be exact quotes from the response. /Rquuirement>>> </ </SpanGuidelines:
22
23
24
<efleectioProcess>
06
<Step>First, carefully analyze the user input to understand the core question or request.</Step> <Step>Next, examine the generated response against each evaluation criterion.</Ster <Step>For each criterion, identify specific strengths and weaknesses with supporting evidence from the response.</ttee <Step>Consider how well the resporse addresses the user's s explicit and implicit needs. needs.</Step: </Step> <Step>Finally, synthesize your analysis into coherent critique that highlights the most important points. </Step> </ReflectionProcen
27
28
30
32
M
36
17

---

Published as a conference paper at ICLR 2026
57
( //UserInputs
<Generateddesspose>>.cc (entry
(entry [' response' ]] /
99
4]
< </Separator>
43
<OutpputFormtt>
45
<Description the critique in the following JSON structure: structure: <JSONExample>>
56
"One-paragraph critique
summarizing strengths and weaknesses, tied to spans.",
19
["phrasel", "phrase2", ...,, // Impact phrases from <GeneratedRespons>> or [] if none : ["phrasel", "phrase2", ...]
// Impact phrases from <GeneratedRspoose>> or [] if none
}
52
< </SONExample> </OutputFormat:>
</CritiquePrompt>
B.3 KODCODE DATASET
The following prompt template used for generating annotations on the KodCode code generation dataset:
Listing KodCode GPT-4o Annotation Prompt
Analyze the following code solution for the given problem:
Problem Description:
(problem)
Submitted Code:
(solution]
11
'
12
13
Test Results: Passed: (passed)
15
16
{ if not passed -$)
17
Test Question:
18
(test_question)
20
Error Output: (stdout)
18

---

Published as a conference paper at ICLR 2026
22
{ endif -8}
24
Please analyze the code and identify the following in JSON
format:
25
1. Identify any error-causing code segments directly from the submitted solution.
2. Provide detailed feedback the code's functionalin issuss and improvement suggestions.
2s
First, understand what the code is trying to accomplish Analyze the algorithm and approach used
29
00
Identify any logical errors or inefficiencies
Consider edge cases and potential improvements
52
3. Point out any code segments from the solution that work
but could be improved.
33
Return your analysis in this JSON structure: vijson
as
37
"Code Feedback"; "Provide a detailed explanation of the code's s functionality, any potential issues, and suggestions for improvement. Use markdown formatting for better readability.",
"wrong_coder ["Extract ONLY the problematic code segments FROM THE SUBMITTED SOLUTION that cause failures Must be exact quotes. Leave empty [] if none found. "], iimprovement_coce code": ["Extract ONLY the working but improvable code segments FROM THE SUBMITTED SOLUTION. Must be exact quotes. Leave empty [] if none needed.")
41
Note: For 'wrong_code' and iimprovement_code only include direct quotes from the submitted code above, not suggested fixes. .un
C ADDITIONAL REWARD MODEL PERFORMANCE RESULTS
This section presents supplementary evaluation metrics for the reward model across three datasets, focusing on span prediction quality (UltraFeedback), code suggestion fidelity (KodCode), and language modeling coherence (SLF5K). These metrics quantify whether feedback actually conditions behavior the criticized spans, demonstrating that span-level rewards reliably localize and influence the regions referenced in critiques.
Table 6: Span prediction performance the UltraFeedback dataset. Reported GT / Pred (Exact- Partial), where GT ground truth span count, Pred = predicted span count. Exact and Partial denote exact and partially overlapping matches, respectively. OUI (Overlap Unit Index) quantifies boundary alignment precision.
Span Type
GOOD Spans
POOR Spans
Count (GT/ Pred)
1150
812 528
Match (Exact Partial)
(394 468)
(167 2274
OUI
0.40
0.27
19

---

Published as a conference paper at ICLR 2026
Table Code suggestion quality on the KodCode dataset. Exact Match measures the proportion of generated suggestions that exactly match the reference code. 2 90% Overlap evaluates the proportion of suggestions with least 90% overlap with ground-truth code segments.
Suggestion Type
Exact Match
90% Overlap
Improvement Suggestions Error Correction
0.47
0.67
0.55
0.79
Table Extended language modeling metrics on the SLF5K dataset. Human reference perplexity: 37.375. Bold: best results.
BERTScoref
Method
Perplexity
Precision
Recall
Proprietary Models
ChatGPT-3.5
27.288
0.806 0.879
0.884
GPT-4o
53.242
0.894
Open-Source (Llama-3. 1-8B-Instruct)
Base
19,248
0.848
0.894
SFT
44.103
0.865
0.885
SFT Reflection
34,823
0.880
0.897
PPO
28.472
0,892
0.895
Ours
TEXT2GRAD
25,423
0.903
0.902
D TRAINING HYPERPARAMETERS
This section provides complete hyperparameters for both reward model training and policy optimiza- tion.
D.1 REWARD MODEL TRAINING
We fine-tune Llama-3.1-811 --8t-ruttttt the reward model using LoRA for parameter-efficien adapta- tion.
Table 9: Hyperparameters for reward model training (shared across all datasets).
Parameter
Value
Parameter
Value
Base Model
Llama-3, 1-BB
Learning Rate
1
10
Hardware
8x V100 32G
Training Epochs
2
Parallelism
ZeRO-3
Optimizer
AdamW
Global Batch Size
Precision
FP16
LoRA Rank (r)
LoRA Dropout
0.1
LoRA Alpha (a)
32
Grad, Clip Norm
1.0
LoRA Targets
pproo, v_pred
Weight Decay
3 x 10-7
D.2 NL-GRADIENT PPO OPTIMIZATION
Hyperparameters tailored per dataset to balance training stability and optimization efficiency.
20

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameters for NL-Gradient PPO across all datasets.
Hyperparameter
SLF5K
UltraFeedback
KodCode
Model Configuration
Policy Model
Llama 3.1 8B
Llama 3 8B
Llama 3.1 8B Llama 3.1
Reward Model
Llama
Llama 3.1
Optimization
Learning Rate
x 10 Linear
x 10 Cosine 4
5 x 10- Cosine 4
LR Scheduler
Training Epochs
Batch Settings
Global Batch Size
8 1
Mini-batch Size
PPO Epochs/Batch
4
4
Gradient Accum.
12
8
8
KL Regularization
Initial KL Coef.
0.2
0.05
0.07
Target KL
3.0
1.0
KL Penalty Type
Full
Full
Adaptive KL.
/
Infraxtructure
Hardware
8x V100 80G ZcRO-3
8xA100 S0G
8xA100 80G
Parallelism
ZcRO-3
ZcRO-1
Notes: SLF5K uses linear scheduler; UltraFeedback adopts conservative KL for diverse responses; KodCode uses strictest KL target (1.0) to preserve code semantics.
E GPT-40 JUDGE COT INFLUENCE ANNOTATION PROMPT
E. SLF5K EVALUATION PROMPT
The following prompt template was used to evaluate model responses on the SLF5K dataset. To prevent position bias in the evaluation, the order of model responses (analysis and analysis was randomly shuffled for each comparison:
Listing 4: SLF5K Evaluation Prompt
Compare and evaluate two different summaries of the same query. You must respond in valid JSON format.
Original Query: {query)
[analysis___iabel] (response_1)
(analysis_2_label]] (response_2)
10
12
Evaluation Criteria:
13
1. Accuracy (0-10):
14
Does it capture the main points correctly?
1
Is it faithful to the original content?
16
Are there any factual errors?
21

---

Published as a conference paper at ICLR 2026
18
2. Completeness :0-10)
Are all key points included?
20
Is any important information missing?
Does it cover the core message?
22
3. Conciseness (0-10)
24
Is it clear and to the point?
Does it avoid unnecessary details?
26
Is the language efficient?
28
4. Coherence (0-10):
Is the summary well-organized?
Does it flow logically?
51
Is it easy to understand?
33
Compare both summaries and evaluate them. Respond ONLY with JSON object in this exact format:
34
[[score_key_l) {
%6
"strengths": ("specific strength 1", "specific strength 2", ...l,
37
"weaknesses": ["specific weakness 1", "specific weakness 2", ...]
score": <overall score between 0-10>,
39
accuracy": <score between 0-10>,
completeness": <score between 0-10>,
conciseness": <score between 0-10>,
coherence"; <score between 0-10>,
43
(scorre_ey__))": {
45
"strengths": ("specific strength 1", "specific
strength 2" ...J,
46
"weaknesses" ["specific weakness 1", "specific weakness 2" ...]
47
"score": <overall score between 0-10>,
48
"accuracy;; <score between 0-10>,
19
completeness": <score between 0-10>,
50
conciseness"; <score between 0-10>,
"ccheeece": <score between 0-10>,
E.2 KODCODE EVALUATION PROMPT
The following prompt template was used to evaluate the quality of code span selections for the KodCode dataset, which resulted in the win-rate metrics (72.17 7.01 20.82) comparing CoT feedback quality:
Listing 5: KodCode Evaluation Prompt
Evaluate the precision and speci ficity of code span selections in two different analyses.
Problem: (problem)
22

---

Published as a conference paper at ICLR 2026
Solution Code: (solution)
[analysis_l_laeel)
Selected spans: (spans_ 1]
11
Suggestions: (improve_l}
12
13
@amalysis_2_label]]
14
Selected spans: (spans_2) 2
15
Suggestions: (improve_2)
16
Please evaluate the quality of span selections in JSON format, focusing on precision and minimality:
18
19
"[score_key_1)"; {
"score": (0-10 score for span selection precision), "Reason": "Explain the reason for the score"
21
23
[sscorekky_2]] "score": (0-10 score for span selection precision),
25
"Reason": "Explain the reason for the score"
"comparison": "Explain which analysis has more precise minimal span selections"
29
Guidelines for span evaluation:
31
1. Each span should capture ONLY the specific problematic code, nothing more
s2
2. Ceneral or overly broad selections (like entire functions) are penalized
s
3. Spans should not include irrelevant surrounding code
4. Multiple small precise spans better than large span
5. Spans must directly relate to the identified issue
E.3 ULTRAFEEDBACK EVALUATION PROMPT
The following prompt template used to evaluate the precision and specificity of text span selections for UltraFeedback dataset:
Listing 6: UltraFeedback Evaluation Prompt
"#"Evaluate the precision and specificity of text span selections in two different analyses.
Problem:
(problem)
Reaponse:
(solution]
Sanalysis_1_labet
Problematic spans: (spans_1)
23

---

Published as a conference paper at ICLR 2026
n1
Good spans: (improve_1)
12
13
Siaalysis___lablll
Problematic spans: (spans_2)
15
Good spans: (improve_2)
17
Please evaluate the quality of span selections in JSON format, focusing precision and minimality:
18
19
('score_ke__l__
20
"score": (0-10 score for span selection precision), "Reason": "Explain the reason for the score"
"score": (0-10 score for span selection precision), "Reason": "Explain the reason for the score"
25
27
"comparison": "Explain which analysis has more precise span selections while avoiding unnecessary overly broad spans"
Guidelines for span evaluation:
1. Each problematic span should capture ONLY the specific problematic text, nothing more
2. General or overly broad selections are penalized Spans should not include irrelevant surrounding text
s3
4. Multiple small precise spans are better than one large span
s
5. Spans must directly relate to the identified issue
36
6. Good spans should highlight the best parts of the response
F CASE STUDIES ON HUMANEVAL
We present three case studies from the HumanEval benchmark to demonstrate the effectiveness of our approach.
F.1 SPECIAL FACTORIAL
24

---

Published as a conference paper at ICLR 2026
Problem Statement
def special factorial (n)
"""The Brazilian factorial is defined as: brazilian_factorian (n) n! (n-1)! (n-2) 1!
where n 0
For example:
>>> Opecial_factoria 288
The function will receive an integer input and should return the special
factorial of this integer.
12
nnn
Instruct Model Solution
def special faall_fattorialll (n)
if n < 0:
raise ValueError (" InInp must be non-negative integer")
result = 1
for range n 1): result == math. factorial (i)
return result
PPO Model Solution
def special, sicciallfacttriaal (n) result = 1 for in range (1, n 1): result x= i return result
Our Approach
import math
def if not isinstance (n, int) or n raise ValueBrror ("Input must be positive integer.") ")
(n)
result =
for range n 1): result math. factorial (i)
return result
25

---

Published as a conference paper at ICLR 2026
F.2
FILE NAME VALIDATION
Problem Statement
def file__nam_chheck (file_name):
"""Create function which takes string representing file's s name, and returns YYes' if the the file's s name is valid, and returns 'No' otherwise.
A file's name is considered to be valid if and only if all the following conditions are met:
There should not be more than three digits ('00-199) in the file's s name.
The file's name contains exactly one dot The substring before the dot should not be empty, and it starts with letter from the latin alphabet ('a'-'z' and '2'
10
The substring after the dot should one of these: ['txt', 'exe', 'dll']
11
Examples:
file_name_check ("example txt") # => YYes' file_nname_check ("1example. dll") # 'No' (the name should start with latin alphapet letter)
13
14 nnn
Instruct Model Solution
def file_name_chie
# First, split the file's name by the dot name, extension =
rsplit .'.,, 1)
# Check if the extension is valid valid_extension ['txt', 'exe', 'dll') if extension not in ealid_extenssons: return 'No
10
# Check if the substring before the dot is not empty if not name: return 'No'
11
12
13
$ Check if the substring before the dot starts with letter from the latin alphabet if not name [0] () return 'No'
16
18
# Check if there are more than three digits in the file's s name
19
if sum (c.isdigit () for in file_name) 3: return 'No'
21
# TF all conditions met, return 'Yes'
return YYes'
26

---

Published as a conference paper at ICLR 2026
PPO Model Solution
def @ile_name_check (file_name)
# Split the file name into two parts: the name and the extension
name, extension
calle_name.spitt
P Check the name starts with latin alphapet letter if not name [0]. isalpha() (): return 'No'
# Check if there are more than three digits in the name if sum sc () for in name) 3: return 'No
11
13
# Check if the extension is valid
if extension not in ['txt', 'exe', 'dll]]: return 'No'
15
16
17
# Check if the file name contains exactly one dot if file_name count (' .') != 1: return 'No"
18
20
# If all checks pass, return 'Yes'
22
return "Yes'
Our Approach
def File_name_check (file_name)
Checks if a file name valid according to the given conditions.
A file's
is considered to be valid if and only if all the following conditions are met:
There should not be more than three digits (f0r-19)) in the file's name.
The file's name contains exactly one dot
The substring before the dot should not be empty, and it starts with letter from the latin alphabet ('a"-'z' and 'A'-'2') The substring after the dot should be one of these: ['txt', 'exe', 'dll'].
10
12
13
Args:
file_name (str): The name of the file to check.
15
16
Returns:
str: 'Yes' if the file name is valid, "No" otherwise.
18
20
# Check the file name contains exactly one dot if file_name.com ('.)) != 1: return 'No"
27

---

Published as a conference paper at ICLR 2026
34
# Split the file name into two parts: before and after the dot before_dot, after_dos file_name name file_nnme.splitt"..
25
27
# Check if the substring before the dot is not empty and starts with letter
28
if not before dot or not before_ dor [0] (salpha() return 'No'
o0
Check the substring before the dot contains more than three digits
32
if sumic.isdigit () for in before_dot) 3: return 'No'
54
# Check if the substring after the dot is one of the allowed extensions
if after_de dot not in ['txt', 'exe', ddll]]: return 'No'
37
39
# If all conditions are met, return 'Yes'
return "Yes'
G LIMITATIONS
Despite its effectiveness, TEXT2GRAD has two potential limitations. First, our framework still depends on the quality of the reward model. While this dependence is inherent to RLHF-style methods, our CoT--based annotation pipeline achieves strong human alignment (82-94% accuracy across datasets, Table 16) and maintains high grounding fidelity with unmatched-spaae rates below 2.5% (Table 15). The framework's 's consistent performance gains across all benchmarks demonstrate that the current reward model quality is sufficient for effective policy learning, though further improvements in critique generation could enhance optimization in tasks requiring highly nuanced feedback.
Second, generating and applying token-level rewards introduces additional computational overhead compared scalar reward methods. Our analysis (Appendix K)) shows that this overhead is modest: approximately 9-11 % per training step (Table T3). primarily from a single reward-model forward pass per trajectory. Moreover, span-level annotation reduces token costs by 85-90% compared to dense token-level labeling, with annotation costs on the order of 10 USD per sample (Table 14). The substantial performance improvements help justify this modest cost increase, and the framework remains practical for large-scale deployments.
In future work, we aim further improve reward model precision and efficiency, and extend our framework to broader generation settings, including more open-ended tasks where fine-grained feedback is harder to define.
H TRAINING DYNAMICS ON KoDCODE AND ULTRRAFEDDBACK
Figure 5 compares the training dynamics of TEXT2GRAD against standard PPO on the KodCode and UltraFeedback datasets. In both cases, TEXT2GRAD exhibits significantly more stable convergence behavior, while PPO suffers from reward oscillation and inconsistent policy updates - indicative of poor gradient signal utilization in scalar-reward settings.
28

---

Published as a conference paper at ICLR 2026
RodCode
Uttebbaaback
7P0
Tolumi dead Booamt (Testatiaal
0:30-
0.20
Sequme Roward
24
Early Stop
02000
0.397
Kaaty Stap
Kaily Stop
01160
10 (b) UltraFeedback: TEXT2GRAD (red) vs. PPO (blue)
20
Traintog Samples (Theusanda))
Training Samplex (Thousand))
(a) KodCode: TEXT2GRAD (red) vs. PPO (blue)
Figure 5: Training reward curves comparing TEXT2GRAD and standard PPO. TEXT2GRAD demon- strates smoother, more consistent optimization with reduced oscillation - particularly critical in structured domains like code generation (KodCode) and nuanced preference modeling (UlraFeed- back). Shaded regions (if present) indicate one standard deviation over three random seeds.
I ABLATION STUDY ON SLF5K
To evaluate the contribution of key components in our framework, we conduct ablation studies on the SLF5K dataset. Table 2 in the main text shows that removing Ohain-of-Thhuuee (CoT) reasoning leads to consistent performance degradation across all metrics, confirming its importance for guiding fine-grained policy updates.
Figure 6 further illustrates the training dynamics by comparing the win rate of our full model against the variant without CoT reasoning. The consistent performance gap demonstrates that CoT-enhanced natural language feedback provides more actionable and semantically grounded signals for policy optimization.
1.1 EFFECT OF SPAN-BASED REWARD SELECTION
To directly address the core design choice in our method, we compare five reward strategies on SLF5K examining three orthogonal dimensions: supervision granularity (dense token-level vs. span-level), feedback source (human vs. model-generatedl, and within-span token weighting strategies (Tablle T
The dense token baseline performs substantially worse despite maximal supervision density. Analysis reveals that dense labeling produces a highly skewed distribution with approximately 70% of tokens labeled, predominantly on verbs and function words rather than semantically meaningful spans. This introduces noise into advantage estimates and destabilizes training, while costing 6-8x x more in annotation tokens.
For within-span token weighting, we test two alternative strategies against our uniform assignment. Token Importance assigns full rewards (=1.0) only to nouns and verbs within each span, while assigning reduced weights (+0.2) to others, under the hypothesis that content words carry greater semantic responsibility. Linear Decay assigns rewards proportionally to token position: for a span 0.9 for positive spans (and negated for of length n. the i-th token receives r; = 1.0 (i - 1) n-I
negative spans), concentrating credit on early tokens under the assumption that initial errors dominate autoregressive generation.
As shown in Table IT] both alternative weighting schemes underperform uniform assignment. Token Importance introduces part-of-spee classification noise that disrupts span coherence, while Linear Decay amplifies variance in the advantage estimator by concentrating gradients on initial tokens. Using human feedback with our span-to-token mapping achieves intermediate performance, confirm- ing that span selection itself provides value. The full TEXT2GRAD method with CoT critiques and uniform weighting achieves the best results, demonstrating that balanced credit assignment combined with structured reasoning sharpens span precision beyond natural human annotation and gradient reweighting heuristics. Our span-based approach maintains high grounding fidelity (unmatched rate <2.5%) while producing more stable advantages substantially lower cost.
29

---

Published as a conference paper at ICLR 2026
Table 1I: Ablation study on reward design choices (SLF5K), Bold: best results.
Method
R-L
BLEU
BERT-FI
Reward Granularity
Dense Token Reward
0.196
0.022
0.888
Token Weighting Strategy
Token Importance
0.237
0.059
0.889
Linear Decay
0.253
0.068
0.890
Feedback Source
Human Feedback (Uniform)
0.286
0.091
0.900
Ours (Full)
GPT--4o + CoT (Uniform)
0.291
0.094
0.902
First Model Win
Ties
Second Model Win
TEXT2GRAD (w/ CoT) vs
48.4%
11.8%
39.8%
TEXT2GRAD (w/o CoT) SIT
52.0%
10.8%
37.2%
PPO
46.0%
11.8%
42.2%
vs SFT
20
40
60
AD
100
Percentage (%)
Figure 6: Win rate comparison during training SLF5K. Our full model (TEXT2GRAD-B0 with CoT) consistently outperforms the ablated variant (without CoT), validating the effectiveness of structured reasoning in generating high-quality natural language rewards.
30

---

Published as a conference paper at ICLR 2026
J PSEUDOCODE FOR THE TEXT2GRAD FRAMEWORK
Algorithm TEXT2GRAD: Reinforcement Learning from Natural Language Feedback (Overall Framework)
Input:
Set of prompts for policy training.
Output: Optimized policy тe.
Phase 1: Dual-Feedback Reward Annotation (Described in Section 3.3)
1: Initialize dataset for reward model training DR +-
2: Generate initial responses y! for set of prompts z; (c.g., using a base policy).
3: for all prompt x, and its corresponding response Y/ do
4: AG))) 52) GennerateDuulFFeeheeek.. yi)
See Algorithm D c; is critique, A(ys) is span-JSON
5: Let 24 + [cx A(ys)
6: Add (zi, Vi. z4) DR.
7:
end for
Phase 2: Reward Model Training (Described in Section 3.4)
8: Rs TrainkewardModel(DD))
See Algorithm
Phase 3: NI--Gradient Policy Optimization (Described in Section 3.5)
9: Initialize policy (e.g., with pre-trained LLM) and value function Vy.
10: me f- OptimizePPlicc SrammizePoliyyiihNnGGrrrer (rathennrradee(((TT,, Rşg V₂)
See Algorithm
11: return Optimized policy πe.
31

---

Published as a conference paper at ICLR 2026
Algorithm 2 Dual-Feedback Reward Annotation (Section 3.3)
1: procedure SIIERAATDDULLFEEDACCKI
2: Input: Prompt æ, generated response = (V11....
3: Output: Natural language critique structured span-level reward map A(y), token-level
pseudo-rewards 8.
//
Annotation using strong LLM (e.g., GPT-4o)
4:
if human-written feedback is lacking (Reasoning-Augnen Augmented Annotation) then
5:
Guide LLM
6:
(1) Reason about the quality response step-by-step.
7:
(2) Output critique based on this reasoning.
8:
(3) Produce span-level JSON map A(y) associating spans 8t. C with labels
lg E (positive, neutral, negative).
9: else
Prompt LLM output critique e and span-level JSON map A(y).
11:
end if
Formally, (LIM( v) (c, A(v)), where A(y) : 8/: + lx
// Token-Leve Level Reward Mapping
13:
Initialize token-level pseudo-rewards ô (ōi...., .i.....,T))) with zeros.
14:
for all labeled span 83 in A(y) do Let & = [s)] if & positive then for all token index i. such that y/x E 8 do
15:
16:
17:
18:
6, +1.
19:
end for
else lg negative then
21:
for all token index t such that y/ E sy do 5, -1
23:
end for end
24:
D neutral spans are typically unannotated and default to 6;
25:
end for
26:
return c, A(y),
27:
end procedure
Algorithm 3 Reward Model Training (Section 3.4)
1:
procedure TRAINR
2: Input: Dataset DR = ((r,, Vs, = where z; [c;;
3: Output: Trained reward model Rg.
4: Initialize reward model parameters Φ
5: The reward model R3 is trained to predict given v, y: Po(2 x, [1] Pa(2)
6: Define the loss function: Ln(Ø) = (Z(z,,))) [logps(> x, 8)]-
7: Train RG by minimizing LR(Ø) on DR using teacher forcing and a standard causal LM
objective.
8: return Trained reward model Rg.
9:
end procedure
32

---

Published as a conference paper at ICLR 2026
Algorithm 4 NL-Gradient Policy Optimization (Section 3.5)
1: procedure WAAMMEEEPPOLYYTTTTL...R.EEEEEEE
Rgg. V Whn
2: Input: Initial policy Tew.. trained reward model Rg, initial value function V
3: Hyperparameters: Learning rates, PPO clipping entropy bonus 3 GAE 7, X.
4: Output: Optimized policy πч.
5: Initialize policy
Te value function Ve <- Vum.
6:
for each iteration iter = 1,.... Maxlterations do Let To ne.
7:
8:
Initialize batch of rollouts B + 0. for each sample I,.... NumSamplesPPrtteeation do
9:
10:
Sample prompt z.
Generate response y (y) (.1..... .r) Te.. xT.
12:
Generate feedback x' = (b)) ~ Ro x. x.
Parse (1y to get leven-level pseudo-rewards 8'
, 87) (using lines 11-20
of
Alg.
14:
A + 8; + +rk For t I,.... T: Trr rt Compute advantages Ai...., Ar. Fort= :...1:
r is an optional KL-penalty term.
15:
16:
A Eker E T
(o(I, (VVol,, (((z)))et1))
total,A
(Or use GAE: A =
E
17:
Add (x, y, ,' Alobal.)) to B. end for
18:
19:
for each epoch = 1,... NumEpochsPPO do
20:
for all (I,, y; 8', A, Alotal,A) in B do For
21:
,,..., T:
(o) no (vll2.
22:
æews (vc (x.))))
23:
-, LCLIP (6) = min (0) - c)A.)
24:
L. (vb) (Vo(s, (Et=r & kotttat 1)2
Value target is
discounted sum of rewards.
L. EENT (0) H(TT( x, xx,YE)).
26:
end for I, I.PPO Es, [L, CLIP (0) B.F BLENT(A))
28:
LV ( = EB, [LY((e)] (e)
29:
Update policy parameters: + aptimizer_step(( (PL Update value function parameters: 1 optimizer_stepp(t VoLVE())..
30:
31:
end for
end for
33:
return Optimized policy
end procedure
33

---

Published as a conference paper at ICLR 2026
K ANNOTATION AND TRAINING EFFICIENCY ANALYSIS
Table 12 compares the token consumption between span-level and token-level annotation approaches across our experimental datasets.
Table 12: Annotation efficiency: Token-level vs. Span-level comparison.
Dataset
Method
Tokens/Sample
Total Tokens
Token-Level
273.3
16.7M
UltraFeedback
Span-Lovel
40.1
2.4M (..)
Token-Level
170.4
1.5M
KodCode
Span-Level
50.1
0.5M (70%.)
First, we report wall-clock training time under identical hardware, data splits, and parallelism settings. Table 13 compares PPO and TEXT2GRAD using the same GPU configuration (detailed in Appendix D
Table 13: Wall-clock training time comparison (minutes per step).
Dataset
PPO
TEXT2GRAD
Overhead
KodCode
0.54
0.60
+11%
UltraFeddback
0.88
0.96
+9%
SLF5K
0.44
0.49
+11%
The extra time comes almost entirely from additional forward pass of the reward model per sampled trajectory. This single autoregressive pass jointly produces both the critique and the span map; does not add extra decoding stages or any backpropagation through the reward model.
Second, we quantify the annotation efficiency of the span span-based pipeline. As shown in Table 14 the total annotation budget small, and the cost per training sample remains on the order of 10 USD, which is consistent with the hear-parity in training time versus PPO.
Table 14: Span-level annotation cost breakdown (GPT-4o pricing).
Dataset
Input Tokens
Output Tokens
Total (USD)
Per Sample
UltraFeedback
5.18M
1.21M
$44.06
$0.0036
SLF5K
1.23M
0.12M
$7.99
$0.0016
KodCode
2.74M
0.46M
$20.60
$0.0025
Total
9.15M
1.79M
$72.65
L SPAN GENERATION FIDELITY ANALYSIS
To ensure the reliability of our reward signals, we verify that all generated spans are exact quotes from the model responses. Our annotation pipeline includes explicit instructions requiring "spans must be exact quotes from the response" and automated post-procsssing to remove any unmatched cases.
34

---

Published as a conference paper at ICLR 2026
Table 15: Span generation fidelity: Unmatched span rates across datasets.
Dataset
Unmatched Rate
Fidelity
KodCodc
0.93%
99.07% %
SLFSK
1.30%
98.70%
UltraFeedback
2.47%
97.53%
Average
1.57%
98.43%
The consistently low proportion of unmatched cases (under 2.5%) demonstrates high fidelity of span generation process, ensuring that reward signals are grounded in actual model outputs rather than fabricated or paraphrased content.
Robustness Tokenization Mismatch. Our pipeline is explicitly designed handle tokenizer differences between the annotation model (GPT-4o) and policy model (LLaMA). GPT-4o generates spans as character-level substrings (exact quotes), not token sequences, decoupling span identification from any specific tokenizer. The mapping procedure operates in three steps: (1) GPT-4o identifies spans as exact character-level quotes from the original response, (2) we locate the character interval [start, end] of each span in the raw text, and (3) re-tokenize this character interval using the policy model's sss tokenizer to obtain token indices. Reward attribution and policy updates operate entirely within the policy model's token space, eliminating dependency GPT-4o's tokenization.
The low unmatched-spe rates (0.93-2.47) 2.47%) demonstrate empirical robustness. Given that only 300 of tokens receive non-zero rewards, these error rates have minimal impact on gradient quality. When boundary mismatches occur due to different byte-pair merges, the affected regions default to zero reward, preserving gradient safety without introducing spurious signals.
M HUMAN ALIGNMENT ANALYSIS
To validate the quality of reward model's ' critique and span-level annotations, conducted comprehensive human evaluation study across all three datasets. For each dataset, we randomly sampled 100 instances and recruited three human annotators with expertise in the respective domains to evaluate the quality of generated critiques and their corresponding span selections.
Human annotators were asked evaluate two aspects of our reward model's output:
Critique Quality: Assess whether the natural language critique accurately identifies the strengths and weaknesses of the model response.
Span Alignment: Evaluate whether the selected spans (both positive and negative) are correctly identified and properly justified by the critique, using binary scale (Correct/Incoreett)
Table 16: Human evaluation: Agreement with reward model annotations.
Dataset
Accuracy
Task Type
KodCode
94%
Code (Objective)
SLFSK
86%
Summarization
UltraFeedback
82%
Open-domain QA
As shown in Table KodCode achieves the highest human agreement (94%), probably due to its well-defined code-centric critique tasks with objective ground truths. SLF5K exhibits strong alignment (86%), reflecting moderate subjectivity in general text evaluation. In contrast, UltraFeedback shows the lowest accuracy (82%). which we attribute to its open-ended, human annotators exhibit greater interrater variability due to the lack of rigid criteria. This trend confirms that our reward model performs most reliably in structured domains and remains robust under high subjectivity, validating its practical utility across diverse evaluation paradigms.
massoning-heave nature, where
35

---

Published as a conference paper at ICLR 2026
M.1 ERROR HANDLING AND FAILURE MODE ANALYSIS
To address concerns about incorrect or adversarial feedback, we analyzed failure modes in GPT-4o- generated annotations. Incorrect annotations are rare (<3%) and primarily involve loosely grounded or overly broad spans rather than hallucinated critiques.
We mitigate these errоrs through two mechanisms: (1) CoT-based reasoning prompts (Section 3.3 which enforce explicit justification before span selection, ensuring that every labeled span must be anchored to evidence the critique; and (2) Offline span-validation pass, which filters or re-annotates inconsistent annotations before reward-model training. This validation step checks for exact-quote matching (Table 15) and removes cases where spans cannot found in the original response.
Consequently, residual noise is minimal and has no measurable effect on downstream policy opti- mization. The combination of high human agreement (82-94%), low unmatched-spat rates (<2.5%), and validation safeguards demonstrates that our annotation pipeline empirically high-quality and robust occasional GPT-4o errors.
N BASELINE SETTINGS: PRM-PPO, DPO, AND PPO
For a fair comparison, we followed the methodology of Lightman et al. (2023). which defines Process Reward Models (PRMs) through explicit step-level supervision. This section details how PRM spans or steps were defined in each domain and explains why PRM-PPO was excluded on UltraFeedback.
Definition of PRM Spans or Steps.
SLF5K (summarization), GPT-4o decomposed each response into sentence- or clause-level content units, assigning binary correctness labels based on factual alignment with the reference summary. KodCode (code generation), GPT-4o segmented each program into code blocks logical statements and labeled each according to unit-test outcomes or reference execution traces. These labeled spans served as the oracle for reward- training and subsequent PPO optimization.
Table 17: Step-level annotations and PRM scores for SLF5K and KodCode.
Dataset
PRM (%)
Annotation Note
SLF5K
78.17
Sentence-fclause-le factual correctness based alignment with the reference summary
KodCode
80.63
Code-block or logical-stare statement correctness based on unit test and execution cecuution-trce -trace outcomes
Exclusion on UltraFeedback. We did not include PRM-PPO on UltraFeedback because the dataset represents open-domain QA, where responses are long (+= 276 tokens on average) and lack clear intermediate reasoning steps. As As[Lightman et al. (2023) show, PRMs are most effective in tasks with explicit multi-step reasoning, such as mathematics and code, where intermediate verification possi- ble. In contrast, survey that PRMs are primarily applied to structured reasoning domains- math, programming, multimodal reasoning, and robotics whilee outcome-level or semantie-feedback models remain more appropriate for general QA.
Applying PRM supervision to UltraFeedback would thus be both conceptually unsuitable and computationally expensive. Each PRM annotation would require GPT-4o to decompose the full answer into reasoning steps and verify each step's correctness, resulting in a 6-8x higher token budget compared with our single-pass critique -> span annotation pipeline. Given the absence of explicit reasoning chains and the high annotation cost, we excluded PRM-PPO for this domain.
Reward Signals for DPO and PPO Baselines. For SLF5K and Ulraaeeddbcck, we use the datasets' existing cejsen-rejected pairs and scalar preferences. DPO is trained exactly as in the original formulation using these preference pairs. PPO uses a ocalar-valued reward model trained on the same preference data via pairwise ranking.
For KodCode, where step-level correctness is directly verifiable, we construct chosen/rejected labels automatically: each candidate program is executed against unit tests, and passing failing
36

---

Published as a conference paper at ICLR 2026
define preferences. We additionally include outputs from GPT-4o and DeepSeek-rl to ensure diverse candidates. These labels are then fed into both DPO and PPO to ensure a fair comparison with TEXT2GRAD.
O SPAN LENGTH ANALYSIS
Our method does not impose fixed span lengths; spans generated dynamically by the reward model based on response content and structure. To analyze how CoT reasoning affects span length selection and annotation quality, examine the span length distribution and the relationship between span length and annotation accuracy on SLF5K.
Table 18 shows the span length distribution statistics, comparing CoT-enabled and NoCoT variants. The results reveal that CoT reasoning produces slightly shorter Good spans but is more willing to precisely localize problematic regions Poor spans. This suggests that CoT reasoning enables more targeted feedback by avoiding overly broad span selections. Q1, Q2, and Q3 represent the 25th-, 50th-, and pere-peee average token lengths per span, respectively.
Table 18: Span length distribution statistics on SLF5K (tokens per span).
Good Spans
Poor Spans
Metric
CoT
NoCoT
CoT
NoCoT
Mcan
8.59
8.09
6.94
7.28
Median
7.5
7.0
6.0
7.0
Min
2
2
1
Max
27
29
22
18
Q1 (25%)
5
5
4
5
Q2 (50%)
7.5
7.0
6.0
7.0
(75%)
10
We further quantify whether span length affects annotation quality by measuring accuracy across different length buckets. Table 19 shows span annotation accuracy stratified by quartiles of span length. Critically, with CoT reasoning. annotation accuracy remains consistently high 153.1%%96.%% across all length ranges for both Good and Poor spans, demonstrating robustness to span length variation. In contrast, without CoT, we observe boundary effects: accuracy notably higher the Q1-Q2 range (94.1%-94 (994.9%) but drops in shorter (0-Q1) and longer (Q4-100%) spans, suggesting that NoCoT struggles with both very short and very long span selections.
Table 19: Span annotation accuracy (%) by length quartile on SLF5K.
Good Spans
Poor Spans
Length Range
NoCoT
CoT
NoCoT
CoT
0-Q1
89.8
93.3
Q1-Q2
94.1
93.4
94.9
96.2
Q2-Q3
90.5
95.6
92.0
94.7
Q3-100%
91.2
93.1
91.4
95.3
Avg.
91.4
94.6
92.9
95.7
This sength-invaring accuracy with CoT reasoning directly improves downstream performance. As shown Table 2 Tex12Grad with CoT achieves ROUGE-L 0.291 and BERTScore 0.902, outperform- ing the NoCoT variant (0.275 and 0.898, respectively). This performance gap confirms that inaccurate or noisy span selections -atirtioonn-aatiuulalllyy the boundary effects observed without CoT-degrade policy learning effectiveness. CoT reasoning enables dynamic, content-driven span selection that maintains 93-96% annotation accuracy across all lengths, providing clean signals for policy gradients regardless of granularity.
37

---

Published as a conference paper at ICLR 2026
P LLM USAGE STATEMENT
Large Language Models (LLMs) played a significant role in both the research methodology and writing phases of this work. We acknowledge that their contribution was substantial enough to warrant detailed disclosure as per ICLR guidelines.
We used OpenAl's GPT-4o model core component of data generation pipeline. Specifically, GPT-4o was employed to generate the dual-feedback annotations (textual critiques and span-level labels) that constitute the training data for our reward model, as detailed in Section 3.3 This automated annotation process was essential for scaling our approach and represents a fundamental aspect of our methodology. The LLM was prompted with carefully designed instructions to ensure consistency and quality of the generated annotations.
Research Methodology:
Writing Assistance: We also used LLMs for language polishing and proofreading of this paper to improve clarity and readability. However, all technical content, experimental design, analysis, and conclusions were developed by the human authors.
38


Reviews:
[{"review_id": "DoHKcy3goF", "reviewer": "Reviewer_mXTR", "summary": "This paper proposes a reinforcement learning (RL) method called TEXT2GRAD, which turns textual feedback into span-level gradients. This method contains three core components, including: (1) a feedback annotation pipeline that pairs critiques with token spans; (2) a reward model that predicts span-level rewards on answers while generating explanatory critiques; and (3) a span-level policy optimizer that back-propagates natural-language gradients. Experimental results on three tasks demonstrate that natural language feedback, when converted to gradients, is powerful for policy optimization.", "questions": "I have included my questions in the weaknesses part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the proposed method is intuitive and the proposed method can support the motivation.\n2. The empirical performance is validated through three different tasks, which shows the generalization ability of the proposed method.\n3. This paper is overall well-written and easy to follow.", "weaknesses": "1. This paper misses existing works on reinforcement learning from natural language feedback especially in the training process, such as [1]. Since the proposed method exactly falls into this line of work, the authors should add a discussion to highlight the core novelty of the proposed method.\n\n2. The proposed method jointly generates free-form natural language critiques and structured span-level reward labels. But how natural language critiques can help improve the learning of span-level rewards is not clear. From Figure 1, the relationship between these two modules seems weak, which should be further clarified.\n\n3. The experimental part still has a lot of room for improvement. Now this part only include the main results of three datasets and a case study. In my view, at least a detailed ablation study should be conducted to show the contribution of the core parts in the proposed method.\n\n\n[1] Learning from Natural Language Feedback. TMLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SwLpez7yKK", "reviewer": "Reviewer_D36W", "summary": "This paper presents Text2Grad, a reinforcement learning framework that transforms natural language feedback into span-level gradient signals for policy optimization. It first builds a scalable annotation pipeline to collect span-level feedback, then trains a reward model on this annotated data to jointly generate critiques and span maps in one pass.  Building on this, the policy is updated through NL-Gradient Policy Optimization. Experiments on summarization, code generation, and open-domain question answering show consistent improvements over scalar-reward and prompt-based baselines, achieving more precise credit assignment, faster convergence, and better interpretability.", "questions": "1. How sensitive is performance to span length? Is there any correlation between span length and policy learning effectiveness?\n2. The paper assigns uniform pseudo-rewards across all tokens within a span. Is it possible to extract more fine-grained token reward signals from the critique?\n3. The PRM-PPO results are reported on SLF5K and KodCode but not on UltraFeedback due to annotation cost. Could you clarify (a) how PRM spans or steps were defined in each domain, and (b) what cost differences led to excluding PRM-PPO on UltraFeedback, given that GPT 4o based step level annotation might also be applicable.\n4. Table 1 shows your reward model has moderately negative token recall (22% and 43% on UltraFeedback and SLF5K respectively). How sensitive is Text2Grad to reward model quality?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The method is novel and well-motivated which bridges interpretable textual feedback and gradient-based optimization, addressing a real limitation of scalar RLHF.\n3. The method shows strong empirical results across diverse tasks with consistent improvements over strong baselines.", "weaknesses": "1. Binary pseudo-rewards discard fine-grained information in textual critiques. Moreover, treating all tokens within a labeled span equally is simple but ignores token importance.\n2. Computation overhead could be critical for method adoption, yet there is no empirical measurement comparing Text2Grad with other baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xhNupODaq5", "reviewer": "Reviewer_Go1F", "summary": "This paper proposes Text2Grad, a framework for converting natural-language critiques into token-level rewards for PPO training.  The paper collects natural language critiques paired with span-level annotations using GPT-4o and trains a generative reward model (RM) to output both textual feedback and token-level labels. Then, it uses the RM to label the correct and incorrect span from original output, scoring -1/+1 for each corresponding span token, respectively. The method aims to improve credit assignment and interpretable policy updates. Experiments in summarization, code generation, and QA show consistent improvements.", "questions": "1. How do the authors handle incorrect or adversarial feedback? What failure modes arise when GPT-4o produces wrong spans or hallucinated critiques?\n\n2. How robust is the method to tokenization mismatch across models or vocabularies? \n\n3. For gpt-4o annotation schema, how did the author handle predicted short spans occurs multiple times in the original response? String matching may fail.\n\n4. The author is suggested to provide results across scales and model families to support claims about method generality.\n\n5. How do the authors justify the claim of \"high-quality annotation pipeline\" without human agreement studies, audits, or error analysis?\n\n6. The authors report the human agreement, and how to measure the recall of span-level annotation?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is straightforward and easy to follow, around credit assignment and interpretability in RLHF.\n\n2. It introduces the LLM annotation pipeline in detail, which makes it clear for reproduction.\n\n3. Experiment results show consistent improvements across different datasets. The authors also provide a cherry-picked example to show its interpretability.", "weaknesses": "1. “NL-gradient” terminology is inaccurate: the method does not differentiate through language, but converts text to discrete span, and assign different span as token-level rewards (+1/-1 ). The method is not language-conditioned gradient flow.\n\n2. All feedback is GPT-4o-generated, making this RLAIF, not RLHF, with no human validation or study of noisy/contradictory real feedback. \nThe training data distilled from gpt-4o lacks real human feedback. The robustness to noisy or adversarial human critiques is unclear.\n\n3. The claim of \"high-quality annotation pipeline\" lacks human agreement analysis or data quality metrics; quality is assumed rather than demonstrated.\n\n4. The paper claimed that \"back-propagates natural-language gradients\" with \"feedback-conditioned adjustment\". However, there is no quantitative measurement of whether feedback actually conditions behavior at the criticized spans. The qualitative span-level evaluation is missing.\n\n5. The method relies on string-span matching tied to tokenizer boundaries, yet tokenization mismatch  (gpt-4o tokenizer vs llama tokenizer) can cause reward attribution noise. robustness to model/tokenizer differences is not evaluated.\n\n6. Evaluation is only on Llama-8B, and the paper does not test across different model scales or model families. Thus, claims about scalability and generalization are unclear.\n\n7. The method appears difficult to scale: span labeling, reward generation, and token-level PPO introduce significant engineering and compute overhead without evidence it transfers beyond the tested setup.\n\n8. The conceptual contribution leans heavily on terminology packaging, and the method does not clearly extend beyond the presented pipeline.\n\n9. Missing RLHF literature related to span-level optimization to improve credit assignment, such as [1,2,3]. The authors should state the position of this work in comparison to the literature and discuss the contribution.\n\n10. The conceptual contribution leans heavily on terminology packaging, and the method does not clearly extend beyond the presented pipeline.\n\n**References**:\n\n[1] MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions. ICLR 2025.\n\n[2] Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation. arxiv 2024.\n\n[3] SCAR: Shapley Credit Assignment for More Efficient RLHF. arxiv 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VCD6hYsNZL", "reviewer": "Reviewer_jPzf", "summary": "The paper introduces Text2Grad, a method for transforming natural-language feedback into token-level rewards for finetuning LLMs. While standard RLHF compares pairs of model outputs and assigns a single scalar reward to update the LLM weights, Text2Grad provides fine-grained supervision by identifying which parts of the generated response are incorrect. The approach works in three stages: first, a strong LLM (GPT-4o in the paper) is prompted to produce both textual critiques and span-level reward annotations for model outputs. The output of the first step is used as teacher knowledge to finetune a reward model. Lastly, the reward model's outputs are used to guide policy optimization via NL-Gradient PPO, giving the generator model more informative feedback during training. This enables the model to learn not just whether a response is good, but also how and where to improve it.", "questions": "- Intuitively, in many cases, only the first token in a \"wrong\" part of the output should be penalized. For instance, if the output sequence is \"The ca-pi-tal of Spa-in is Bar-cel-lo-na\", I would expect that the \"Bar\" would be penalized more than the following tokens, as all the tokens that are generated afterwards are conditioned upon the first mistake. Why do you think giving equal negative signal ($\\delta_t$ is always -1) to the entire part is justifiable?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Paper tackles a relevant and timely issue - specifically, with the proposed approach, authors enable more precise finetuning of LLMs and at the same time their method natively presents a basic form of explainability of the training process\n- Innovative idea of combining structured token-level rewards with gradient optimization\n- Evaluation of the method across multiple tasks", "weaknesses": "- Authors acknowledge that the one limitation is that the optimization severely depends on the quality of the reward model, but in addition to that, the method also requires that the \"upstream\" LLM (GPT-4o) produces valid training data for teaching the reward model. It's unclear how generally this assumption can be satisfied\n- Evaluation is in several parts unclear: For instance, in table 1 it is not clear how human annotation accuracy was obtained, nor what it represents. Also, how were the signals for finetuning models with DPO/PPO obtained? This is particularly important because without CoT, Text2Grad typically performs worse than PPO/DPO (tables 3 and 4)\n- No quantification of uncertainty in any table: how many times were the experiments run? Are the improvements over baselines statistically significant?\n- The experimental setup seems biased toward GPT-4o's judgments and lacks independent evaluation.\n- Method description should be improved: especially section 3.5 contains several symbols that are never introduced (e.g., $V$, $\\psi$, $r_k^{KL}$)\n\nMinor things:\n- Typo: `optimization. and present` (page 2)\n- Table 4: for ARC-C, GPT-4 has a higher value than GPT-3.5, so it seems that the wrong value is highlighted\n- GPT-4o and ChatGPT-4o refer to the same model, so I'd recommend using consistent naming throughout the paper\n- I'd suggest aligning the text contents with the content of the figure on page 4 (specifically, in the JSON excerpt, \"first time author\" is good, while in the figure the same part is poor span)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TS9sE1UFhL", "reviewer": "Reviewer_BDSv", "summary": "The paper presents Text2Grad, an RL framework that learns from natuaral language feedback to train LLM. Specifically, the method transfers the natural language feedback into span-level rewards, and use it in PPO.\n\nSpecifically, the method contains three steps:\n1. Use GPT-4o to generate a natural language model feedback, and lists of \"good spans\" and \"bad spans\"\n2. Train a reward model with GPT-4o generated feedback and lists of spans\n3. PPO with token-level rewards. Specifically, the tokens in the \"good spans\" has reward +1, in \"bad spans\" has reward -1\n\nThe authors experiment this method on Llama3.1-8B, and GPT-4o generated feedback, on summarization, code generation, and QA. The method outperforms scalar reward RL with PPO, DPO, reflection, and PRM-PPO on several metrics and LLM-as-a-judge comparisons.", "questions": "1. Performance when using other critics that is not GPT-4o. What if we use another LLM (for example, Qwen, or o3) as the source of feedback? or use human feedback? The paper will be more convincing if experimented with more than one source of feedback.\n\n2. More human evaluation. Is the trained models really better in humans' eye, or it is just doing some overfitting on styles?\n\n3. More baselines, as said in the weakness section.\n\n4. Talk about failure modes. Is there any reward hacking observed in the experiments? Is there any failure cases? Is that due to reward model failure, or some other reasons?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea that transfers language feedback into span-level rewards is interesting, and make sense. It also naturally fits into PPO frameworks\n\n2. The authros conduct experiments on multiple domains, showing gains on all the areas.\n\n3. The experiment and theory shows that token-level rewards give faster convergence compared with scalar-level rewards.", "weaknesses": "1. All experiment depdends on GPT-4o generated feedback. There are some other important baselines to compare with. For example, the algorithms that just SFT with responses refined by language feedback (in Scheurer et al., Training language models with language feedback at scale), and even just distilling from GPT-4o. Are these more effective approach to learn from human feedback.\n\n2. What if we use critics from sources different from GPT-4o. For example, from another LLM, or from humans. How will the performance change? Is the algorithm robust to the source of critics?\n\n3. Many results are heavily relied on LLM-as-a-judge. Many judges are GPT-4 based, which may inflate the gains because the model is trained with GPT-4o feedbacks. There lacks human evaluation on policy models' performance. There is only human evaluation on reward model quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)