Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations
Authors: Xinyi Yang
Liang Zeng
Heng Dong
Chao Yu
Xiaoran Wu
Huazhong Yang
Yu Wang
Milind Tambe
Tonghan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Continuous Normalizing Flow
Diffusion Model
RLAIF
Explainable AI
Primary area: foundation or frontier models, including LLMs

Abstract: As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain agent policies in natural language is vital for reliable coexistence. We introduce a general-purpose framework that trains explanation-generating LLMs via reinforcement learning from AI feedback, with distributional rewards generated by generative continuous normalizing flows (CNFs). CNFs capture the pluralistic and probabilistic nature of human judgments about explanations. Moreover, under mild assumptions, CNFs provably bound deviations from true human reward distributions when trained on noisy proxy rewards from LLMs. We design a specialized CNF architecture that selectively attends to linguistic cues in decision context and explanations when generating rewards. Human and LLM evaluators find that our method delivers explanations that enable more accurate predictions of true agent decisions, exhibit greater logical soundness and actionability, and impose lower cognitive load than explanations trained with proxy LLM rewards or state-of-the-art RLHF and RLAIF baselines.

Published as a conference paper at ICLR 2026
TRANSLATE POLICY TO LANGUAGE: FLOW MATCHING GENERATED REWARDS FOR LLM EXPLANATIONS
Xinyi Yang': :Liagg Zeng" Heng Dong', Chao Yu², Xiaoran Wu', Huazhong Yang',
Yu Wang', Milind Tambe'. Tonghan Wang"!
1 AIPD, Tencent, Shenzhen, China
2 III, Tsinghua University, Beijing, China
3 EE, Tsinghua University, Beijing, China
4 CS, Tsinghua University, Beijing, China
5 SEAS, Harvard University, Cambridge, USA
6 College of AI, Tsinghua University, Beijing, China
ABSTRACT
As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain agent policies in natural language is vital for reliable coexistence. We introduce general-purposs framework that trains explanation-generann with distributional rewards generated by generative continuous normalizing flows (CNFs). CNFs capture the pluralistic and probabilistic nature of human judgments about explanations. Moreover, under mild assumptions, CNFs provably bound deviations from true human reward distributions when trained on noisy proxy rewards from LLMs. We design a specialized CNF architecture that selectively attends to linguistic cues in the decision context and explanations when generating rewards. Human and LLM evaluators find that our method delivers explanations that enable more accurate predictions of true agent decisions, exhibit greater logical soundness and actionability, and impose lower cognitive load than explanations trained with proxy LLM rewards or state-of-the-as RLHF and RLAIF baselines.
LLMs via reinforcement learning from AI feedback,
INTRODUCTION
Intelligent agents, ranging from reinforcement learning (RL) agents (Kiran et al.
1021
Zhao et al.
202T Liu & Zhu 2024 Qiu et al. 2024), large language models (LLMs) Yao et al. 2022 Shinn et al.
2024
Wang et al. 2023a Kang al. 2020). to robotic systems (Ismail et al. 201 8). are becoming increasingly intertwined with daily lives (Wang et al. 2024a). To foster transparency, safery, and alignment, promising avenue is communicating the reasoning behind actions decisions generated by agent policies in natural language (Lazaridou et al. 2016). A good explanation provides a rationale that helps recover the underlying policy, enabling third parties to infer the decision. Such explanations allow humans to comprehend decisions, provide meaningful feedback, and intervene when necessary, ultimately enhancing trust and reliability within intelligent ecosystems (Cambria et al. 2023).
This effort is situated within the broader field of explainable AI and RL Arrieta et al. 2020 Car valho al., 2019 Ehsan et al., 2019) Gunning [0177] Ras al., 2018 Gilpin et al. 2018). which encompasses methods such causal inference and contrastive learning. Nevertheless, existing natural language explanation methods are often tailored to specific application scenarios, including self-driving (Cai et al. 2024), recommender systems (Lubos al.] 2024), stock prediction (Koa et al., 2024), robotics (Lu et al. 2023), and autonomous navigation (Trigg et al. 2024). To circumvent the ex- tensive task- and agent-specific engineering, there need for generalizable and mask-agnostic expla- nation method. We achieve this by concealing agent decisions from an EXPLANATION LLM, prompt- ing it solely with decision contexts, and training it to recover a specific underlying policy and generate
'The work was done while the author was a PhD student at Tsinghua. 1 "ooreeppnneecce tonghanwang 1996 @00mmill..cm

---

Published as a conference paper at ICLR 2026
explanations that reflect that policy's 'siicsss behavior. This approach forces the LLM to reason only from contextual information, independent of task or agent specifics, and thereby achieves generalizability.
In reinforcement learning from human feedback (RLHF), such third-party evaluations are provided by humans, In explanation tasks, human feedback is inherently pluralistic (Feng et al., 2024 Sooensen et al.] 20244] Kirk et al. 2024 Sorensen et al.] 2024b) and probabilistic (Lou al. 2024, Dorka 2024). as explanations are open-ended and subject to wide variation in individual judgments. This pluralism necessitates the costly collection of diverse human feedback in RLHI
Reinforcement learning from AI feedback (RLAIF) (Bai et al. 2022b Pace et al. 2024) Liu et al.
2024b) economizes human feedback, typically by adapting auxiliary LLMs to generate proxy rewards that approximate human feedback. Although early work (Li et al., 2024a) shows promise, current
RLAIF approaches have not yet been rigorously studied on how to generate distributional rewards that explicitly manage proxy errors relative to human reward distributions.
We close this gap by extending RLAIF with generative continuous normalizing flows (CNFs) (Chen) et al. 2018 Lipman al. 2023) as reward models. CNFs map base noise distributions to complex distributions, a process we exploit to represent reward distributions for training the EXPLANATION LLM. We train CNF to fit reward samples generated by separate PROXY LLMs. This method enjoys the theoretical benefit that, assuming the noise in these rewards generated by PROXY LLMs has the same functional form the CNF's base distribution (e.g., Gaussian), the CNF provably bounds its deviation from the underlying human reward distribution. Intuitively, the PROXY LLM acts as forward process that injects noise into human rewards, and the CNF learns its inverse, partially denoising and recovering the underlying human rewards. These analyses extend to all generative models based on ordinary differential equations (ODEs), including score-based diffusion models (Song et al. (20TT) Among them, we choose rectified flow (RF) (Liu et al. 2022 Albergo & Eijdeen (223) for its fast inference and high distributional fidelity. Appx. D demonstrates
the proof Ideas of this theoretical benefit.
To operationalize these theoretical benefits, we condition the RF reward model on decision contexts and explanations. We learn to transform inputs of the RF model into flow tokens that cross-attend to latent embeddings of decision contexts and explanations. The resulting attention summary informs the flow's output, allowing it to selectively integrate linguistic and contextual cues when assigning reward distributions.
We evaluate our method on both RL tasks (SMAC, (Samvelyan et al., 2019)) and LLM tasks (MMLU
(Hendrycks et al. (220)), MathQA (Amini et al. 2019)). Removing the rectified flow and directly using rewards from proxy LLMs decreases performance by 6.%%-12.6 12.4%, demonstrating the effectiveness of the generated rewards. Moreover, our method outperforms SFT, RLHF (Bai et al. 2022a), RLAIF (Bai et al., 2022b), and reasoning baselines (Trung al. 2024) by 1.9 1.9%-18 18.2% across all tasks and backbone models. Human evaluations further confirm the high quality of our generated explanations, which receive the highest scores of accuracy, logicality, actionability, and cognitive load.
1.1 RELATED WORKS
LLM explanations. Methods such as AMPLIFY (Krisha et al. 2024 Self-Explain (ajaaaaaaaaa et al.
2021), and Summarize and Score (SASC) (Singh et al. 2023) generate concise rationales based on agent decisions, sometimes accompanied by an explanation score to assess reliability. Unlike these methods, our approach trains LLMs to generate explanations without knowing agent decisions a priori. Chain-of-Thought (CoT) prompting (Wei et al. 2022) is widely adopted in-context learning technique that relies on step-by-step explanations or reasoning to enhance decision- -making. Self- Taught Reasoner (STaR) (Zelikman et al. 2022) introduces an iterative refinement method, where model improves its own explanations through self-generated rationales. While these methods are prompting-based and do not require additional training, optimization-basse CoT methods like CoT+SFT (Wei et al. 2022) and ReFT (Trung et al.) 2024) have been developed. We compare against these optimization- based methods in our experiments for their proven powerful reasoning capability.
RLHF, Reinforcement learning from human feedback (RLHF) (Bai et al. 2022a Wang et al. 2023b Ouyang et al. 2022 Dong et al. 2024) is critical to aligning LLM behavior with human preferences such as helpfulness, harmlessness, and honesty (Ganguli et al. 2022 Achiam et al. 2023 Team ct al. 2023). An RL-based method uses a reward model (Liu et al. 2024b) to approximate human preferences. Given dataset of preferred versus rejected responses, reward model can be trained
2

---

Published as a conference paper at ICLR 2026
using the standard Bradley-Terry model (Bradley & Terry 1952) with pairwise ranking loss. The policy model (LLM) can then be fine-tuned by learned rewards via PPO (Schulman et al. (017). However, training reward model can be costly. Direct preference learning (DPO) (Rafailov et al.) 2024) enables direct training on preference data, which can be adapted to different human utility models (KTO, (Ethayarajhh al. 2024). We compare against PPO, DPO, and KTO in experiments.
RLAIF, (Bai et al. 2022b) introduces RLAIF, integrating LLM- -LM-eerrrated preferences with human- defined principles for improved reward modeling. Subsequent studies, such Skywork (Liu et al.) 2024b) and West-of-N (Pace et al. 2024). further enhance this area by refining methods for curating preference data. While these approaches still partially rely on human annotations, other methods entirely exclude human data (Wang et al. [2024b; Wu et al. 2024b). In our experiments, we include comparisons with Skywork (L.uu al. 2024b), the leading RLAIF algorithm on the RewardBench leaderboard (Lambert et al. 2024).
Several recent RLHF and RLAIF studies train distributional reward models for LLM post-training. QRM (Dorka 2024) employs quantile regression to estimate multimodal preference distributions. DPRM (Li et al. (2024a) represents multiple preferences by categorical distribution and approximates the target distribution via an optimal transport objective. In our setting, both methods would need to discretize explanation feedback the probability that a third party can infer the true agent decision from explanations- and then fit distribution on that discretized space. By contrast, to model continuous feedhack, learning distributions without discretization. URM (Lou et al., 2024) parameterizes Gaussian distributions to model disentangled human preference attributes, whereas approach accommodates arbitrary reward distributions.
method seeks
Please see Appx. A for more related works on Diffusion in Transformer, Cross-attention, Explain- able AI/RL, and LLM Training in Explanations.
2 PRELIMINARY
Rectified Flow (Liu et al. 2022 Albergo Vanden-Fijnden, 2023) emerges as a powerful generative
model and has recently served the basis for popular commercial tools like Stable Diffusion
3 (Stability Al 2023). Based on continuous normalizing flow (CNF) (Chen et al. 2018 Lipman al.) (023), it models generative process
Ordinary Differential Equation (ODE). Formally, CNF transports an input žu E R to 21 = (t, zu) at time t E [0, 1] via the ODE:
_ (1,) 20) = (1, 0(1, 20))
Φ: (0,0 1] Rd - -R is the flow, and the vector field sp [0, 1] x Hd + Rd specifies the change rate of the state 2t. (Chen et al. 2018) suggests representing the vector field yp with a neural network. The flow transforms an initial random variable Zo ~ Po(2o) to Z₁ p1 (21) time
1. Rectified flow drives & to follow the linear path in the direction (2, Zo) as much as possible by
optimizing the objective: min fo E [][(Z] Zo) - sptt, 1= dt, where Z, t 21 + (1 (1-t) t) Zo the linear interpolation of Zo and Z1. Typically, the vector field network oP is implemented as a MLP for vector inputs (Wang al. 2024c).
U-Net (Ronneberger et al. 2015) for image inputs
Transformer. The Transformer architecture Vaswaani et al.
2017 is foundational to recent progress
in LLMs (Liu al. 2024a Zeng et al. 2024 Yang et al.
2024a
Team et al.) 2023). For an input ..... (...,,(((())) denote the sequence of
token sequence
(x ....., x(M)), let E [e(x
")...,)((()))))
token embeddings up to position n, where e(-) is the token embedding function. standard LLM
generates its output by
H TRANSFORMER (B(m)): M (y((t2)
x(Si) (h(")
where (n) E Rxd denotes the last layer hidden states for the first n tokens, with d representing the hidden dimensionality. The last layer hidden state at position n is (m)) = (n),,, :]. We denote the output projection matrix by W. the generation logits by M, and the output tokens by y.
3

---

Published as a conference paper at ICLR 2026
Explanation LLM
Reward Generation by Flow Matching up
Guidance LLM R
Input Context: The agent observes
Traio positive sumples
Loss
10)
Flow matching model merged with the
Previous actions rowards are
True action
Projectoon
[0] The decision is (Loog for decissions].
Output: Reasoning
Ricous
PC jsf
(57%) (Sentence 13] Initially,
moving south likely nim to
(s)(2)
(Sentence 2)] Moving casl
likely brings the ugent...
BC (559
True action
[s)
(Sentence 3)] Compared to
Resez
Roseza
(Loop for all decisions).
[Sentence
other directions, cast offers
Linear Laye:
Loyer
PPO Trinning
PC 1e
Reward(s;
Rewardts (12)
Emboedding
Layers
True action
PC (x) PC-18)
PC ln)
Gaussian Noise
PSC
[Loop for all decisions]
[Sentence decision
An overview of method. (Left) We prompt EXPLANATION LLM to generate reasoning about an agent decision based on the context information. Our focus is on whether a third party can infer the actual decision from this explanation. (Middle) We employ rectified flow model to generate rewards capturing how likely the actual decision appears as plausible outcome after each sentence of the explanation. Per-sentence rewards for training the EXPLANATION LLM are the changes in the probability of the actual decision (highlighted in blue). (Right) The architecture and training of the rectified flow jo are based on PROXY I.LM. The PROXY LLM provides positive samples, where, with the context and explanation as input, it produce distribution that assigns the highest probability to the actual decision. The rectified flow &p trained to produce such distributions with cross-attention layer in its middle that selectively leverages information from the PROXY LLM input, enabling generalization to negative samples.
Figure I:
METHOD
Our goal is to train an EXPLANATION LLM Te (0,) to recover a specific underlying policy and generate natural language explanations that reflect that policy's decision of RI. agents and LLMSs given only context. In RL, the decision is an RL action, and the context is the trajectory of preceding states, actions, and rewards. In LLM tasks, the decision could be. for example, the selected option for multiple-choice question, and the context is the question itself. Given set D. {(a,. ,))} of decisions aj and their context Cj. we construct prompt u for n. (0.): Given [Context c, Please analyze the reasoning behind the agent decision based on the context. The EXPLANATION LLM then generates the corresponding explanation yff Te (z);
To prevent superficial explanations that simply restate the decision, we intentionally exclude the actual decision a, from the prompt 2z. For the generated content be effective explanations, we train the EXPLANATION LLM using RL with rewards generated by rectified flow ((8.)). These rewards are probabilistic, reflecting the fact that human individuals have different opinions about the efficacy of the same explanation. Different explanation models correspond to different policies differentiated by training with distinct rewards. Fig. [1 illustraaees our learning framework. The architecture and training of p(B,)) are described the following subsection.
3.1 RECTIFIED FLOW REWARD MODEL
We aim at generating rewards that reflect how likely human individuals can infer correct decisions from explanations. To save costly human feedback (Bai al.) 2022b), we use LLMs proxy feedback providers when training the rectified flow reward model. Proxy LLM feedback may deviate from real human feedback. We empirically show the influence of these deviations baseline method Proxy LLM in Tab, and formally analyze how the rectified flow manages these deviations in Sec. 4
4

---

Published as a conference paper at ICLR 2026
Specifically, we query PROXY LLM with the prompt a/: Given [Context oy], the reasoning is [. Thus, the decision is (a decision a]. For each candidate decision a, we measure the likelihood of it being plausible outcome based on the token-level logits M (alaz) (Eq. 2 which
indicate how likely the PROXY I.L.M to output a at the end of xn.. If a spans multiple tokens, we compute the mean of their logits (Yang et al. 2022).
To capture the probabillistic nature of judgments on an explanation, we issue the same query for K independent PROXY LLMs, collecting logits {M, (= The rectified flow s0(pp) is trained
by minimizing the following loss function:
CF (9p)) = ExO Zo.(oz D.
[[](z] - -20) - (ttt, 2 cg. ff: 0f:00) 02)] (3)
where D, = [[(g. vf. (=))))))))) is the training dataset and
L. + (1 - -1) =o. for t E 10. The flow model transports each initial random sample 20 E TRAI drawn from Zo to a logit vector of the same dimension, where A is the set of all candidate decisions. With initial samples from Zo. (00+) effectively constructs a distribution over logits. We define rewards as the logits corresponding to the ground -ruth decision. Selecting them from the generated logits yields corresponding distribution over rewards. We discuss how to use this reward distribution in Sec. 3.2
The choice of Zo is critical for the rectified flow to handle noise in the PROXY LLM, as will discuss in Sec. 4
Rectified Flow Model Architecture. As in standard rectified flow model, the input to o includes (1) the current state E TR [R L E [o, 1), with zn sampled from Zo: and (2) a positional encoding (t(() corresponding to the ODE time E [0, 1]. Apart from these, the inputs also include (3) context Cj and explanations vf. This requires that yp can understand the linguistic cues in Cj and yf. which is beyond the capacity of typical rectified flow models based on fully-connected networks (Wang et al. 2024c or U-Nets (Liu et al. 2022). To solve this problem, we propose to embed the rectified flow model into an LLM.
Embedding. The context сj and explanations vff are encoded by the EXPLANATION LLM, and use its last layer latent states. ReLtivvatie layer-normalized MLPs şPEMr are used to project z, and (L((L) to have the same dimension as the LLM tokens. (F)(() and z, use two separate embedding MLPs: MEMB, = PEMB, c (t)) hEMB. = PPMM =, (z)). The resulting embeddings, stacked as E (hemm,r; fenn...)) ) E 2xd are called flow tokens.
Cress-Altention. We use the LLM's last layer weight matrices (Wo: W K, Wr) to generate queries, keys, and values of the flow tokens: (QEMB, KEMB, VEMB) = H EMR ( We, WK, Wv), which are concatenated with the last layer hidden states of the ej and yf to calculate cross- attentinn More details are discussed in Appx. [
Projector. Define HATIN, as the latent state of the flow token feMM. after cross-attention. This state has incorporated the ODE time t and explanatory information through cross-attention. Progressing from this state, we use a fully-connected network SPPRo Rd - IRAA with ReLU activation and layer normalization generate the vector field sp SPPro (hooshhArme (TATTNTE )- We find that skip-layer connections are important for training stability, and we append the inputs and t to the hidden layers of (PROS.
3.2 OVERALL TRAINING PROCEDURE
Once the rectified flow model yp is trained, we use it to generate rewards for samples in D..- Concretely, we solve the ODE dz, = (t,,, ze )dt starting from Zp sampled from the initial distribution. An advantage of rectified flow is that the vector field yp is encouraged be straight lines, allowing efficient and accurate solution of the ODE in a few steps. The solution z (at time 1) is taken as the
estimated logit MFLow (-1z7) = 21 20 + fo' #:)dt.
Given different initial samples 2g. we might get different logits MFLoW. which reflect probabilistic opinions. We use the average logit of the true decision MFLOW (a,lær)) as the reward. We then use PPO to train the EXPLANATION LLM to improve its explanations that achieve higher rewards from the rectified flow model. The rectified flow model and EXPLANATION LLM are trained alternately until convergence. The EXPLANATION LLM is men-t using PPO with LoRA. The rectified flow model consists of frozen LLM backbone and two trainable MIPs: PPEME and (PrOI-
5

---

Published as a conference paper at ICLR 2026
Implementation details. In practice, adopt rejection sampling (Tovrron et al.
2023 when training the flow model. Specifically, we only use samples where PROXY LLMs assign the highest likelihood to the true decision: aj arg max, (a] (a]xx)) 2x)). To mitigate the influence of raggemaant logit values, we apply softmax activation to M (- (x(r)) before using in the loss function (Eq. 3). Moreover, our model supports dense rewards. The logits change as we feed sentences in an explanation incrementally. We define per-sentence rewards as the changes of the true decision aj 's logit after each newly added sentence. The sentence-level reward is used in addition to the sparse reward, where a reward of is assigned only to the last token of the explanation if the predicted decision is correct; otherwise, the reward is O. More details are discussed in Appx. C We study the effects of rejection sampling and sentence-level rewards in Sec.
4 THEORETICAL ANALYSIS
In this section, we analyze the usage of rectified flow model to approximate rewards.
Let ((,) ) be the unknown human reward distribution for the explanation vf- As discussed, using LLMs as proxy human reward approximators might introduce errors. We first assume a Gaussian additive error model, and po (N(0, of (A)) is the corruption noise. The LLM-generated distribution is given by the convolution p = p * Pu- For rectified flow, we adopt an initial distribution of the same functional form the corruption noise: qo = N o (A))) Define the terminal distribution of the flow Φe as Q1 = 01:00- We can prove the following theorem.
Theorem 1. [Error bound for recovering the human reward distribution with a rectified flow] The flow is trained so that the fitting error to the proxy L.M reward distribution, measured in quadratic Wasserstein distance, satisfies W2(Φ, (P, 91) S. e. Prlow == @1:Po the denoised candidate obtained by pushing po through the learned flow. Errors relative to the true human distribution are bounded by
W W2(Pnoww.P) P) S L. LVA a -arl
where T. = exp(fo V/V di) is the Lipschitz constant. In many architectures, I, Ra 1.
The proof in Appx. D applies to any ODE-based generative models like score-based diffusion models. For other learning methods, such as previously proposed distributional reward modeling approaches (Dorka 2024 Lou et al. [024 Li al. 2024a), we have the following lemma applicable when they learn from AI feedback.
Lemma 2. |Error bound without rectified flow] Without rectified flow model, the error of learning model approximating p relative to the true human distribution is bounded by
W2(P. p) S E' +
|r
where e' is the model's approximation error with respect to P. Moreover, the inherent error in the proxy LIM reward distribution is W2(6.p) = VTA [or].
The major takeaway is that rectified flow turns an unavoidable bias term W2(2,)) = VAAl or into a controllable one as long as the initial distribution of the flow and the eттor model share the same functional form. reasonable estimation of hyperparameters, e.g., (o R σr), guarantees that a rectified model can well recover the true human reward distribution. We further provide theoretical analysis in Appx. [ for the case where Zo and po follow different functional forms. Figure 3 in Appx. D iillustrraes the proof ideas of our theorems.
5 EXPERIMENT
Benchmark. We evaluate our method on three testbeds: (1) SMAC (StarCraft Multiagent Chal- lenge (Samvelyan et al., 2019)) is an RL benchmark based on the real-time multi-agent strategy game StarCraft II that emphasizes micromanagement challenges. Explaining decision here requires analyzing the status of ally and enemy units, such as their remaining health points and positions, and inferring their intentions from trajectories. embodied Al benchmark in RL research (Li et al., 2024b] Wu et al. 2021). featuring household activity scenarios that demand multi-step planning, multi-robot coordination, and interaction with
(2) AI2-THOR (Kolve et al. 2017) is widely used

---

Published as a conference paper at ICLR 2026
dynamic environments. We use the Complex Tasks split from SMART-LLM (Kannan et al., 2024 and expand it from 8 to 59 tasks. (3) MMLU (Massive Multitask Language Understanding (Hendrycks) jet al. 2020 is multiple-choice QA benchmark for LLMs. We choose 4 challenging Professional Knowledge subsets (Professional Medicine (272 samples), Professional Law (1,53K samples), Professional Accounting (282 samples), and Professional Psychology (612 samples)). (4) MathQA (Amini et al., 2019) is a large-scale multiple-choice QA benchmark for math word problems, comprising 30K training samples and 3K testing samples. Please refer to Appx. Ffor more details.
Baselines. We evaluate our method against seven baseline families. (1) Proxy LLM. This baseline di- rectly uses rewards generated by the PROXY LLM to train the EXPLANATION LLM. (2) SFT is a pop- ular paradigm for adapting LLMs to downstream tasks. (3) RLHF We consider PPO (Xu et al. 22024).
DPO (Rafailov et al. 2024) and KTO (Ethayarajh et al. 2024); using the TRI. implementations (von)
Werra et al. 2020). (4) Reasoning Frameworks. We compare against CoT+SFT (Wei et al.) 2022) and
ReFT (Trung et al., 2024). (5) RLAIF. We report results for SKYWORK (Skywork (Skywork-Rewar--I.. 3.1-8B-v0.2) (Liu et al. 2024b), which ranks in the top two 8B models on the RewardBench leader- board (Lambert et al. 2024). (6) DeepSeek. We compare with DeepSeck-RI-Dee SLL-Distil--Llmma-- (Guo)
et al.
2025). (7) 03-mini. We directly use 03-mini (OpenAl 2025) for evaluation without fine-tuning.
Ablations. (1) Ours w/o Attn. We remove the cross -ross-attention layer in the reward model o and use fully-connected network to generate the vector field. (2) Ours (Sparse Reward). We use a single reward for the last token of the explanation instead of sentence-level rewards. (3) PPO (Sentence Reward). We apply sentence-level rewards to PPO. (4) Ours w/o Flow is the same the baseline Proxy LLM.
Models and Training. By default, we use Llama-3. -aama-3.1-8B-mme 1-8B--nstttett (Dubey et al. 2024) as the backbone of the EXPLANATION LLM. We also test different backbone models, including Llama-3.1- 8B, Qwen2.5--B-Insstee I-I-tuuttt (Yang et al., 2024a) and Gemma-2-2B-Ic (Lieberum et al. 2024) assess the robustness of our method.
All methods except o3-mini are trained on a shared synthetic dataset generated by ol-mini (Jaech ct al. 2024) and verified by human evaluators. We prompt ol-mini to generate paired Preferred explanations given ground- truth decisions and Rejected explanations given randomly selected incor- rect decisions. The training dataset for CoT+SFT and ReFT is further enhanced by asking ol o1-mini to adopt the CoT paradigm. We didn't directly use established instruction-tuning datasets because they provide preference data for answers, not explanations. To evaluate the quality of our synthetic dataset, 12 participants (2 females, 10 males, aged 20-40) are invited to rate 5% randomly chosen samples from each benchmark and their corre- sponding explanations on a 1-4 scale based on decision inferability. As shown in Tab. I the results (score mean + variance) indicate that preferred samples significantly enhance human inference ac- curacy compared to rejected samples, confirming the reliability of our synthetic dataset.
Table 1: Human evaluation of our constructed datasets.
Preferredt Rejected] 3.53(0.53) 2.26(0.76)
All methods except c3-mini undergo three epochs of SFT the preferred subset. Baselines (PPO, DPO, KTD, Skywork and ReFT) then perform preference optimization with the full preferred/rejected pairs. For our approach, the EXPLANATION LLM and the rectified flow model are trained iteratively for two rounds. As shown in Fig. 2 additional rounds provide only marginal gains. For the rectified flow model, we choose a 1. We use K 3 independent PROXY LLMs: Llama-3. 3.11-8..... -ns--untt 2ren2.557BBInntruue and 2em-a-2-2B-It. Complete training details and hyperparameter settings can be found in Appx. F
Evaluation and Metrics. For LLM evaluation, an explanation generated by the EXPLANATION LLM is considered effective if GPT--4o correctly infers the ground-truth decision based on it, with accuracy (ACC) measured as the proportion of effective explanations. We also consider the success rate of the whole task as the evaluation metric. For human evaluation, we recruit 45 participants (30 males, 15 females, aged 20-40) from diverse backgrounds (engineering, chemistry, machine learning, and economics). Each participant reviews 3 randomly selected samples from the benchmark and predicts the ground-truth labels. We report prediction accuracy and the percentage of participants rating each method the highest logical soundness, the highest actionability, and the lowest cognitive load. More details can be found in Appx. F
7

---

Published as a conference paper at ICLR 2026
5.1 COMPARISONS AGAINST BASELINES
LLM Evaluation. Tab. reports the results. For SMAC, MMLU, and MathQA benchmark, compared to Proxy LLM, our method achieves 12.4%, 6.9%, and 11.0% higher ACC on these datasets, respectively. These results support our theoretical analyses that rewards from PROXY LLMs yield ineffective optimization, while the rectified flow model better handles noisy proxy rewards. Among RLHF baselines, KTO and DPO perform relatively well due to strong human utility models. In Reasoning Frameworks, although ReFT improves ACC to 72.2%, 74.3%, and 76.3% on these datasets, still lags behind our method. Skywork, DeepSeek and o3-mini also underperform our method. The performance of our method remains significantly better than baselines in the challenging AI2- THOR benchmark. It improves the success rate by 7.4%, 6.0%, and 2.5% over KTO, ReFT and o3-mini, respectively. In summary, our method consistently surpasses all baselines, demonstrating its effectiveness in generating more reasonable explanations that support inferring correct decisions.
Table 2: Accuracy results of our method and baselines on SMAC, MMLU and MathQA. Success rate of our method and baselines on AI2-THOR.
Proxy LLM
RLHF
Reasoning
RLAIF
o3 mini
Datasets
Ours
SFT
DeepSeek
PPO
DPO
KTO
CoT+SPT
ReFT
Skywork
SMAC
0.764
0,582
0.640
0.653
0.713
0.721
0.642
0.722
0.692
0.658
0.455
MMLU
0.772
0.689
0.703
0.723
0.746
0.753
0.705
0.743
0.737
0.721
0.707
MathQA
0.804
0.667
0.694
0.719
0.753
0.758
0.712
0.763
0.729
0.718
0.739
AI2-THOR
0.702
0.447
0.628
0.642
0.483
0.476
0.677
Human Evaluation. In LLM evaluation, DPO and KTO lead the RLHF baselines, ReFT tops the reasoning frameworks. Therefore, for human evaluation, we compare our method against DPO and ReFT. Results on MathQA are shown in Tab. 3 Our method achieves the highest accuracy, exceeding the strongest baseline by 25.7%. Around 60% of participants judge our explanations to be the most logically sound and the least cognitively demanding, indicating improved decision comprehension.
Table 3: Human evaluations on MathQA, Table 4: Accuracy of our method and baselines with
Tabbl
where Cognitive denotes cognitive load.
rejection sampling.
Methods
ACC
Logic
Actionable
Cognitive
Datasets
Durs
PPO
DPO
KTO
Skywork
Oura
0.892
0.60
0.46
0.60
SMAC
0.764
0.642
0.704
0.708
0.713
DPO
0.591
0.17
0.28
0.18
MMLU
0.772
0.715
0.738
0.759
0.720
ReFT
0.635
0.23
0.26
0.22
MathQA
0.804
0.732
0.765
0.773
0.731
Rejection Sampling. We conduct control study by applying the same rejection sampling procedure from our method to the baselines. For preferred data, we select those where the PROXY LLM can infer the correct decision (called filtered preferred data). This removes possible noisy data. Our method uses this subset, while baselines that require pairwise training data use both this subset and rejected data. As shown in Tab. our method still achieves the best performance under this setting. By contrast, rejection sampling is not guaranteed to improve the performance of other baselines. These results show that the effectiveness of our method is not due the higher quality of the training data.
Masked Explicitly Stated Decisions. We conducted an additional evaluation in which any occurrence of the explicit decision in the generated explanation was masked before passing the evaluator model. As shown in Tab & this causes only slight drop in performance. Specifically, our method's accuracy decreases by 2.1% on SMAC, 1.8% on MMLU, and 3.6% on MathQA. We find that in most instances where the explanation states the decision explicitly, the decision can still be inferred from the reasoning trace (e.g., detailed numerical derivations or clear contextual cues). In these cases, masking does not significantly hinder the evaluator's 's ability to infer the decision. Baselines are also affected under this evaluation protocol, yet our method still outperforms the strongest ones, KTO and ReFT.
5.2 GENERALIZATION TO NEGATIVE SAMPLES: DISTINGUISHING TRUE AND FALSE NEGATIVES
As discussed in Sec. 2 train the rectified flow model o solely positive caaplls--xxplnaations that allow the PROXY LLM to infer the ground- ground-truth decision. It is critical that y generalizes to other unseen, negative samples and assigns them reliable rewards. Otherwise, the EXPLANATION
8

---

Published as a conference paper at ICLR 2026
08-
0.5
Curs we im
08
0.
0.6
0.
Negativ Nav Nagalioe
0.4
04
0.
Our
0.24
0:3 0.72
0.86
1,20
0.24
0
5.72
0.96
1.20
0.24
0.45
0.72
0.56
1220
43
06
122
Trnestops
(a) Positive Samples
(b) Negative Samples
(c) True/aalee Negative Samples
(d)
Figure 2: (a-b) Accuracy of the rectified model y unseen test samples. Flow ACC is reported as the percentage of samples for which the flow model ip assigns the highest logit (reward) to the ground- truth decision. (c) Our rectified reward model learns to assign low rewards to true negatives- explanations that lead to wrong decisions and high rewards to false negatives. Rewards are positively related to accuracy, represented by the y-axis. (d) EXPLANATION LLM accuracy increases with each training round. Each training step corresponds to processing a single training sample.
Table 5: A true negative sample from MMLU. Initially, the explanation from the EXPLANATION LLM erroneously supports answer B that focuses on the influence of the filter placement. Post- training with our method, the EXPLANATION LLM accurately justifies the correct answer.
Context (shortened): 67 67-year-old woman had pulmonary embolism... She underwent placement of an inferior vena cava (IVC) filter. She had a hematoma that was resolving... Which of the following the most likely cause of this patient's decreased sensation?
Decision set A: [A. Cerebral infarction during the hospitalization; B. Complication of the IVC filter placement; C. Compression of the lateral femoral cutaneous nerve D. Hematoma of the left thigh.]
Explanation: ... The patient's 's symptoms began after hospitalization, aligning with the placement of the IVC filter... The improvement over time supports a reversible cause related to the filter placement rather than other options... (B,x)
Improved Explanation (Ours): .i.nvolvvngg nerves near sites of cannula placement or previous bleeding sites... The sensation loss around the left thigh aligns with nerve compression near v the sites of recent interventions.
LLM would not receive informative reward signals. These negative samples fall into two categories. True negatives are invalid explanations rightly rejected by the PROXY L.M,, while false negatives are effective explanations misclassified by the PROXY LLM. Effective EXPLANATION LLM training requires distinguishing and differentially rewarding these types. We assess whether our method achieves this separation by first using ol ol-mini to label negatives as true false, and then examining the rewards that our algorithm assigns to each type.
Fig. 2(c) demonstrates that our reward model effectively differentiates between the two. Over the course of training, it gradually learns to grant higher rewards to false negatives and lower rewards to true negative
the desired behavior. This pattern is mirrored in the accuracy curve: in our framework, rewards are positively related to accuracy, and our reward model achieves approximately 80% accuracy on false negatives and only around 5% on true negatives. To see how this improvement happens, Tab. Tab.5lgives a true negative example where our method successfully guides the EXPLANATION LLM to refine the explanation, an update attributable to correctly computed rewards. Tab. example of false negative sample. Although the PROXY LLM cannot infer the correct decision, our method is able to deliver correct rewards. We hypothesize that this generalization capa- bility stems from the cross-attention mechanism. Fig. 2 (a) and (b) compare the accuracy of our flow model and the ablation Ours w/o o/o Attn on unseen samples. As we can see, without cross-attention, performance negative samples stagnates throughout training, underscoring its importance.
5.3 ABLATION STUDY
Different Backbone Models. In Tab. we change the backbone model of our method to Llama- 3-B, where our method still surpasses all baselines, achieving a minimum of 2.6% higher ACC on SMAC, 2.0% higher ACC on MMLU, and 3.7% higher ACC MathQA. We further evaluate the

---

Published as a conference paper at ICLR 2026
Table 6:
false negative sample from MMLU where the explanation is correct but the PROXY LLM produces an incorrect distribution. Although the rectified flow model was not trained on this sample, it correctly identifies "the egg roll was present for a substantial time", cue missed by the PROXY LLM, and thus provides a correct distribution.
Context (shortened): A wife and her husband were dining restaurant owned by a chef. As the wife walked past a table, she slipped on an egg roll that had been on the floor for quite some time, although the chef was unaware it had fallen there. If she sues the chef, she will most likely:
Decision set A: [A. Recover, because the egg roll on the floor constituted an unsafe condition of the premises; B. Recover, because the egg roll was on the floor for a substantial period of time before the accident V; C. Not recover, ...; D. Not recover, ...]
The fact that the egg roll was present for a substantial time v suggests that the owner should have been aware of the potential hazard and taken steps to address it. This situation falls under premises liability where maintaining safe conditions is crucial to
Explanation:
Distribution from the PROXY LLM: [0.9297, 0. 0.0674, 0.0010 0.0013] + 'A' X Distribution ş from the rectified flow model: [0.0547, 0.9330, 1089, 0.0685] - 'B'
robustness of our method using two more backbone models: Qwen2.5-7B-Instruuc and Gemma-2- 2B-I (abbreviated as Qwen and Gemma). As shown in Tab our method consistently improves ACC by up to 19.5% over SFT, confirming its effectiveness across diverse LLM backbones.
Sparse Reward Sentence-level Reward. As shown in Tab. 10 Ours (Sp Reward) represents the sparse-reward variant and PPO (Se Reward) represents PPO with sentence-level reward. Using sparse reward in our method leads at least 1.2% lower ACC, indicating that the sentence-level reward design offers better guidance for learning high-quality explanations. With sentence- reward, PPO still lags behind Ours by at least 5.2% ACC, underscoring the effectiveness of our rectified flow model.
Table 7: Accuracy of our method and baselines with differ-Table 8: Accuracy of our method and ent backbone model, Llama-3.1
3.1BB...
baselines with masked explicit decisions.
Datasets
Oura
SFT
DPO
XTO
CoI+SFT
ReFT
Datasets
Ours
KTO
ReFT
SMAC
0.689
0.583
0.632
0.650
0.655
0.614
0.663
SMAC
0,743
0.702
0.696
MMLU
0.724
0.638
0.682
0.698
0.673
0.653
0.704
MMLU
0.754
0.709
0.725
MathQA
0.746
0.626
0.662
0.683
0,709
0.631
0.695
MathQ4
0.768
0.732
0.735
Table 9: Accuracy of our method with another two Table 10: Ablations. (Sp Reward): Sparse reward; (Se Reward): Sentence reward.
backbone models.
Llama (Default)
Qwen
Gemma
SMAC
MMLU
MathQA
Datasets
SFT
Ours
SFT
Ours
SFT
Ours
Ours
0.764
0.772
0.804
SMAC
0.582
0.764
0.594
0.783
0.536
0.731
Ours (Sp Reward)
0.752
0.757
0.773
MMLU
0.689
0.772
0.693
0.789
0.653
0.740
PPO (Se Reward)
0.688
0.720
0.734
MathQA
0.667
0.804
0.645
0.778
0.626
0.723
Ours w/o Attn
0.662
0.721
0.728
Without Attention. As shown in Tab. 10 Durs w/o Attn underperforms Durs with a minimum of 5.1% lower ACC, indicating that the cunsformer-based architecture can better capture the complex context information. Fig. 2(a) and Fig. 2bb further elucidate the reason: it cannot even reproduce 100% accuracy on positive samples and cannot generalize to negative samples.
Appendix H reports additional results under noisy reward, varying and excluding SFT.
CONCLUSION
We demonstrate that a flow-matching generative model can provide sentence-level and reliable rewards for training LLMs to recover specific policy and explain that policy's decisions in both RL agents and other LLMs. Looking ahead, we envision extending this approach into a general-purpose L.M training framework that automatically generates high-quality reward signals.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
To assess the quality of the synthetic dataset generated by ol ol-mini, we enlist 12 participants (2 females, 10 males, aged 20-40) evaluate whether the samples are reasonable and human-like. For the human evaluation, we recruited 45 participants (30 male, 15 female, aged 20-40) from diverse fields, including engineering, chemistry, machine learning, and economics. To ensure consistency, we provided standardized instructions and illustrative examples before the evaluation. All participant information was anonymized, and no personally identifiable data were collected, ensuring that there was risk to privacy.
REPRODUCIBILITY STATEMENT
have undertaken comprehensive efforts to ensure the reproducibility of this work. The complete source code is provided in the supplementary materials. For theoretical analysis, detailed explanations of underlying assumptions and full proofs of the claims are presented in Appx. [ A complete and transparent description of the data processing steps employed in experiments is also included in the supplementary materials.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In conffeeencceccoeceecceccc Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arX3v:2303.008 2300008774 2023.
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-23756 2022.
Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023. URL
net Fowoororidooe. 17
Aida Amini. Saadia Gabriel, Peter Lin, Rik Joncel-Kedziorske Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- bassdd formalisms. arXiv preprint arXiv: 13319, 2019.
Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pp. 1168-1176. 2018.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):669 483, 2009.
Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82-18 115, 2020.
Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In International Conference Learning Representations, 2019.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 2022a.
11

---

Published as a conference paper at ICLR 2026
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXin:2212.08073, 2022b.
Osbert Bastani, Yewen Pu, and Armando Solar- Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
Tom Bewley and Jonathan Lawry. Tripletree: A versatile interpretable representation black box agents and their environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11415-118 11422, 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):32-3455, 1952.
Tianhui Cai, Yifan Liu, Zewei Zhou, Haoxuan Ma, Seth Z. Zhao, Zhiwen Wu, and Jiaqi Ma. Driving with regulation: Interpretable decision-making for autonomous vehicles with retrieval-augmented reasoning via Ilm. arXiv preprint arXiv:2410.04753 2024.
Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Navid Nobani. A survey on xai and natural language explanations. Information Processing & Management, 60(1):10311, 2023.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering. 2024.
Diogo V Carvalho, Eduardo M Percira, and Jaime S Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8):832, 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Mohamad H Danesh, Anurag Koul, Alan Fern, and Saeed Khorram. Re-understanding finite-state representations of recurrent policy networks. In International Conference on Machine Learning. PMLR, 2021.
pp.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 1188880-8944 2021.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405. 247800630,. 2024.
Nicolai Dorka. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409.1016 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The laaa herd of models. arXiv preprint arXiv:2407.21783 2024.
Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark Riedl. Automated rationale generation: technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pp. 263-274, 2019.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.013066 2024.
Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-llm collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4151-4171 2024.
12

---

Published as a conference paper at ICLR 2026
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07888, 2022.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE Sth International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEFF, 2018.
Omer Gottesman, Joseph Futoma, Yao Liu, Sonali Parbhoo, Leo Celi, Emma Brunskill, and Finale Doshi-Vedd Velez. Interpretable off-police evaluation in reinforcement learning by highlighting influ- ential transitions. In International Conference on Machine Learning, pp. 3658-3667, PMLR. 2020.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Femn. Visualizing and understanding atari agents. In International conference on machine learning, pp. 1792-1801. PMLR, 2018.
David Gunning. Explainable artificial intelligence (xai). Defense advanced research projects agency (DARPA), nd Web, 2(2):1, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Mohammadhosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Tom Melham, and Daniel Kroening. Deepsynth: Automata synthesis for automatic task segmentation deep rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35,
pp. 7647-7656, 2021.
Peter Hase and Mohit Bansal. When can models learn from explanations? formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201, 2021.
Ali Hatamizadeh, Jiaming Song. Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. In European Conference on Computer Vision, pp. 37-55. Springer, 2025.
Daniel Hein, Steffen Udluft, and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76:158-169, 2018.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Sandy H Huang, Kush Bhatia, Pieter Abbeel, and Anca D Dragan. Establishing appropriate trust via critical states. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3929- 3936. IEEE, 2018.
Zool Hilmi Ismail, Nohaidda Sariff, and E Gorrostieta Hurtado. A survey and analysis of cooperative multi-agent robot systems: challenges and directions. Applications of Mobile Robots, 5:8-14, 2018.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint 2xXiv:2004.0368. 2020.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Aman Jhunjhunwala. Policy extraction via online v-value distillation. Master's thesis, University of Waterloo, 2019.
Yipeng Kang, Tonghan Wang, and Gerard de Melo. Incorporating pragmatic reasoning communica-
tion into emergent language. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Published as a conference paper at ICLR 2026
Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Cheol Min. Smart-llm: Smart aaget- robot task planning using large language models. In 2024 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 22140-12147. IEEE, 2024.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, 6(4):383-39) 392, 2024.
Kelvin JL Koa, Yunshan Ma. Ritchic Ng, and Tat-Seng Chua. Learning to generate explainable stock predictions using self-reflective large language models. In Proceedings of the ACM Web Conference 2024, pp. 4305-4315, 2024.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt. Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 2017.
Anurag Koul, Alan Fern, and Sam Greydanus. Learning finite state representations of recurrent policy networks. In International Conference Learning Representations, 2018.
Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations of language models can improve language models. Advances in Neural Information Processing Systems, 36, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models 200::2403.378. 2024.
for language modeling. arXiv preprint
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference on Machine Learning, pp. 5979-5989. PMLR, 2021.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint 161iv:161.007/18 2016.
Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu, Aligning crowd feedback via distributional preference reward modeling. arXiv preprint 4:X1v:2402.09764, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19744 PMLR, 2023.
Miao Li, Wenhao Ding, and Ding Zhao. Privacy risks in reinforcement learning for household robots. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 5148 4154, IEEE, 2024b.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once on gemma 2. arXiv preprint arXiv:240Ss 05 147, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
14

---

Published as a conference paper at ICLR 2026
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork- Bag of tricks for reward modeling in llms. arXiv preprint MrXiv:2410.18451, 20244.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforeement learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 414-429. Springer, 2018.
Shicheng Liu and Minghui Zhu. Learning multi-agent behaviors from distributed and streaming demonstrations. Advances in Neural Information Processing Systems, 36, 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint urXiv:2209.0300) 2022.
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty-aware reward model: Teaching reward models to know what unknown. arXiv preprint arXiv:2410.00S: 2024.
Wenhao Lu, Xufeng Zhao, Sven Magg, Martin Gromniak, Mengdi Li, and Stefan Wermter. A closer look reward decomposition for high-level robotic explanations. In 2023 IEEE International Conference on Development and Learning (ICDL), pp. 429-436. IEEE, 2023.
Sebastian Lubos, Thi Ngoc Trang Tran, Alexander Felfernig, Seda Polat Erdeniz, and Viet-Man Le. Lim-generated explanations for recommender systems. In Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization, pp. 276-285, 2024.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison- Burchh. Faithful chain-of-thouge reasoning. In The 1 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia Asia-Pacific Chapter of the Association for Computational Linguistics (ICCNLP-AACL 2023), 2023.
Andreas Madsen, Sarath Chandar, and Siva Reddy, Are self-explanations from large language models faithful? arXiv preprint arXiv:2401.07927, 2024.
Katie Matton, Robert Osazuwa Ness, John Guttag, and Emre Kıeman. Walk the talk? measuring the faithfulness of large language model explanations. arXiv preprint arXiv:2504. /4150, 2025.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296-4304, 2024.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. Wt5?! training text-to-text models to explain their predictions. arXiv preprint urXiv:2004.145.0 2020.
OpenAl. Openai 03-mini system card. Technical report, OpenAI, 1 2025. URL https://cdn openai www/o3- --13-mmnni--
febd- .pff Accessed: 2025-09-21.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
Letitia Parcalabescu and Anette Frank. On measuring faithfulness or self-consistency of natural language explanations. arXiv preprint arXiv:2311.07466 2023.
Wenjie Qiu, Wensen Mao, and He Zhu. Instructing goal-conditioned reinforcement learning agents with temporal logic objectives. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PMLR, 2021.
Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilè Lukošiūté, et al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307. 1 1768, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Dheeraj Rajagopal, Vidhisha Balachandran, Eduard Hovy, and Yulia Tsvetkov. Selfexplain: A self-explaining architecture for neural text classifiers. arXiv preprint arXiv:2103.12279, 2021.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv: 190.002600, 2019.
Gabriëlle Ras, Marcel van Gerven, and Pim Haselager. Explanation methods in deep learning: Users, values, concerns and challenges. In Explainable and interpretable models in computer vision and machine learning, pp. 19-36. Springer, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiistd intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-24422222 241. Springer, 2015.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 1902::: 2019.
Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, and Markus Leippold. Towards faithful and robust Ilm specialists for evidence-based 2024.
arXiv preprint arXiv:2402.08277,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00::0077.7777 2017.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Andrew Silva, Taylor Killian, Ivan Dario Jimenez Rodriguez, Sung-Hyun Son, and Matthew Gombo- lay. Optimization methods for interpretable differentiable decision trees in reinforcement learning. arXiv preprint arXiv: 1903:09900003 2019.
Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. arXiv preprint arXiv:2305.09866 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021.
Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19937-19947, 2024a.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning, pp. 46280-46302 2024b.
16

---

Published as a conference paper at ICLR 2026
Stability AL. Stable Diffusion 3. http://////bbblllaaaaaaaaaaaaaaa 2023. [Online; accessed 24-Jamuary-2022
DeepSeek-A1 Team. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprini arXiv: 1805, 2023.
Jean-Francois Ton, Muhammad Faaiz Taufiq, and Yang Liu. Understanding thain-of- in llms through information theory. arXiv preprint arXiv:2411.1190 2024.
Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 25 141 2521, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXin:2307.09288, 2023.
Lena Trigg, Brandon Morgan, Alexander Stringer, Lacey Schley, and Dean Hougen, Natural language explanation for autonomous navigation. In 2024 AIAA DATCЛEEE 43rd Digital Avionics Systems Conference (DASC), pp. 1-9. IEEE, 2024.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7601-7614, 2024.
Anwaar Ulhaq and Naveed Akhtar. Efficient diffusion models for vision: A survey. arXiv preprint arXiv:2210.09292. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https:: / github. wwtp///gggggggcc//////aaaaa 2020.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ender embodied agent with large language models. arXiv preprint arXiv:2305. /629/, 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18:1-26, 2024a.
Tianlu Wang, Ili Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-YY Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint arXiv:2408. 02666, 2024b.
Tonghan Wang, Heng Dong, Yanchen Jiang, David C Parkes, and Milind Tambe. On diffusion models for multi-agent partial observability: Shared attractors, error bounds, and composite flow, arXiv preprint arXiv:2410. 13953, 2024c.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:231 1.0952,, 2023b.
17

---

Published as a conference paper at ICLR 2026
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 200.0.2200 2021.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems, 35:24824-225 2222472--4837, 2022.
Qiaoyun Wu, Kai Xu, Jun Wang, Mingliang Xu, Xiaoxi Gong, and Dinesh Manocha. Reinforcement bearning-based visual navigation with coormatiionthhoreete regularization. IEEE Robotics and Automation Letters, 6(2):731-738 2021.
Shaojin Wu, Fei Ding, Menggi Huang, Wei Liu, and Qian He. Vmix: Improving
diffusion model with cross-attention mixing control. arXiv preprint arXiv:2412.20800, 2024a.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar, Meta-rewarding language models: Self-improving alignment with llm--s-a-metta-jddge. arXiv preprint arXiv:2407. 2et:::2077.999.4.. 2024b.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111 15/15, 2024a.
Heeyoon Yang, Gahyung Kim, and Jee-Hyoung Lee. Logit averaging: Capturing global relation for session-based recommendation. Applied Sciences, 12(9):4256, 2022.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406. 10216, 2024b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:2461 246244 2022.
Tom Zahavy, Nir Zri-er,,em, and Shie Mannor. Graying the black box: Understanding dqns. In International conference machine learning, pp. 1908. PMLR, 2016.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35: 00.555477818888 2022.
Liang Zeng, Liangjun Zhong, Liang Zhao, Tianwen Wei, Liu Yang. Jujie He, Cheng Cheng, Rui Hu, Yang Liu, Shuicheng Yan, et al. Skywork-math: Data scaling laws for mathematical reasoning in large language models-the story goes on. arXiv preprint arXiv:2407.08..
2024.
Hengzhe Zhang, Aimin Zhou, and Xin Lin. Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis. Complex & Intelligent Systems, 6(3):741-753, 2020.
Mingde Zhao, Zhen Liu, Sitao Luan, Shuyuan Zhang, Doina Precup, and Yoshua Bengio. A consciousness -nnspiueee planning agent for model-based reinforcement learning. Advances in neural information processing systems, 14.1569-1581.. 2021.
Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. Towards interpretable natural language understanding with explanations latent variables. Advances Neural Information Processing Systems, 33:6803-6814, 2020.
18

---

Published as a conference paper at ICLR 2026
A MORE RELATED WORKS
Diffusson in Transformer (DiT, Yang et al. 2023)) leverages the strengths of self-attention of Transformers to improve the performance of diftusion models across a range of tasks, including image and text generation (Cao et al. 2024). (Dhariwal & Nichol (02T) demonstrate how Transformer-
based architectures can optimize the denoising process in diffusion models, resulting in high-quality image synthesis. (Uhhaq & Akhtar 2022) explore efficient implementations for diffusion within Transformer. These works are related to our work, as we embed flow matching into the last layer of an LLM. The difference is that we train the LLM-augmented flow matching model as reward model for the purpose of LLM post-training.
Cross-attention is popular technique for processing information across multiple modalities (Rad- ford et al. 2021 Alayrac et al. [002] Li et al. 2023). Approaches such as 121-Adapter (Mou et al. 2024) and VMix (Wu et al. 2022a) use cross-attention mechanisms between text encoders (an LLM) and diffusion models to enhance the generation of high-quality images from textual descriptions. More generally, cross-attention has helped solve tasks that require both vision and language under- standing (Hatamizadeh et al. 2025) Cao et al., 2024). Different from previous work on DiT and pross-attention-biae image/vides generation, our best knowledge, the proposed method is the first to use generative models and cross-attention to generate rewards for Rl.-based LLM training.
Explainable AL/RL without natural language explanations. Our method is suited within the domain of explainable AI (Arrieta et al. 2020; Carvalho et al. | | |Ehsan et al., 2019) Gunning 2017 Ras et al. 2018 Gilpin et al., 2018). Faithfulness (Jacovi & Goldberg, 2020) is one of key concepts from the XAI literature, which is defined the degree to which an explanation reflects the model's 's actual decision- process, rather than merely presenting a narrative that sounds plausible to humans (i.e., plausibility).
Recent studies have explored how to generate faithful explanations from large language models. Some approaches (Lyu et al. 2023 Radhakrishnan et al. 2023) focus enhancing faithfulness through Thought prompting, breaking down the reasoning process into multiple sub-sentences. Others (Schimanski et al., (024)) improve faithfulness by augmenting the training data to provide stronger supervisory signals. For measuring rational faithfulness, Matton et al. (2025) introduces causal counterfactual evaluation framework using an auxiliary LLM and Bayesian hierarchical model. [Parcalabescu & Frank (2023) proposes CC- CC-SHAP, token-level self-consistenge measure that records model contributions. [Mdsenn et al. (2024) measures self-explanations and find that faithfulness varies significantly across models and datasets. Our method captures the essential behavior of the model by faithfully reproducing its outputs from the given inputs, thereby serving as an interpretable intermediate representation of model behavior. In this sense, our approach offers a form of faithfulness from a functional and behavioral perspective.
For explainable RL (XRL), our approach draws particular parallels with it. Post-hoc XRL methods focus on relating inputs and outputs of a trained RL policy an interpretable way, using an inter- pretable surrogate model as policy approximation. Examples of surrogate models include imitation learning (Abbeel & Ng 2004), learning from demonstration (Argall et al. 2009), finite state ma- chines (Koul et al. 2018 Danesh et al., 2021), decision trees and their variants (Bastani al. 2018 Jhunjhunwala, 2019, Beeley & Lawry, 2021 Liu et al. 2018), genetic programming (Zhang et al. 2020). programmatic policy searching (Verma et al. 20118), and deterministic finite automata (Hasan- beig et al. 2021). However, in order to be interpretable, surrogate models are designed as simple as possible. Ad-hoc XRL methods represent policies as inherently interpretable models. For example, Silva et al. 22099) Topin & Veloso )2019); Hein et al. (2018); Landajuela et al. (2021) use decision trees as policy approximators. However. the capacity of these models is typically ly limited. Saliency maps distinguish observation elements that influence decisions (Atrey et al. 2019: Greydanus et al.) 2018 Gottesman et al.) 2020), but does not capture the reasoning behind decisions (Atrey al. 2019).
lcaving humans to give ad-hoc explanations based on these visual cues. Example-based methods use similar experiences to explain decisions (Amir & Amir 2018 Huang et al., 2018 Zahavy et al.
2016 Topin & Veloso 2019), but largely overlook underlying reasoning.
Use explanations to train LLMs. Various prior works have explored training or tuning language
models with explanatory cues, often relying on techniques such span word highlighting (Hase & Bansal [202T: Zhou et al., 2020 Narang et al.) 20200 Rajani et al., 2019). Moreover, explicitly training language models with natural language explanations instructions has been shown enhance their
19

---

Published as a conference paper at ICLR 2026
ability to leverage explanations in prompts (Wei et al. 2021 These works are different from method, because we do not use explanations to train LLMs. Instead, we train LLMs to generate explanations.
B NOTATIONS FOR TRANSFORMERS
We consider LLM with L layers, the hidden state after / layers, (( is projected by three weight matrices Wo. WK. and Wv to the query, key, and value embeddings Q(m.9) K(nl), and (n.)), respectively. The self oeff-attention is calculated as:
(Q(m.), ((n.),, v(nl)) (H (Wo, WK, Wv),
T A (m.1) ((..)) k(ad)" /Vdx.
Attn(H (m.)) (A(ln))) where (.) is SoftMax, and A is the self-attention matrix.
C LEARNING FRAMEWORK
EXPLANATION LLM. Our method is developed around the EXPLANATION L.M.. Given a set D. {(aj,c)) 3ز= of decisions a, and their context cj. we use the following prompt zÿ to ask the EXPLANATION LLM generate explanations: Given [Context cj] Please analyze reasoning for the agent decision based on the context.
Suppose that the EXPLANATION LLM generates Ke sentences as output: v5= For the generated content to be effective explanations, we hope that one can consistently infer agent decisions from explanations across various contexts. To this end, we seek feedback regarding how likely the actual decision aj is a plausible outcome given each incremental portion of the explanation.
,8 ())
Such feedback is most accurate when provided by human annotators. However, human feedback is expensive (Bai et al.] 2022b). We first discuss alternative approach that uses PROXY LLM as surrogate feedback provider, whose downside will motivate the proposed generative reward method introduced in the next subsection.
We query the PROXY LLM with the prompt: Given [Context cj]; the reasoning is
((I.)) ). 8y
Thus, the decision on decision E A). Here, 8y ((Lok) is the first sentences of the explanation
o(k) uf. We denote this prompt to the PROXY LLM by a (uf). The dependence on y will be omitted when unambiguous.
We are interested in the likelihood decision a E A appearing at the end of 2;j g(k) (vf), influenced by
logits ((a [ (x) (v()) (Eq. 2). In practice, a is represented by some tokens describing the decision. If it involves multiple tokens, we calculate their mean Yang et al. 2022).
By applying the SoftMax operation to (alx)) (uf)), A, get distribution decisions:
p(ay p(aylag (K) (1g(0) = SOFTMAX, (M(alog()))
This distribution is defined for the first & sentences, == 1,... Ke. Intuitively, the distribution p changes as we feed the sentences in the explanation y incrementally. These changes measure the contribution each sentence to the effectiveness of the explanation, allowing us to define per- sentence rewards for vff the changes of the likelihood of the actual decision ay after each newly added sentence:
r(s (k))( = p0pa,a=3()))) p(a, 1 (1)-1)))
(5)
which can be understood information gain (Ton et al. 2024). Calculating sentence-level rewards a trade-off (Team) 2025). We benefit from denser reward signals compared to a single reward for the whole explanation, and also avoid the costs of per-token reward calculation.
20

---

Published as a conference paper at ICLR 2026
Human Feedback Distribution
Transform
Transform
Distance
Initial Distribution
LLM Feedhack Distribution
Generative
Diffusion-based Learning
Process
Rectified Flow Model
Feedback Distribution
Figure 3: Overview of the idea of our theoretical analysis for the proposed flow- flow-matching -a--hhhhhinn-eedd reward generation framework. We train a Now-matching model to approximate the distribution of L.M--genertte feedback. Assuming the L.M feedback distribution is transformation of the ground- truth human feedback distribution, we derive a closed-form expression for the distance between the induced fow-matching reward distribution and the human feedback distribution.
We intentionally exclude the actual decision a, from the prompt He to the EXPLANATION LLM. Otherwise, the rewards ((s)(k)) might be trivial and encourage merely restating the decision: the sentence that discloses the decision will get a very large reward, while the following sentences get fairly small rewards, regardless of their content.
The dasade of directly using this PROXY LLM that the rewards r(s, (()) could noisy or inefficient (Yang et al. 2024bb, proven by baseline method Proxy LLM in Tab. We propose to fix this problem by introducing a rectified flow model for reward generation.
Rectified Flow Model Architecture.
Cross-Attention. (WQ. WK, Wv) are last-layer attention weights of the EXPLANATION LLM. We apply these weights to the flow-token hidden states (HEMB) to obtain corresponding queries, keys, and values: (QEMB, KEMB. VEMB) = HEMB (Wo, WK. Wy). In parallel, we also obtain the last-layer queries, keys, and values from the EXPLANATION LLM for the given context (i.e., the query and its explanation), denoted Q(4-1), (L-1), ((L-1), where L is the number of transformer layers. We then perform standard multi-head attention using the concatenated queries, keys, and values ([Q(L-1), Q FM s), (L-)) K KEMB), [V(L-1), VEMB)). Finally, we extract the attention hidden states at the flow-token positions as the output.
Projector. Define hATTN. as the latent state of the flow token hEMR.E, after cross-attention. This state has incorporated the ODE time and explanatory information through cross-attention. Progressing from this state, we use four-layes fully-connected network (PPRo IR + R AAl with ReLU activation and layer normalization to generate the vector field spt,, 2:) sProo (ATTT...).. We find that skip- layer connections are important for training stability. Specifically, we append the inputs z, and to the hidden layers (PPro)-
D RECTIFIED FLOW CAN CORRECT ERRORS
Theorem 1. [Error bound for recovering the human reward distribution with a ]low The flow is trained so that the fitting error to the proxy L.M reward distribution, measured in quadratic Wasserstein distance, satisfies Wz(p, 91) < PFLow == 01 is the denoised candidate obtained by pushing po through the learned flow. Errors relative to the true human distribution are bounded by
W W2(proow.D) .) Set L
where L exp(Io || di) the Lipschitz constant. In many architectures, L Rs
Proof. Recall that for any probability measures μa, Rdd the quadratic Wasserstein distance is
21

---

Published as a conference paper at ICLR 2026
Jeuwpe -112= ())
W (1, -
inf
where T(,, the set of all couplings of j and Moreover, if f Rd - IRd is L-Lipschitz then
Wa(Sun, feu) s L22(1)))
We want to bound W2 (PeLow, p). By the triangle inequality.
W2(PFLOw; p) W2 (PFLOw n1) + W2 P) + W2 (p: p).
The middle term is exactly training error: W2(gi, p)) S s. For the last term, (2 p), we observe Po shares its mean with since the corruption is mean-zero Gaussian, the
the fact that f optimal coupling between and ő is to pair each clean sample with x where & ~ pos giving zero squared cost in expectation.
Hence suffices to bound the first term, W2(Proow qı).
Since
PrLOw
01;Po.
91
$1:90.
and Φ1 is L-Lipschitz by construction.
W2 (PrLow (1) = W2(O11Po, )1:40) Σ L L22(po go).
For two sero-mean Gaussians in R Al, N/O, oi and N r² one shows
W2(2o, m0) = V A or a..
Combining,
W2(pnow, p) S L LVA σ orl + E,
which is exactly the claimed bound.
Lemma [Error bound without a rectified flow] Without a rectified flow model, the ervor of learning model approximating p relative to the true human distribution is bounded by
Wz(6.p) S E' VAi ar|:
where e' is the model's approximation error with respect to P. Moreover, the inherent error in the proxy L.M reward distribution is W2(6.p) = V|Al |r|
Proof. First, by the definition of e' the model's approximation error to have
W2 p) < e'.
Next, apply the triangle inequality for the 2- 2-sssserstein distance:
Wa p) < Wa(6. D) Wa(0, p). Combining with the above bound gives
(2 p) < e + We( p)-
22

---

Published as a conference paper at ICLR 2026
It remains only to compute (2(P,)) p). Recall that p Po where Pu =N(00 Since p and P differ by the addition of zero-mean Gaussian of covariance 1/ one shows directly for two zero-mean Gaussians in |R|A|| that
(2 p) VIAl or|-
Hence
W2(6, p) S + VIA |rll
as claimed.
D.1 WITH DIFFERENT FUNCTIONAL FORM
The following is the general case where the initial distribution go and the error model po follow different laws.
For the true distribution and the flow model's output Pnow, the error is bounded by
W2(Pnow p) sc Ep, + L//Mpp + tr Epo + tr Ego
where E. denotes the covariance matrix and μ.. is the mean. When both Po and are Gaussian, this reduces exactly to Theorem 1.
The human-feedbaed distribution is unknown, but we assume we observe noisy proxy provided by the LLM with an additive noise model po. the proof, the flow network acts as a denoiser: transforms the proxy toward the latent human distribution. The inequality above quantifies how closely the denoised distribution can approximate the true one.
E PROMPTS
Below are the prompts used for the EXPLANATION L.LM and the PROXY LLM on the SMAC datasets. For the EXPLANATION LLM,, the prompt describes the overview of the scenarios along with the previous action-state pairs and instructs the EXPLANATION LLM to analyze the provided information. In response, the EXPLANATION LLM generates summary of the preceding trajectories and implies the action selection. For the PROXY LLM, the prompt includes a statement summarizing both the input to and output from the EXPLANATION LLM. followed by the phrase: 'Therefore, the missing action is a possible action from A'. Then, we calculate the mean token logits for each possible action in A, which then used to determine the action with the highest likelihood.
23

---

Published as a conference paper at ICLR 2026
An example that the prompts used for the EXPLANATION LLM and the PROXY LLM on the SMAC datasets
User for EXPLANATION LLM:
Q: Here are 5 allies and 6 enemies in SMAC. The goal of each ally is to attack all the enemies together with other allies. At every step, each ally can choose one action take from the Action set. The trajectory of each ally consists of a series of State, Action, Value pairs. Here are 4 consecutive State, Action, Value pairs where the last state-action pair misses the Action: xxx. Please help me concisely summarize and analyze this part of the trajectory for the missing Action prediction.
EXPLANATION LLM:
R: Throughout the provided trajectory, the ally consistently maneuvered strategically while maintaining full health and ensuring weapon readiness. Initially, moving SOUTH likely aimed to xxx. The subsequent WEST movement improved xxx. The predicted next action should be a directional movement aimed at advancing toward a more optimal attacking position. Compared to other directions, EAST offers the best alignment for coordinated attacks, ensuring allies can effectively converge and target enemies together.
User for PROXY LLM:
Q+ R + Therefore, the missing action is (a possible action from A).
PROXY LLM:
The token logits the given possible action.
An example that the prompts used for the EXPLANATION LLM and the PROXY LLM on the
AI2 THOR datasets
User for EXPLANATION LLM:
Q: There are multiple robots in the fimulation,eade with different skills and capabilities. The robots are tasked with performing a series of actions in a simulated environment complete certain task. Robot one xxx. Task is xxx. Current environment condition is xxx. The previous actions include xxx. The available actions are xxx. You act as an Al assistant to help the robots decompose the tasks into actions step by step.
EXPLANATION LLM:
R: Toggling on the television and placing the keychain on the coffee table. Robot 0 now should take corrective navigation step (GoToObject) before successfully executing ToggleOnObject. After completing this sub-task, Robot 1 should navigate to the coffee table and keychain, pick up the keychain, and place it on the table in the correct order.
User for PROXY LLM:
Q + R + Therefore, the next action is (a possible action from A).
PROXY LLM:
The token logits of the given possible action.
Above are the prompts used for the EXPLANATION LLM and the PROXY LLM on the AI2-THOR datasets. For the EXPLANATION LLM, the prompt describes the overview of the scenarios along with the previous action pairs and instructs the EXPLANATION LLM to help robots to take action step by step. In response, the EXPLANATION LLM generates potential trajectories and implies the action selection. For the PROXY LLM, the prompt includes statement summarizing both the input and output from the EXPLANATION LLM,, followed by the phrase: "Therefore, the next action is
24

---

Published as a conference paper at ICLR 2026
possible action from A'. Then, calculate the mean token logits for each possible action in A, which are then used to determine the action with the highest likelihood.
An example that the prompts used for the EXPLANATION LLM and the PROXY LLM on the MMLU datasets
User for EXPLANATION LLM:
Q: Below a multiple-choice question about professional law. Please explain your reasoning without explicitly revealing your selection. The question is: Which of the following is not within the power of federal subpoena command person place? A: Submit to physical or mental examination.; B: Attend and testify as to persons and/or events related to a cause of action.; C: Produce designated documents, electronically stored information, or tangible things in that person's 's possession, custody, or control.; D: Permit the inspection of premises.
at a specified time and
EXPLANATION LLM:
R: Federal subpoenas are designed to compel testimony, the production of documents, and permit inspections related to case. However, requiring individual to undergo physical or mental examination typically falls outside the scope of what a subpoena can mandate. Such examinations usually require separate legal processes warrants, ensuring individuals' rights are adequately protected.
User for PROXY LLM:
Q R Therefore, the correct answer is {a possible answer from A}.
PROXY LLM: The token logits of the given possible answer.
An example that the prompts used for the EXPLANATION LLM and the PROXY LLM on the MathQA datasets
User for EXPLANATION LLM:
Q: Below is multiple-choice question about math problem. Please explain your reasoning without explicitly revealing your selection. The question is: an auction house charges commission of 18 % on the first $50,000 of the sale price of an item, plus 10 % on the amount of the sale price in excess of $50.000. what was the price of painting for which the house charged total commission of $24,000? $115,000, $160,000, S 200,000, d 240,000, $365,000
EXPLANATION LLM:
R: Say the price of the house was $ x, then 0.18 * 50,000 + 0.1 * *(x 50,000) = 24,000 $ 200,000 18 % of 50,000 plus 10 % of the amount in excess of $ 50,000, which is 50,000, should equal to total commission of 24,000)
User for PROXY LLM:
Q R + Therefore, the correct answer {a possible answer from A).
PROXY LLM:
The token logits of the given possible answer.
There are the prompts used for the EXPLANATION LLM and the PROXY LLM on the MMLU and MathQA datasets. For the EXPLANATION LLM, the provides the question along with the possible answer options and requests the EXPLANATION LLM to explain the reasoning process without explicitly revealing the selection. In response, the EXPLANATION L.L.Mgnerrtts explanation that
25

---

Published as a conference paper at ICLR 2026
analyzes the problem and implicitly suggests answer. For the PROXY LLM, the user presents statement that includes both the imput to and the output from the EXPLANATION LLM, followed by the phrase: 'Therefore, the correct answer is possible answer from A', to caleulate the mean token logits of the given possible answer. Then, we calculate the mean token logits for each possible answer in A, which then used determine the answer with the highest likelihood.
F ADDITIONAL EXPERIMENTAL SETUP
Benchmark.
(1) SMAC. We generate explanations for an ally agent based on action-state history. Our dataset consists of 2K trajectories (1.5K for training and 0.5K for evaluation), with each trajectory containing states, actions, and rewards for 30 time steps. We feed information of previous consecutive time steps to the EXPLANATION LLM to generate explanations.
Specifically, we first train an RL policy using MAPPO (Yu et al. 2022 on SMAC and then collect the action-state pairs each timestep during the evaluation phase. Each action-state pair contains information about the current agent's position, health, and weapon cooldown, as well as the corre- sponding details for visible allies and enemies. Additionally, the action selected by the current agent is included. All action-state pairs are stored in JSON format:
State {visible allies {relative (I, u); wwwpon__cooleeeeeeeeeeee
health}
{idO [0.0765, - -0.0765) ; 0.0; 1.0); (id1 [0.0765, 0.0] 0.0; 1.0):
{id2 (0.0765, 0.076); 1.0); (id3 [0.153, 0.0); 0.0;; 1.0};
None visible enemies; own_health 1.0 } Action SOUTH
We feed the EXPLANATION LLM with consecutive action-state pairs, masking the action in the final pair, and then instruct the EXPLANATION LLMto analyze the provided information and infer the missing action.
(2) AI2-THOR. We evaluate our method on AI2-THOR, an embodied AI benchmark widely used in RL research, thereby testing the proposed method fully sequential decision-making environment.
Specifically, AHOR includes household activity scenarios that require multi-step planning, multi- robot coordination, and interaction with dynamic environments. We use the Complex Tasks split from the SMART-LLM dataset, in which each task involves 2-4 heterogeneous robots with complementary skills. These tasks are challenging because they cannot be decomposed into independent sub- problems solvable by single robot; agents must reason over past actions and observations, perform long-horizon planning, and coordinate multiple robots strategically to leverage their combined capabilities. This setting demands strong reasoning and sequencial decision-making, making it suitable benchmark for evaluating our method's s ability to recover a complex underlying policy. The original Complex Tasks split contains only 8 tasks. To broaden task diversity, expand the set to 59 tasks by prompting GPT-4o to generate additional tasks that follow the same structural requirements.
The prompt is formatted
AI2THOR is a embodied-Al benchmark xxx. The robots should be xxx.
The tasks should be xxx.
Here are some examples of embodied-Ar task. Example 1: xXx.
Example n: xxx.
Please help me generate new tasks that satisfy the above requirement. After generating these new tasks, we ask both human and GPT-4o to verify the correctness of these tasks.
We then apply RL to obtain a near-optimal policy over these 59 tasks and record its successful trajectories of 70% tasks for training our method and baselines. During evaluation, measure the average success rate our method and baselines on the remaining 30% unseen tasks to assess their generalization capability.
26

---

Published as a conference paper at ICLR 2026
(3) MMLU. We choose 4 challenging Professional Knowledge subsets (Professional214 Medicine (272 samples), Professional Law (1.53K samples), Professional Accounting (282 samples),215 and Professional Psychology (612 samples)). For each dataset, 70% of the samples selected randomly for training, and the remaining 30% for evaluation.
Training Details. For training dataset construction, established instruction-tuuiing datasets provide (general-purpose) preference data for answers, but we optimize explanations for answers. As a result, existing instruction-tuning datasets can't perform very well in our tasks. Constructing of human-annotated dataset is labor-intensive, so we leverage OpenAl ol-mini to generate explanations for each sample, providing a more efficient alternative. To generate positive samples, ol ol-mini is prompted with a question and its correct answer. The prompt used for positive sample generation is as follows: 'Below a multiple-choice question about xxx. Please explain your reasoning without explicidy revealing the correct answer: xxx. The question is: xxx. The options are xxx... Similarly, to generate negative samples, ol-mini receives a question with randomly selected incorrect answer in the prompt: 'Below is multiple-choice question about xxx. Please explain your reasoning without explicitly revealing a wrong answer: xxx. The question xxx. The options are xxx.'.
To further assess the quality of the generated samples, we enlist 12 participants (2 females, 10 males, aged 20-40) to evaluate whether the samples are reasonable and human-like. We randomly select 5% randomly chosen samples from each datasets and ask panticipants to rate them scale of to 4: 1 = unreadable, = readable but unreasonable, 3 = reasonable but unnatural, and 4 = natural and human-like. The example of sample is listed below.
An example that the evaluation of the quality of our constructed dataset MathQA
Below is multiple-choice question about professional law. Please explain your reasoning without explicitly revealing your selection. The question is: At a defendant's s trial for drug dealing, prosecution witness testified that he had heard the defendant telling group of people to come to the defendant's 's house because he had "a great crop of apples for sale." " The prosecutor then called and qualified drug enforcement agent as expert in how drug dealing is conducted. The prosecutor now seeks to have the expert testify that in her opinion, based on years of experience with drug dealers, the defendant's statement about the apples was code for his having drugs for sale. Should the expert's 's opinion be admitted?
Decision set =
(a: No, because an expert must base his or her opinion on scientific or technical data, not merely on experience.;
b: No, because the opinion based on the criminal actions of others engaged in drug dealing, and thus is substantially more unfairly prejudicial than probative.;
e: Yes, because the opinion is based on her specialized knowledge, and it will assist the trier of fact in understanding the evidence and determining facts in issue in the case.; d: Yes, provided that the expert first testifies about the specific information upon which she based her view that the defendant was using "apples" as code word to refer to drugs.]
The expert's testimony is likely admissible because her specialized knowledge can clarify the defendant's 's statements in the context of drug dealing. Courts generally allow experts to provide opinions that assist the jury in understanding evidence or determining facts. As long as her opinion is based on her experience and doesn't 't rely solely on hearsay, it should help the trier of fact make informed decisions.
Explanation:
Score: (1-4, I=unreadable, 2=readable but unreasonable, 3=reasonable but unnatural, 4= natural and human-like).
We train the EXPLANATION LLM and the rectified flow model iteratively. In the first step, EXPLA- NATION LLM generates explanation for each sample 3 times in the training set. The PROXY LLM then classifies these explanations as positive or negative. We use the positive samples to train the rectified flow model. Once trained, the rectified flow model is employed in subsequent the EXPLANATION LLM training. This entire process constitutes one round. As shown in Fig. 2(d) the EXPLANATION LLM's accuracy plateaus during the third round because the updates the reward
27

---

Published as a conference paper at ICLR 2026
model no longer yield significant performance improvements. By comparison, in the first two rounds, the accuracy initially increases rapidly due to substantial updates to the reward model, followed by a more gradual improvement. Therefore, perform two rounds of training. Note that the training process can be adapted for online learning, where the EXPLANAATIO LLM and the rectified flow model are trained simultaneously.
In each round, the EXPLANATION LLMand the flow model are trained for 10 and 100 epochs, respectively. The learning rate for the EXPLANATION LLMis initialized 2e-5 and then decays linearly to zero, whereas the learning rate for the flow model is fixed at 2e-4. EXPLANATION LLM uses the Adam optimizer, 6F16 and LoRA with hyperparameters Flora = 16 and Qiora 16, while DeepSpeed ZeRO-3 is employed to accelerate training. The rectified flow model uses the Adam optimizer. The batch size is 12 for the EXPLANATION LLM training and 128 for the flow model training. We run our method on 4 parallel 80GB Nvidia A100 GPUs. For EXPLANATION LLM training, each GPU processes at most sample at a time, with gradient accumulation over steps. For the flow model training, each GPU processes at most 32 samples takes 0.5 to hour, while our method requires approximately 14-30 hours on different datasets.
time. SFT typically
Evaluation. For evaluation, we use temperature of generated tokens 350 for SMAC, and 200 for MMLU and MathQA.
0.7 across all settings. The maximum number
For human evaluation, recruit 45 participants(30 males, 15 females, aged 20-40) from diverse fields (engineering, chemistry, machine learning, and economics). Each participant completes questionnaire containing 3 randomly selected benchmark samples. For test samples outside an annotator's expertise, we suggest that annotators consult authoritative sources for unfamiliar terminology background context to better judge whether the explanations are factually grounded. To ensure consistency across participants, we provided clear instructions and illustrative examples prior the evaluation. For each task. participants were presented with multiple explanations generated by different methods and asked to make a decision based on each one independently. Immediately afterward, participants selected the explanation they perceived as least cognitively demanding, most logically sound, and most actionable, with the order of explanations randomized for each participant to ensure fairness and minimize bias. This procedure enabled us to analyze trends with statistical confidence.
We define cognitive load as the perceived mental effort required to complete task. Importantly, actionability and ACC capture distinct concepts. Actionability reflects how easily third party can select an answer based on an explanation, emphasizing its clarity and usability. By contrast, ACC denotes the accuracy with which third party can predict the underlying decision based on the explanation. Notably, an explanation may be highly clear (high actionability) yet still lead to an incorrect prediction (low ACC). The score for each metrie is computed as the proportion of participants selecting a given method as the best. The example of a sample is listed below. We make sure that all the information about the participants is invisible, and there no potential risk their privacy.
28

---

Published as a conference paper at ICLR 2026
An example that the evaluation of the effectiveness of our method on MathQA. The explana- tions generated by DPO; ReFT and our method are named as A1; A2; A3 and A4 in a random order.
Description Optimal MathQA Problem- Problem-Solving Strategies: Choose the Best Explana- tion: Below a question along with three explanations (A1, A2, A3). Please review the question and the three explanations carefully, then answer the following questions accordingly
Q A shopkeeper sold article offering discount of 5% and earned profit of 31.1%. what would have been the percentage of profit earned if no discount had been offered? A: 38, B: 27.675, C: 30, D: Data Inadequate, E: None of These.
A1: Let's assume that the price of the article 'C'. When the shopkeeper offered discount of 5%, the selling price the article became 95% of the original selling price, Let's call the original selling price 'S', Therefore, the new selling price is 0.95S. We know that the shopkeeper carned profit of 31.1% %. Therefore, we can write the equation: Profit percentage CJ x 100. Now, let's assume that the
(Profit Cost price) x 100. 31.1 [(0.95S shopkeeper did not offer any discount. In that case, the selling price would have been the original selling price 'S' Let's call the new profit percentage 'P', Therefore, we can write the equation: P P= [SS C) / C] 100. We need to find the value of "P" in terms of "C" and 'S' From the first equation, we can solve for :C" C= (0.95S 0.311S) / 0.311. 2.97SS. Substituting this value of CC' in the second equation, get: P [(S 2.9788) (.978SJ 100 P 33.1%.
After reading Explanation A1, which answer do you think is leading to? A: 38; B: 27.675; C: 30; D: Data Inadequate; E: None of These. F: Not Sure
A2:
After reading Explanation A2, which answer do you think is leading to? A: 38: 27.675; C: 30; D: Data Inadequate; E: None of These. F: Not Sure
A3: xxx
After reading Explanation A3, which answer do you think it is leading to? A: 38; B: 27.675; C: 30; D: Data Inadequate; E: None of These. F: Not Sure
A4: xXx
After reading Explanation A4, which answer do you think it is leading to? A: 38; B: 27.675; C: 30; D: Data Inadequate; E: None of These. F:Not Sure
Which explanation do you think the most logically sound? A1; A2; C: A3; D: A4
Which explanation induce the least cognitive load? A: A1; B: A2; A3; D: A4
Which explanation is the most actionable? (Knowing which answer to select after reading it.)
A: A1; B: A2; C: A3; D: A4
G BASELINES
SFT a widely used approach to fine-tune LLMs for specific downstream tasks by training them on labeled datasets. In this process, pre-trained LLM is exposed to task-specific data containing input-output pairs. The goal is to minimize the difference between the model's 's predictions and the ground truth, typically using supervised learning objective like cross-entropy loss. We use training dataset consisting of preferred examples generated by ol-mini. These examples are effective explanations. We SFT the initialization stage enable the EXPLANATION LLM adhere
29

---

Published as a conference paper at ICLR 2026
to the target format and to set a foundation for subsequent refinement toward the target explanatory behavior. We use BF16 and DeepSpeed ZeRO-2 to accelerate training. The batchsize is 12 and the optimizer is Adam. The training epochs of SFT is 3 and the learning rate is initialized as 2e-5 recommended by the TRI. library and then decays linearly to zero.
PPO is a reinforcement learning algorithm commonly used to fine-tune LLMs, enhancing their alignment with specific goals or human preferences. PPO fine-tunes a pre-trained model by optimizing its policy based on feedback signals. These signals typically come from reward model, which is trained on labeled data that reflects desirable outputs. PPO adjusts the model's parameters to maximize expected reward while constraining updates within a predefined range, avoiding large deviations that could destabilize training. We initialize the model with the same SFT procedure, leveraging the same ol-mini-distillle we further fine-tune this SFT model with reinforcement learning algorithm, PPO, which is supposed to reinforce explanations that better support inferring the policy's 's decision by assigning higher rewards. To achieve this, we train reward model on paired preference dataset (queries with corresponding positive/negative explanations). We then fine-tune the SFT model with PPO using this reward model. We use BF16 and LoRA with hyperparameters Flora 16 and Giora 16, while DeepSpeed ZeRO-3 is employed to accelerate training. The batchsize is 12 and the optimizer is Adam. The training epochs of the reward model and PPO both 10 and the learning rate is initialized as 3e-6 recommended by the TRL. library and then decays linearly to zero. The main difference between PPO and our method lies in the reward modeling. Our method trains a probabilistic reward model using flow matching, while PPO uses a standard deterministic reward model trained with the Bradley-Terry model.
preferred examples. To generate more effective explanations.
DPO is a technique for aligning LLMs with human preferences by directly optimizing their outputs using labeled preference data. Unlike traditional reinforcement learning from human feedback, which relies on a reward model to evaluate responses, DPO simplifies the alignment process by directly using preference comparisons to guide optimization. In DPO, the labeled data consists of paired responses where one option is preferred over the other. The model learns to produce outputs that align with these preferences by optimizing a contrastive objective. For DPO, we initialize the model with the same SFT procedure, leveraging the same preferred examples distilled from ol-mini. To generate more effective explanations, we then fine-tune this SFT model using preference- based learning via DPO. Unlike PPO, DPO does not require separate reward model. Instead, we use paired preference data (each query paired with a preferred and a dispreferred explanation). freeze the SFT checkpoint as the reference model, and fine-tune the EXPLANATION LLM using the DPO objective. After training converges, we generate explanations using the same prompts as the other baselines. We use BF16 and LoRA with hyperparameters Flore. 16 and atore = 16, while DeepSpeed ZeRO-2 is employed to accelerate training. The batchsize is 12 and the optimizer is Adam. The training epochs of DPO 10 and the learning rate is initialized as 5e-6 recommended by the TRL. library and then decays linearly to zero.
KTO is an advanced approach for aligning LLMs with human preferences or specific task objectives. It draws inspiration from prospect theory, a behavioral economics framework that models how humans evaluate potential gains and losses under uncertainty. In this context, KTO optimizes the alignment process by weighting outputs based on their perceived utility, rather than treating all errors cqually. The core idea of KTO is to model alignment as an optimization problem where the goal is to maximize expected utility under prospect-theoretie framework. For KTO, initialize the model using the same SFT procedure, leveraging the same preferred examples distilled from ol o1-mini To further improve explanation quality, we fine-tune the SFT model using preference-bare based learning via KTO. Unlike PPO, KTO does not require a separate reward model. Instead, we construct a dataset of (query, explanation, label) triples, where the label indicates whether the explanation is preferred or dispreferred. We then optimize the EXPLANATION LLM using the KTO loss, with the SFT checkpoint kept as a frozen reference for regularization. After training, the KTO-tuned model prompted with the same context as the other basclines to generate explanations. We use BF16 and LoRA with hyperparameters Tlora = 16 and Oiera = 16, while DeepSpeed ZeRO-2 is employed to accelerate training. The batchsize is 12 and the optimizer is Adam. The training epochs of KTO is 10 and the learning rate is initialized as 5e recommended by the TRL library and then decays linearly to zero
Skywork. Beywork-Reward-Llane 1at.d----aaaaa-------00 1-B-v00.2 is reward model developed by Skywork Al as part of their efforts to improve alignment and instruction-follooiing abilities in large language models. Based on the LLaMA-3.1 8B architecture, this version-v(l is specifically trained to evaluate and
30

---

Published as a conference paper at ICLR 2026
rank responses generated by language models in a preference- learning setting. The model is fine-tuned using a dataset of paired responses labeled by humans or heuristic methods, where each pair contains a preferred and less preferred answer. This enables the model to predict scalar reward score for any given response, facilitating its use in fine-tuning chat models for improved helpfulness, relevance, and alignment with human expectations. It ranks among the top two 8B models on the RewardBench Leaderboard, and all top five 8B models adopt it as their backbone and utilize its preference dataset for training. For Skywork, we adopt Skywork- -err---aaa--------------- base model. This checkpoint has already been optimized for reward modeling. We further fine-tune it using SFT on the same preferred explanation dataset to adapt to the explanation generation task. Then, we apply PPO using this trained reward model to further align the model's outputs with human preferences. We using BF16 and LoRA with hyperparameters Fiora 16 and Ofora = 16. DeepSpeed ZeRO-3 is employed to accelerate training. The batchsize is 12 and the optimizer is Adam. The training epoch of SFT is 3. The training epoch of the reward model and PPO is both 10, and the learning rate is initialized as 3e- 6 recommended by the TRL library and then decays linearly to zero.
as the
is peat-perffrmmnce open-source language model developed by DeepSeek. This model is a distilled version of DeepSeek-RL. R1, built upon the Llama-3.1 8B-Instruct architecture. By distilling knowledge from the larger DeepSeek-R] I model, it provides state-of-the-ar performance with reduced computational requirements, making it more accessible for various applications. For DeepSeek, we use DeepSeek-RR -asesee---R---ssil--------aeeee apply SFT using the same preferred explanation dataset to adapt it to the explanation generation task. We use BF16 and LoRA with hyperparameters Flora = 16 and alora = 16. DeepSpeed ZeRO-3 is employed to accelerate training. The batchsize is 12 and the optimizer is Adam. The training epoch of the SFT model is and the learning rate is initialized as 2e-5 recommended by the TRL library and then decays linearly to zero.
Deepseek. DeepSeek- D---Dstill--Llamaa
8B as the base model and
CoT+SFT a fine-tuning method that enhances the reasoning capabilities of LLMs by combining SFT with the structured reasoning paradigm. CoTT' uses explicit programmatic representations, such as pscudo-code or structured logic, to model complex groblem-solving tasks, In this approach, SFT is performed using datasets annotated with both input-output pairs and detailed programmatic reasoning traces. These traces serve as templates for step-by-step reasoning and enable the model to break down complex problems, such as mathematical reasoning or logical inference, into manageable sub-tasks, The explicit program-like structure helps the model perform multi-step computations and enhances interpretability, making it especially useful for domains requiring precision and transparency. For CoT+SFT, we construct a CoT-augmented version of our preferred explanation dataset via o1-mini by replacing each explanation with a CoT variant, containing more explicit step-by-step reasoning. We then apply SFT using this modified dataset to the same base model as our method. We adopt the open source library from (Trung et al., 2024). where we use BF16, DeepSpeed ZeRO-2, a fixed learning rate of le-5, Adam optimizer and a batchsize of 12.
ReFT is a training approach designed enhance the reasoning capabilities of LLMs by combining supervised fine-tuning (SFT) with reinforcement learning. In ReFT, the initial training begins with SFT, where the model is fine-tuned using datasets annotated with reasoning traces, such as step-by- step explanations or logical chains of thought. Once the model achieves a baseline performance, reinforcement learning is applied to further refine its reasoning capabilities. For ReFT, we use the same COT- augmented dataset as in CoT+SFT and the ReFT algorithm to the same base model as our method. We adopt the open source library from (Trung et al. 2022), where we use BF16, DeepSpeed ZeRO-2, fixed learning rate of 3e-7, Adam optimizer and batchsize of 12.
o3-mini is a smaller, more cost- reasoning model in OpenAl's "o3" family. It's optimized for tasks in STEM domains especially math, science, and coding with the goal of higher accuracy, lower latency, and lower cost compared to larger models. We directly use it for evaluation without fine-tuning.
o3-mini is used exclusively for inference without fine-tuning, whereas CoT+SFT and ReFT rely on the CoT-enhanced dataset. All other baselines, as well as our method, are trained on the same dataset.
31

---

Published as a conference paper at ICLR 2026
H ADDITIONAL RESULTS
H.1 DIFFERENT CHOICES OF o IN 40
We evaluate the impact of varying a in on final performance across different datasets. As shown in Table 11 the results remain stable across a wide range of values (o² 0.5, 1. 5). Specifically, on SMAC, the performance slightly decreases from 0.772 at x² = 0.5 to 0.761 at x² 5, while MMLU exhibits a small improvement from 0.759 to 0.781 as o² increases. For MathQA, performance is highest at d² 0.5 (0.809) and decreases mildly as o² grows. These variations are minimal, underscoring that our method is not sensitive to the choice of a and thereby confirming the robustness predicted by our theoretical analysis.
Table 11: Performance of our method with different o in qo-
Datasets
o² 0.5
SMAC
0.772
0.764
0.761
MMLU
0.759
0.772
0.781
MathQA
0.809
0.804
0.783
H.2
WITHOUT SFT
To evaluate the influence of supervised fine-tuning on our method, we train our method and baselines without conducting SFT. Results are shown in Tab. [2] Our method does require SFT to achieve strong performance, possibly because SFT data is crucial for obtaining enough positive samples to train the rectified flow model. Removing SFT, the performance of our method and baselines drops particularly sharply on RL tasks like SMAC. RL tasks typically involve sequential decision-makee and even interactions among multiple agents, making the reasoning of correct actions particularly challenging. SFT offers valuable examples about how to reason in such complex scenarios. Notably, our method without SFT significantly outperforms baselines without SFT SMAC, demonstrating its effectiveness in cases characterized by higher reasoning demands.
Table 12: Performance of our method and baselines without supervised fine-tuning.
Dalaseis
Ours
PPO
DPO
KTO
ReFT
Skywork
SMAC
0.457
0.302
0.324
0.381
0.405
0.357
MMLU
0.605
0.586
0.590
0.621
0.602
0.608
MathQA
0.634
0.618
0.630
0.638
0.643
0.629
H.3 UNDER NOISY REWARD
To evaluate the robustness of our method under noisy reward signals, conduct additional experiment where the reward (0/1) of the final token is randomly flipped with probability 0.03. As shown in Tab. 13 under the noisy reward condition, the performance of our method decreases only slightly from 77.2% to 74.3% on unseen tasks in the MMLU benchmark. Notably, even with these noisy rewards, our method still outperforms PPO (72.3%). Skywork (73.7%), and DeepSeek (72.1%) )%) trained without noise. This demonstrates the robustness of our approach under imperfect reward feedback.
Table 13: Performance of our method under noisy reward.
SMAC
MMLU
MathQA
Durs (Noise)
0.743
32

---

Published as a conference paper at ICLR 2026
H.4 VISUALIZED EXAMPLES
SMAC. To showcase the effectiveness of our method, we visualize an example on SMAC that a negative explanation classified by PROXY LLM is corrected by the rectified flow model yp As shown above, this explanation infers the answer is related to 'Attack Enemy 0', however, the PROXY LLM fails to predict the correct action. Instead, the rectified flow model yp can predict the action correctly based on the explanation.
Besides, visualize another example that a negative explanation generated by EXPLANATION LLM is optimized to positive explanation after training for two rounds. As shown below, the explanation from the SFT model opts for the movement of EAST. The explanation of our method can infer the correct movement of NORTH with solid evidence.
MathQA. To showcase the effectiveness of our method, we visualize an example on MathQA that negative explanation classified by PROXY LLM is corrected by the rectified flow model As shown below, this explanation infers the answer related option d, however, the PROXY LLM fails to predict the correct answer. Instead, the rectified flow model y can predict the answer correctly based on the explanation.
Besides, we visualize another example MathQA that negative explanation generated by EXPLA- NATION LLM is optimized
positive explanation after training for two rounds. As shown below, the explanation from the SFT model is confusing and misleading. The explanation of our method can infer the correct answer with solid evidence.
An example that negative explanation classified by the PROXY LLM is corrected by the rectified flow model yp of on SMAC
Q = Here are 5 allies and enemies SMAC. The goal of each ally to attack all the enemies together with other allies. At every step, each ally choose action to take from the Action set. The trajectory each ally consists of a series of State, Action, Value pairs. Here are 4 consecutive State, Action, Value pairs where the last state-action pair misses the Action: xxx. Please help me concisely summarize and analyze this part of the trajectory for the missing Action prediction.
Action set A ['DEAD', 'STOP' 'NORHH' 'SOUTH', 'EAST', 'WEST', 'Attack Enemy 0' V, 'Attack Enemy 1', 'Attack Enemy 2', 'Attack Enemy 3', 'Attack Enemy 4', 'Attack Enemy 5']
Explanation: ...In the first state, Ally opts to move EAST, likely positioning itself xxx. Ally's next action, "Attack Enemy 3", shows shift offensive behavior, Ally's following action, "Attack Enemy O", further prioritizes offense, which might reflect high-risk engagement. Ally's last action solidifies this pattern, further engaging Enemy 0, likely due to the imminent engagement's hazards. Throughout these actions, The allies prioritize attacking specific foes based their availability and proximity.
Distribution p from the PROXY LLM: [0.0074, 0.0031, 0.0260, 0.0062, 0.0055, 0.0096, 0. 0.1069, 0.2148, 0.2812, 0089, 0. 1600 0. 0.1045] + 'Attack Enemy 2' x
Distribution P from the rectified flow model: [0.0084, 0.0040, 0.0084, 0.0061, 0.0064, 0.0099, 0.3254, 1100, 0. 1137, 0. 1771, 0. 0.156,, 0.1218]
- 'Attack Enemy 0' /
33

---

Published as a conference paper at ICLR 2026
example on SMAC that negative explanation generated by the PROXY LLM is optimized to positive explanation after training for two rounds
Q= Here are allies and 6 enemies in SMAC. The goal of each ally is to attack all the enemies together with other allies. At every step, each ally can choose one action to take from the Action set. The trajectory of each ally consists of series of State, Action, Value pairs. Here are 4 consecutive State, Action, Value pairs where the last state-action pair misses the Action: Please help me concisely summarize and analyze this part of the trajectory for the missing Action prediction.
Actual Action = NORTH
In the first state, ally 0,1,2, and 3 all move south, The following action, STOP, ... Subsequently, ally and move WEST, ... Throughout these steps, allies prioritize movement over direct engagement, likely because no enemies are in view. Given the current state, where allies are arranged with their backs touching and no Enemy is in sight, the next logical action is EAST. This directional movement likely aims to realign the formation back to its original or a more udvantageous state, ensuring that allies are positioned optimally for future engagements. This EAST movement contrasts with WEST, which may have been short-term repositioning to balance the formation (EAST, x), and NORTH and SOUTH, which were likely attempts regroup reposition relative other allies.
Explanation:
Improved Explanation (Ours): In the first state, the ally chooses move SOUTH, In the second state, the ally STOPs, Moving WEST in the third state likely aims to cover more ground or flank the enemies, ... In the latest state, with no visible enemies and all allies maintaining full health and no weapon cooldowns, the ally opts to move NORTH. This directional movement decision likely stems from a strategic need to reposition for better vantage point to avoid potential threats, given the NORTHward movement may offer more advantageous stance compared to other directions.v Choosing NORTH over EAST, for instance, may be due to factors such as better cover better line-of-sight, enhancing the ally's ' readiness to engage when enemies become visible.
An example that negative explanation classified by the PROXY LLM is corrected by the rectified flow model y of on MathQA
Q = the cash realised on selling 14% stock rs. 101.25, brokerage being 1/4% is.
Decision set A = [a: 123, b: 106, c: 100, d: 101 v, e: 240]
Explanation: cash realised = rs. (101.25 0.25 >< rs
Distribution from the PROXY LLM: 1035, 0.0481, 0.6289, 0.2061, 0.0122] + 'c' x
Distribution from the rectified flow model:: [0.2363, 0.0477, 0.0114, 0.6683, 0.0567] +
'd'
34

---

Published as a conference paper at ICLR 2026
example on MathQA that negative explanation generated by the PROXY LLM is opti- mized to positive explanation after training for two rounds
Q A pet store regularly sells pet food at discount of 10 percent to 30 percent from the manufacturer's suggested retail price. if during sale, the store discounts an additional 20 percent from the discount price, what would be the lowest possible price of a container of pet food that had a manufacturer's suggested retail price of S 40.00?
Decision set A = [a: 10.00, b: 11.20, c: $ 22.40 V. d: S 16.00, e: $ 18.00.]
Explanation: retail price = manufacturer suggested retail price = $ 40 first discount price retail price -1 to -30 presence of retail price retrievable.x
Improved Explanation (Ours): retail price =40 first maximum usually discounted price =40 30 % of 40 =40- 12 =28 price after additional discounting during sale =28 20 % of 28 =28 -2.60
I
LIMITATION
While the flow model is theoretically grounded and performs well in capturing reward distributions aligned with human preferences, it may still face generalization challenges edge cases-aa such ambiguous preference data, rare unseen scenarios, and sensitive topics- to other LLM-based reward models.
Besides, our method assumes that decisions context-dependent. For random actions, which lack consistent rationale, the model cannot generate faithful explanations.
Moreover, learning from human feedback and ethical reasoning two examples of such tasks that involve multiple plausible choices. Human feedback and ethical judgments may vary with different individuals, and the viewpoints suitable for such learning tasks. By contrast, our method may be less effective in domains such as abstract knowledge, where producing logical or factual rules is required. In such cases, constructing meaningful rewards from PROXY LLM becomes more challenging.
population together form complex distribution. Our method is
Although our design choice foregoes direct analysis of internal mechanisms, we think that this pragmatic trade-off is warranted when developing methods intended to generalize across model classes with fundamentally different or opaque internals. affords two advantages that are central to goals: (i) it enables unified treatment of diverse archiecctuee--icluuinn RL policies and LLM agents-whoo whote internal structures may be opaque, inaccessible, or highly heterogeneous; and (ii) guarantees behavioral (functional) faithfulness, which is often primary concern in safety-critical or deployment settings.
J IMPACT STATEMENTS
This paper presents work whose goal is to advance the field machine learning by developing a model-agnostic explanation generator for intelligent agents, enhancing transparency and interpretabil- ity in agent decision prediction. The ability to generate effective and interpretable explanations has the potential foster trust in AI systems, improving effectiveness in high-stakes applications such as healthcare, finance, and autonomous systems. Overall, we believe our work contributes positively to the broader AI ecosystem by promoting more explainable and trustworthy AI.
35


Reviews:
[{"review_id": "ZaBHoQt5DT", "reviewer": "Reviewer_1rW4", "summary": "This paper proposes a framework for producing natural language explanations for an agent policy. This framework trains a dedicated LLM to generate explanations by using a distributional reward model based on continuous normalizing flows and an RLAIF training pipeline. The method is evaluated over three benchmarks, showing improved action prediction performance compared to baselines.", "questions": "1. It is not clear to me how the baselines are organized. Is SFT training the ExplanationLLM still? What about RLHF? Does DeepSeek and o3-mini mean they are used in place of the ExplanationLLM?\n2. In the case of MMLU example in Table 5, this is a very technical explanation. How would either the LLM or human annotators actually know whether the explanation is correct or not without having domain expertise? How do you know the explanation in Table 5 is correct at all, for example?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* I like the formulation of the rectified flow model for modeling rewards. In general, the setup does make sense to me for training a model to generate explanations of other policies, and the pluralistic/probabilistic view of explanations is reasonable.\n* Outside of the choice of the benchmarks (see weakness below), I appreciate the thoroughness of the empirical results. The ablations over different backbones and rewards (Table 7-10) help better understand the method’s performance and the generalization figure (Fig. 1) helps support the idea that the reward model is able to separate true/false samples.", "weaknesses": "* I think the framing is a bit confusing. The stated goal is to explain a policy’s decision, where the policy is that of an RL agent. But the only actual “policy” that is explained is for the SMAC benchmark, with MMLU and MathQA being language-based multiple choice questions where the agent’s “policy” is simply selecting a single action out of a set. Thus, for these benchmarks there is no sequential decision making aspect (unless we are referring to sequential prediction of tokens, which doesn’t seem to be the case), and the “context” is simply the question which is independent of the policy (this could be considered the observation to an agent policy). Thus, I find the whole framing around recovering the policy and explanation for some RL agent to be needlessly confusing, as 2/3 of the benchmarks seem more like a commonsense reasoning task to me as they are agnostic to any policy.\n* The value of the explanation is measured purely in the action prediction performance. Thus the validity or faithfulness of the explanation is not measured at all. Which is also related to the above weakness: what does it mean to have a “faithful” explanation for a multiple choice question with no information about the model which is generating the answer itself (only access to the question)? This seems like an ill-posed problem to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L3Ba7XIfTc", "reviewer": "Reviewer_BsL4", "summary": "The paper studies how to train LLMs to produce natural language explanations for the underlying decision. The authors use a distributional reward model based on a rectified flow that is conditioned on the context and on the explanation. This flow model is trained to model noisy proxy rewards  from proxy LLMs and the authors argue that if the base noise matches the proxy noise, the learned reward distribution stays close to the unseen human reward distribution. The explanation model is then optimized with PPO using sentence level rewards. Experiments on SMAC and on multiple LLM benchmarks show improvements over RLHF and RLAIF baselines.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The use of a language conditioned rectified flow as a reward model is novel.\n- The empirical section is broad, with many benchmarks and baselines.\n- Human evaluation is convincing.", "weaknesses": "1. The positioning of the paper is hard to grasp. Stripped of its technical machinery, the method is essentially training a model that, given only the current context or prompt, produces a reasoning trace without ever emitting the final decision/action/answer. It is therefore unclear what the concrete utility of such an “Explanation LLM” is. The title “translate policy to language” confuse the reviewer that the goal is to explain an existing policy’s decision process, but the actual pipeline trains a model from offline data to generate reasoning. This makes it difficult to see what problem for the community the work is solving. Existing reasoning LLMs already produce step-by-step rationales and, importantly, also output the final answer; the proposed model looks like a strict subset of that capability. Without a clear downstream role, the contribution feels under-motivated.\n\n2. This ambiguity in purpose is reinforced by the evaluation. The paper trains a model whose defining feature is that it only generates explanations, yet the core metric is whether another LLM (GPT-4o) can recover the correct action from that explanation. This shifts the assessment away from explanation quality and back to task accuracy. There are no direct measures tailored to the produced explanations; instead, the paper relies on inference accuracy as a proxy. If the work is to be read as an interpretability paper, this is a substantial gap: explanation-centric models should be judged on properties of the explanations themselves. As a result, the current evaluation protocol does not convincingly demonstrate that the method has advanced explanation quality.\n\n3. A further limitation is that the method is demonstrated only in settings with small, discrete action spaces such as multiple-choice QA and SMAC-style actions. This leaves out a very large portion of LLM use cases, where the output space is open-ended. In such cases, training a standard reasoning LLM that emits both rationale and answer is simpler and more broadly applicable. Because the paper does not show how to extend the method to richer action spaces, the overall practicality and generality of the proposed architecture is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lnsgqc4SwA", "reviewer": "Reviewer_ZwDH", "summary": "This paper proposes a general-purpose framework for training an LLM-based explainer using reinforcement learning. The core idea is to map random noise to a complex reward distribution by training a continuous normalizing flow (CNF) model to approximate reward samples generated by a separate proxy LLM. The authors further support their method with both theoretical guarantees and empirical validation on real-world datasets.", "questions": "Suggestions and Questions:\n1 I suggest that the authors analyze the robustness of the proposed method under noisy or imperfect reward signals.\n2 Since the framework is claimed to be general-purpose, could the authors demonstrate its applicability in a standard RL environment (e.g., Mujoco)? This would further support the generality of the method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1 The topic is interesting, and to my knowledge, this is the first work that applies diffusion model to fit the reward for LLM explanations.\n2 The experiments are abundant.\n3 The paper provides a thorough analysis of the proposed algorithm, including an error bound for recovering the true human reward distribution (Theorem 1), which reinforces the soundness of the approach.", "weaknesses": "Weakness:\n1 In Section 3.1, the authors approximate the likelihood of producing multiple actions by averaging token logits. This approximation may introduce significant estimation error, especially when a single abnormal token contributes disproportionately.\n2 The framework does not address the issue of long-term reward accumulation, which is a key consideration in RL-based settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DJdeBro1Eb", "reviewer": "Reviewer_T4VY", "summary": "This paper proposes a framework that translates reinforcement learning policies into natural language descriptions and then uses these textual representations as intermediate supervision for imitation learning. By bridging policy representations with LLM reasoning, the approach aims to improve interpretability, sample efficiency, and generalization. Experiments on MiniGrid, BabyAI, and Crafter show consistent improvements over imitation and language-conditioned baselines.", "questions": "1) How sensitive is the approach to translation quality—does noisy or partial text reduce imitation performance?\n\n2) Could the same idea extend to multi-step reasoning or continuous control tasks?\n\n3) Have the authors considered analyzing the semantic consistency between translated language and policy behavior?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) Conceptually interesting: The idea of using language as an intermediate representation is intuitive and well-motivated, offering a new perspective for explainable and interpretable RL.\n\n2) Clear methodology: The framework is well structured, with each component (policy translator, language learner, and imitation stage) clearly explained.\n\n3) Empirical consistency: Results are stable across multiple benchmarks, with useful ablation studies supporting the core claims.\n\n4) Readable and coherent writing: The paper is well organized and easy to follow.", "weaknesses": "1) Limited experimental scope: All tasks are symbolic or low-dimensional toy environments. It is unclear whether the method scales to more complex RL domains (e.g., continuous control, high-dimensional observations). This substantially limits the claim of generality.\n\n2) Dependence on pretrained LLMs: The policy-to-language translation relies on large models like GPT-4; the paper does not analyze whether smaller models could achieve similar results.\n\n3) Lack of deeper analysis: The paper does not provide theoretical or qualitative explanations for why language intermediates improve policy learning beyond empirical gains. It could be an add if provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HNg7SYK2NM", "reviewer": "Reviewer_e3dx", "summary": "LLMs have been used to explain other AI agents’ behaviors, but the authors argue that these ‘Explanation LLMs’ can be biased by AI agents’ decisions. To address this issue, they propose using continuous normalizing flows (CNF) models to approximate/generate reward distributions and use them to train Explanation LLMs. As explanation LLMs are trained with rewards predicted by CNF models, training is independent from AI agents’ decisions.", "questions": "In Line 203, they say, “The context c_{j} and explanations y^{e}_{j} are encoded by an LLM.”  What does LLM refer to? It looks like they are referring to Explanation LLM, but this needs to be clarified.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "In this study, Proxy LLMs are used to estimate the likelihood of humans inferring AI agent’s actions from Explanation LLMs’ outputs (i.e., explanations). Proxy LLMs accept contexts and explanations as input prompts and predict AI agent’s actions. To incorporate language cues into the flow model, the authors build a new rectified flow model by using Proxy LLMs’ modules. After training, this rectified flow model can model human evaluation scores. Such proposals are novel, and their empirical evaluation suggests that CNF allows Explanation LLMs to generate high quality explanations on agents’ decisions on complex tasks. \n\nGiven the novelty and strong empirical evidence, I believe this study would bring about strong interest from the readers.", "weaknesses": "I would like to point out that the main text was a bit difficult to follow at first, as some of essential information (e.g., Figure 2) were found in Appendix. I recommend including the information (relevant details of the rectified flow model and Figure 2) in the main text.  \n\nIn lines 062-064, the authors state: “Intuitively, the PROXY LLM acts as a forward process that injects noise into human rewards, and the CNF learns its inverse, partially denoising and recovering the underlying human rewards.” This sentence would make sense only in the context of diffusion process, and therefore I recommend providing more context (a figure or diagram) for the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)