BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning
Authors: Han Zhong
Yutong Yin
Shenao Zhang
Xiaojun Xu
Yuanxin Liu
Yifei Zuo
Zhihan Liu
Boyi Liu
Sirui Zheng
Hongyi Guo
Liwei Wang
Mingyi Hong
Zhaoran Wang
Venue: ICML 2025 poster (2025)
Keywords: reasoning
boostrapping
RL
rational generation
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Our framework addresses two critical questions: (1) how to generate high-quality reasoning processes during inference automatically, and (2) how to integrate these processes into post-training. We propose the \emph{Bootstrapping Reinforced Thinking Process} (BRiTE) algorithm and demonstrate its theoretical convergence at a rate of $1/T$, where $T$ is the number of iterations. The algorithm operates in two steps. First, it generates high-quality rationales by approximating the desired posterior distribution using a reinforcement learning approach with a novel reward shaping mechanism. Second, it fine-tunes the base LLM by maximizing the joint probability of rationale generation with respect to LLM parameters. Empirical evaluation on GSM8K and MATH benchmarks demonstrates that our approach consistently improves performance across different model sizes without requiring human-annotated thinking processes, outperforming standard chain-of-thought prompting while enhancing existing post-training methods.

BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language
Model Reasoning
Han Zhong" 1
Yutong Yin 2 Shenao Zhang *2 Xiaojun Xu
Yuanxin Liu *2 Yifei Zuo " Zhihan Liu
Boyi Liu Sirui Zheng²
Hongyi Guo Liwei Wang'
Mingyi Hong*
Zhaoran Wang" *
Abstract
1. Introduction
Large Language Models (LLMs;; OpenAI, 2023; Anthropic, 2023; Team et al., 2024a), have emerged as a breakthrough in artificial intelligence, demonstrating unprecedented capa- bilities in natural language processing and generation. The training pipeline of these state-of-the the-arr models consists of two critical phases: pre-training and post-training. During the pre-training phase, LLMs learn from vast datasets to predict subsequent tokens in sequences, enabling them to learn extensive linguistic patterns, contextual understand- ing, and general world knowledge. The post-training phase further refines these models through two stages: Supervised Fine-tuning (SFT) and Reinforcement Learning from Hu- man Feedback (RIHF; Christiano et al., 2017; Ziegler et al., 2019; Ouyang al., 2022). Recent research (OpenAI, 2024) has shown that by scaling the inference time, these models demonstrate sophisticated reasoning capabilities, particu- larly in domains such as mathematics and programming.
Large Language Models (LLMs) have demon- strated remarkable capabilittes in complex rea- soning tasks, yet generating reliable reasoning processes remains significant challenge. We present unified probabilistic framework that for- malizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithi,, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using novel re- ward shaping mechanism. Second, it enhances the base LLM by maximizing the joint proba- bility of rationale generation with respect the model's parameters. Theoretically, we demon- strate BRiTE's convergence T representing the number of iterations. Empiri- cal evaluations on math and coding benchmarks demonstrate that our approach consistently im- proves performance across different base models without requiring auman-annotated thinking pro- cesses. In addition, BRiTE demonstrates supe- rior performance compared existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through su- pervised fine-tuning with
Unlocking LLM reasoning abilities typically relies on struc- tured prompting methods that break down problems into step-by-step solutions, known as chain-of-thought (CoT) reasoning (Wei et al., 2022). While this approach has shown promise and inspired various extensions (Wang et al., 2022; Yao et al., 2024), the fundamental challenge of reasoning reliability remains unresolved. Generated rationales often lack logical completeness or validity, with their quality heav- ily dependent on task-specific prompting strategies. Recent developments in inference-time scaling techniques (Snell et al., 2024) have shown potential improvements. However, these approaches primarily address surface-level symptoms rather than the core challenge of generating high-quality reasoning processes. Furthermore, the field increasingly seeks automated improvements to reduce reliance on man- ual prompt engineering. This context motivates our design- ing mechanism for high-quality thinking process generation. Meanwhile, prior research (e.g., Zelikman et al., 2022; Yuan et al., 2023) indicates that reasoning processes, when prop- erly selected via verifiers, enhance LLM's reasoning capabilities during post-training. This leads to our research objective: developing a framework for the automated gen- eration of high-qualit, (correct) reasoning processes and
rate of 1/T with
data.
'These authors are the main contributors to this work; detailed contributions are provided in Appendix A. 'Center for Dada Sci- ence, Peking University "Northwestem University Bytedance Re- scarch ' UUniversity of Minnesota. Correspondence Han Zhong <hanzhong @ stu, stw.pku.edu.nn..
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
incorporating them into the post-training stage improve existing algorithms.
To this end, we propose a unified probabilistic framework that formalizes the reasoning process accompanying evalua- tion signals, followed by a generic algorithmic framework. Specifically, our work has three key contributions:
X
Y
We formulate the problem as probabilistic graphical model (Figure 1), characterizing the generation flow from prompt X to latent rationales Z to answer Y. along with their corresponding evaluation signal O. This explicit mathematical characterization serves two essential pur- poses: first, introducing Z breaks down the complex distri- bution P X) into more tractable marginal distributions P(Z X) and P(Y X, Z), which aligns with Chain-of- Thought (CoT) methods (Wei et al., 2022); second, intro- ducing O provides crucial rationale-answer quality signal, making the generation of desired (correct) rationales more achievable.
Z
Figure I. LLM as a probabilistic graphical model. X and Y rep- resent prompt and response, respectively. The latent variable Z indicates the intrinsic thinking process behind generation. Evalua- tion signal O is influenced by X, Z. and Y.
processes generated by our framework are of high quality, surpassing not only those produced through CoT prompting, but also outperforming when applied to fine-tuning. Our framework represents a significant advancement in improving LLM reasoning capac- ity through the creation of synthetic data that incorporates detailed thinking processes (CoT data).
thinking processes
Under this framework, our learning objective is to maxi- mize the probability of generating high-quality rationales and answers that yield optimal evaluation signals. To achieve this, we propose the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm consisting of two stages: first generating high-quality rationales by training an LLM whose output approximates the desired posterior of thought given the question-anae answer pair; and then fine- tuning the seed LLM by maximizing the joint probability of rationale generation with respect to the LLM's parame- ters. Theoretically, we prove our algorithm converges at a rate of 1/T, where 7 is the number of iterations. Regard- ing the practical implementation, to address the challeng- ing Bayesian inference problem in the former step, we develop a novel reward shaping mechanism that converts it into reinforcement learning optimization problem.
1.1. Related Works
We discuss mostly related works here. More discussion RLHF is deferred to Appendix B.
Reasoning in LLMs. Prior work has explored various prompting techniques to enhance language model reasoning capabilities. CoT prompting (Wei et al., 2022) has emerged as a particularly effective approach, encouraging models to break down complex problems into intermediate steps by demonstrating step-by-step reasoning paths. The following works, such as Zhou et al. (2022); Yao et al. (2024), design more prompting techniques to enhance the model's capacity. However, these methods typically rely on manually crafted (CoT) prompts to elicit reasoning processes, which are then used guide the model's generation process. While such ap- proaches have shown promising results in improving model performance across various reasoning tasks, they remain de- pendent human-designed prompting templates and may not fully capture the natural reasoning patterns that emerge during model inference. To this end, a line of works aims to boost the latent reasoning process quality or even achieve automatic reasoning process generation, where the latter one our focus but our method is based on reinforcement learning and thus is different from previous works. Previous methods can be roughly regarded as an EM-type algorithm, and detailed comparisons are presented below.
Our empirical evaluations of math and code generation benchmarks show consistent improvements across multi- ple LLM series, including Gemma, Llama, and Mistral. Our experimental results demonstrate BRiTE's ability enhance existing post-training algorithms (rejection sampling type methods and iterative DPO) through RL- based rationale generation with consistent improvements. Notably, BRiTE achieves a 10-point improvement on GSM8K benchmarks when applied to the Gemma-1. 7B-it base model. Notably, our algorithm matches or even exceeds the performance supervised fine-tuning meth- ods that use human-labeled thinking processes, despite data.
not requiring any
In summary, we present a provable and practical frame- work for automated thinking process generation that can be seamlessly integrated into the training stage. The thinking
EM-type Methods. Our algorithmic framework builds upon the Expectation- Cinitttion-Maimmzzttiinn (EM) algorithm (Dempster et al., 1977) and its variant, rejection sampling EM (Neal
2

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
& Hinton, 1998; Rush & Ritter, 2024). In standard EM, the E-step learns posterior over latent variables while the M-step maximizes the expected log-likelihood; rejection sampling EM approximates complex posteriors through re- jection sampling in the E-step before proceeding with the standard M-step. This motivates various rejection sampling fine-tuning methods employed in modern LLM training (e.g.. Cobbe et al.. 2021; Zelikman et al., 2022: Dong et al.,
MDP Formulation of LLMs. Following the notations in Zhong al. (2024), we consider MDP M (S, A. P, T. H). In this framework, S and A represent the state and action spaces, respectively. The transition ker- nel is denoted by P : S x A + (SS), while r indicates the reward function. The initial distribution is defined by P E ASS), and H specifies the horizon length. A policy π : S + A(A) maps a state a distribution over the action space. Initially, an initial state is sampled using 81 p. At the h-th step, the agent receives the state 8,, and chooses the action on क 8x). This interaction continues until a specified ending condition is met, which will occur within H steps.
2023a; Yuan al., 2023; Gulcehre et al., 2023; Singh et al.,
2023; Zelikman et al., 2024; Yuan et al., 2024; Chen et al., 2024). These algorithms filter outputs or/and latent reason- ing process through reward functions or verifiers (E-step), then perform fine-tuning on the selected samples (M-step). Our algorithm generalizes these approaches (Example 3.6) and introduces novel E-step based on reinforcement learn- ing (Section 3.4). While Hu et al. (2023); Hoffman et al. (2024) also conceptualize LLM generation as latent variable models and propose EM-type algorithms based on Markov chain Monte Carlo (MCMC) or generative flow networks, our work distinctively focuses on enhancing the reason- ing capabilities of LLMs through automated reasoning pro- cesses using reinforcement learning (e.g., PPO). Notably, in contrast to these prior works, we provide a more gen- eral and rigorous mathematical framework fro LLM reason and unified theoretical guarantees. Furthermore, our frame- work extends beyond supervised and rejection sampling fine-tuning to advance iterative direct preference learning (Xiong et al., 2024a), contributing to improved LLM rea- soning in the RLHF paradigm. Finally, two recent works by Liu et al. (2024a) and Wang et al. (2024) also apply RL tech- niques to enhance LLM reasoning. However, these works do not focus on improving the thinking process generation, making them not directly comparable to our approach.
In the context of generating large language models (LLMs), let si ~ P represent the prompt I p. At each step h, the state S), = ($1, 01:h-1) consists of the prompt z and all to- kens generated up to that point. The LLM acts as policy that maps 8, to distribution over the action a₁ ~ .T-- |sk), where the action signifies a token (or a series of consecutive tokens). The transition process is deterministic; it simply concatenates 8p. = (81, 01:A-1) and a₁ to create a new state Sh+11 ($1, (1 The generation process concludes with a special end-of-sentence token Eos, which will be generated within H steps. For simplicity, we consider the length- trajectories {(s), ) noting that this does not lose generality since we can pad the Eos token to the text to reach length H. With this notation and recognizing the autoregressive nature of LLMs, for any realizable trajec- tory {Sh, ) the generation probability is given by I [aI. (1) = I Taa | $1, Q1:h-)).
Regularized Value Functions. For policy π, its entropy- regularized value function is defined as
1.2. Notations
V (s; E.
Σ (r(s). - B. log T(an 1
For any space X, we denote ((X) as the set of distributions over X. For any positive integer h, we denote the sequence {ar,'.- ".a,) , by aish. We use 1 1 to denote the indicator function.
(2.1)
where 3 >>> 0 is regularization parameter. The regularized Q f-uuction Q" of a policy π is related to the regularized value function V as
(x, = r(s,u) + Egrpi ( (Va V (=((s))= = (s | - (a 8) + Q (s. a; a)r)
2. Preliminaries
In this section, we present the single-step bandit formula- tion and the multi-step Markov decision process (MDP) formulation for LLMs.
(2.2)
The regularized optimal policy x² is the policy that maxi- mizes the regularized value function defined in (2. and its corresponding optimal Q-function and value function are denoted as Q' and V*, respectively. By (2.2), can be shown that
Bandit Formulation of LLMs. A simple way to understand LLMs is through the bandit formulation. In this context, the prompt and the response are represented as e X and y E V, respectively. Here, X' refers to the set of prompts, while )' represents the set of responses. The LLM corresponds to the policy π in this bandit framework, where (t x)) indicates the probability of generating the response given the prompt x.
(*(r))) = log
Σ exp (Q" aAA
(2.3)
**(a| (xp((Q"(s,a;;)) V (*rr)))
x exp (Q'(s, a)r))
(2.4)
3

---

achieved when q(w) BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
3. Unified Framework and Generic Algorithm
rewrite it
In this section, we present a new framework for LLM rea- soning and our generic algorithm within this framework.
[(0) log P(2 E 2,
E ,, 0 x1
log
Σ
P(z,y,0|2,,0 xx,
(2,1)0) x
3.1. LLM as A Probabilistic Graphical Model
max W6v (. (0),
(3.3)
We consider four different spaces: X' represents the prompt space, Z denotes the latent space that captures the intrinsic thought process (CoT). y/ signifies the response space, and O stands for the evaluation signal space, which reflects the optimality of the prompt-latent-respon
where Q(+ xt v) can be regarded as another LM parametrized by &i and (0)
tuple. Furthermore, for any (z, 2, y: o) E X' x Z x y x O, we describe the generation process using the probabilistic graphical model illustrated in Figure 1. This indicates that
(0) Σ (z.V.0)
log v, 1,,00,00))))))00))))00))))))), x, Q22,,,,1
EXX
Σ (2.U.0)
log (( Q(2, U(,,,,,),,4)
c2'xX'x x
P(z 0 0) P(z, y x. 0) IP(o x, 2,
(3.1) = P( xr 0) P(y x 2, 0) IP(o| x. 2, v),
The last equality of (3.3) follows the following lemma: Lemma 3.1. For any set W and non-negative numbers
[Pw 2 O.wew. it holds
where f is the parameter of the LLM that guides the gen- eration process. First, a latent variable is generated from the distribution P(- xr 0). and then the response P(- 2, 0) is produced based on both the prompt x and the latent variable z. Importantly, the probability Po x, 2, 2) is independent of the L.LM parameterized by 0, as we assume there exists ground-truth judgment for the 2, y). such as a groundatruthhhuman reward func- tion. Unlike traditional LLM frameworks that only consider the prompt space X' and output space V, our framework incorporates both latent thinking process space and an observation space. These additional components are cru- cial for mathematically understanding how to improve the quality of thinking processes using evaluation signals.
log
Pu== max (EA(W)
Pw
Eu-g(t) llog Pa- log q(w)].
wEW
The maximum Pw// (Ewre Pwr).
is
Proof of Lemma 3.1. This lemma is equivalent to the non- negativity of the KL divergence. For a detailed proof, please refer to Appendix C.I.
triplet (z,
We have transformed the problem of maximizing L(0) into maximizing its lower bound Z.- This shares the same spirit with the Expeecation-Maxiiiaation (EM) algorithm (Dempster et al., 1977) and Variational Autoencoders (VAE) (Kingma, 2013). To solve this optimization problem, we iteratively update the parameters v and @ to maximize (3.3). Given vh, and O., the updating rules of v/++1 and 0.41 are specified
Under this probabilistic graphical modeling of LLMs, our learning objective is maximize
(()) = P( E 2, y E 00 o x),
(3.2)
where 2' 2, 9/ y, and 0 C O denote the subsets of spaces representing the latent thinking process, response, and evaluation signals, respectively. Section 3.2, we de- velop a general optimization algorithm that works with any choice of these spaces (2, " (0,0) O). Subsequently, in Sec- tion 3.3, show how this framework unifies existing learn- ing approaches by demonstrating how different choices of
choosing 4441 such that
Q(z o x, vr+1) = argmax (O.)
(3.4)
((,vl1,v))
P(2,N,o|z 0.)
P(22, y, o x. O.).
E PP 2, y, o xx 0.)
This is implied by the optimality condition in Lemma 3.1.
these spaces (2,, , V O) correspond to various established learning paradigms and algorithms.
choosing 0c+1 such that
oct1 = argmax Ce (0)
(3.5)
3.2. Bootstrapping Reinforced Thinking Process
argmax
logP 2,0(2,,,012,0) x, ol,,,,ola,vvu
We propose the algorithm, Bootstrapping Reinforeed Thinking Process (BRiTE), to maximize the objective (3.2) within the framework proposed in the previous subsection. Since this objective may be difficult to optimize directly, we
(#.8.0)
6 XXX
argmax
Σ
log 12 Q(2, (1,11))
(#.g.0) xxxx

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
where the second equality is implied by (3.1).
Theorem 3.3 establishes a convergence rate of 1/T, where T represents the number of iteration steps. In our analy- sis, we link our algorithm to classical mirror descent algo- rithms (Nemirovskij & Yudin, 1983; Bubeck et al., 2015), which have also been utilized in recent studies of policy optimization algorithms (Agarwal et al., 2021; Cai et al., 2020; Zhong & Zhang, 2024). Additionally, our results rely on a concave assumption. This assumption is crucial for proving global convergence results, and we can still obtain weaker guarantee for stationary points even without this concave condition; see Appendix D.2 for details.
Intuitively, during the vi-updating step in (3.4), the proposed algorithm learns to generate high-quality thinking processes by focusing on verified correct responses. Specifically, when we define O = {0*}. where O" represents the correct evaluation signal mathematical tasks, P(z, y; O x) represents the probability distribution of generating both high-quality thinking process and correct final answer y- Following this, in the following 9-updating step in (3.5), the algorithm fine-tunes the Large Language Model by maxi- mizing the joint distribution between the learned thinking process generator and the LLM's output with respect to pa- rameter 9. This fine-tuning approach encompasses several learning paradigms and algorithms, including SFT, RLHF, and rejection sampling-based algorithms, detailed in Ex- amples 3.4, 3.5, and 3.6. The two-stage BRiTE algorithm draws inspiration from the classical EM algorithm. While the probabilistic graphical model used in BRiTE differs from traditional latent variable models, the vu-updating and 0-updating steps correspond to the E-step and M-step in the EM algorithm, respectively.
Having established a theoretical guarantee for our proposed generic algorithm, we will demonstrate how our algorithm can incorporate many existing learning algorithms, high- lighting the broad applicability of our algorithmic frame- work and its theoretical guarantee.
3.3. Connections to Existing Learning Paradigms and Algorithms
Example 3.4 (Pre-training, SFT and Conditional SFT). We first set aside the latent space and observation space, mean- ing that Z = O 0. In this case, our learning objec- tive (3.2) encompasses two key processes: (I) pre-training, where we let x represent the prompt and y the next to- ken; and (II) fupervisedfime-tunne fine-tuning (SFT), where we define y = {x)) as the expert response corresponding to the prompt Moreover, for the prompt-
Before presenting the theoretical results for BRİTE, we make the following assumption about the generation proba- bility of parametrized LLMs.
Assumption 3.2. Assume that LLM is parameterized by 0, we denote that the logits of generating (2, y/) conditioned on by fo(2, z, y), then the probability of (2 ar, takes the form
pair (z, v) and singleton space y =yy, we consider Z = B and x = [R(x, u)} be the reward corresponding to the prompt- response pair, then our objective recovers the conditional SFT (Lu et al., 2022: Dong et al., 2023b; Yang et al., 2024). Example 3.5 (RLHF: PPO and DPO), We choose ạ = y and L = Z as the complete response space and latent space, respectively. Additionally, we set O O= = 13. where indicates optimality and indicates non-optimality, re- spectively. Since our goal is to maximize the probability of observing the signal of optimality, we focus on 0 {1}. We also assume that IP(o 1 F. 2, y) exp( R(x,, 2, /3)) for some reward function R and 3 > we have
P(Z,y/,, xr exp (Jo(z, 2, A(x, 0))
exp (Ja( v,,)),
where (z 0) is the normalization factor. Moreover, we assume that so E H for some reproducing kernel Hilbert space (RKHS)' associated with the kernel K. (X x 2 x )) x (X' x Z x )/) + R.
O. With these choices,
The fo(v,, z. v) in Assumption 3.2 represents the logits for predicting (v. based the prompt x; thereby capturing the current generation method of modern transformer-basee LLMs. With this assumption, we establish the convergence result of our algorithm in the following theorem, whose proof is deferred to Appendix D.I.
() =
Σ (x,))EX))
log v. 11 0) Q(z 012,8.116 112,,))
Σ
LogQ(e,y, Q(z, v, |2, v)
(x.))
8 Blog Q(2,1) v x 2b) P(z, y xx
-B(e.))-Q(--Im.)) [(x,,,))
Theorem 3.3. Suppose Assumption 3.2 holds. Given that L is concave and 0* argmaxg L(0), we have
where Q(2, y (2) = E.Eo Q(z, v,o x,
Q(2,y, ((2.8.11x, 11x,)) This expression recovers the proximal pol- icy optimization (PPO; Schulman el al., 2017) for RLHF (Christiano et al., 2017; Ouyang et al., 2022), where vb represents the LLM being optimized, and stands for the reference policy. Assuming that the preference data ((x, zt, +t,,,++ ,,,=,,,,,,+>' ==== ,, }} is drawn from the Bradley-Tere (BT)
KT. (P, 10, 0 0:) 1,,, 1x,00)
min (C(0-) - C(0,) < 1818T
T
We say H is RKHS on the set W with the reproducing kernel IC W W + IR if the inner product (:-)) satisfies f(w) = (K(w, ). for any (f, w) E H x W.
5

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
model (Bradley & Terry, 1952), with (=+ )t) denoting pre- ferred data and (z-,y") indicating dispreferred data, one can follow Rafailov et al. (2024) to derive the latent direct preference optimization (latent DPO) objective:
3.4. Practical Implementation: The Power of Reinforcement Learning
We have demonstrated that our generic algorithm encom- passes wide range of existing learning paradigms and algorithms, with provable convergence guarantee. In this section, we carefully examine the practicality of our algo- rithm.
LLatent-DDD -DO
(3.6)
=o -o(pI Q(=t, J. 2) - B Blgg P(zt,y yt x.,0) 0)
Q(= ,y x2,))
P( 3 x, 0)
where o is the sigmoid function. In contrast to the standard DPO objective in Rafailov et al. (2024), the objective in (3.6) additionally incorporates the latent variable z.
First, the d-updating step is relatively straightforward to implement, as this step simply maximizes the predicted probability of the next tokens after we sample (2, y, o) from Q(2, y, o xz 1)x+1). In contrast, the uuupdating step can be challenging in certain contexts. For instance, when y {v} represents the response corresponding to x. 6 = {1} indicates the optimality signal of interest, and 2 = Z, we have Q(z, y, x 0) x P(z,y: y: 1 xr 0) = P(= xr y, 1,0) the posterior of the latent variable z. Intuitively, obtaining this distribution requires us to identify the ideal latent (Co'T) based on the pair (x.y). which represents an intractable posterior.
Example 3.6 (Rejection Sampling EM Methods). Let ④ Y represeni the complete response space, and define O O= fo. 1,, where 1 indicates the optimal outcome and O indi- cates otherwise. We focus on the optimal signal, denoted as O- = =[1]. {1). Additionally, we denote x= Z as the complete thinking process space. The updating rule in (3.4) is given by Q22 y. 1 Ix 1)+1) x P(2,y. y. xr 0t) = P(z,g) y x, 0,)))) P((o = 1 x, 2, y). Consequently, the update in (3.5) can be expressed as
0c+1= = argmax 8
Σ (EN))zyY
logP(z,ulz, Q(2, 12, 6.)
To achieve this goal, we aim to use RL to train an LLM that characterizes the distribution Q(z, y, o x. 1b41)) or P(z, y, o x. 0z) in (3.4). Since an LLM acts as the pol- icy of an MDP, our approach involves two main steps: (i) constructing an MDP whose optimal policy matches the distribution Q that we need to learn; and (ii) applying RL algorithms to solve this MDP and identify the optimal policy to learn the desired distribution. These two steps convert the challenging sampling problem to a more amenable RL optimization problem. Notably, the main challenge we face is reward shaping, which involves designing appropriate re- ward functions to ensure that the optimal policy aligns with the intended LLM that accurately represents the posterior. Our approach to reward shaping is based on the follow- ing proposition, which characterizes the optimal policy for deterministic entropy regropy-regulrrized MDPs.
= argmax { (Besa) 12.00 log FogF(e,,/2 22 0) P(1 x. a. v)
(3.7)
1. If we simplify the scenario by omittiing the latent space
(i.e., Z = 0) and assume that P( 1 I, z.
1 (y is the correct answer), this leads to the STaR algo- rithm (Zelikman et al., 2022) or rejection sampling fine- tuning (Dong et al., 2023a; Yuan et al., 2023: 2024), which performs supervised fine-tuning after rejection sampling based on verifier results.
If we simplify the scenario by omitting the latent space (i.e., Z = 0) and assuming IP(o 1 1|,, x. y) (xp(R(zz)) for some true reward function R, the updating rule in (3.7) simplifies to
0c+1= = argmax { [E (2,0,) Pyy 0)
Proposition 3.7. Assuning the transition dynamic of regropy-re MDP is deterministic, then for any tra- jectories (s), H a satisfying ay = EOS, we have
exp
(3.8)
This recovers the ReSTEM algorithm presented in (Singh et al., 2023).
*"n ((s (()) 8n) x exp (1
1 Σ ai))
Finally, we note that Neal & Hinton (1998) first introduced the unified view of rejection sumpling EM, which Rush & Ritter (2024) later expanded upon.
The proof is deferred to Appendix C.2. By Proposition 3.7, if we select / = 1 and use the total reward as log P( y, o x, 6:), the resulting optimal policy recovers Q defined in (3.4). This choice of total reward function can naturally be assigned to each token as the rewand function, due to the autoregressive nature of LLM generation. Specif- ically, when (z, y, o) is represented by a token sequence a the total reward log P(2, y, o x. 6.) can be expressed as ΣS= log P(a, a1:5-1, I),, where log P(a,) | 11,)-1,z) serves as the reward for the j-th token in RL training.
By combining Theorem 3.3 with the examples in this sec- tion, we provide theoretical guarantees for these learning algorithms. As result, we establish theoretical foundations for two approaches: (1) PPO, connecting to the work of (Schulman et al.. 2017; Cai et al., 2020; Zhong & Zhang, 2024), and (2) Generalized FM in Example 3.6 (which includes vanilla RM (Singh et al., 2023) as a special case), presenting the first theoretical analysis of its kind.
6

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Finally, we remark that the application of RL text genera- tion has been explored in previous studies, such Guo et al. (2021). However, there appears to be no work applying RL to generate the thinking process to enhance reasoning capabilities within the context of L.MMs. Additionally, ex- isting studies do not address the issue of reward shaping, which is crucial problem in our scenario and is tackled by Proposition 3.7.
(=+, ",,,,, ,,,,7 based on the correctness of the y; and (3) apply DPO training (see Rafailov et al. (2024) or (3.6) on these preference tuples {(z, zt, yt, 27 ,2 ,).
Implementations of BRITE. To systematically demon- strate the effectiveness of our algorithm, we implemented BRiTE in three distinct configurations:
1. We implement the ui-updating step of BRiTE to obtain Q using (3.4), examining how the generated thinking process aligns with the desired reasoning path to en- hance LLM reasoning capabilities. Following our theo- retical framework in Section 3.4, we optimize two types of reward functions in the entropy-regularized MDP: (i) log(2, y, o x) = log P(z, y (x) æ), where y² repre- sents the correct answer for question x. This equality holds because we focus solely on correct questions, using proper choices of O = {answer verified to be correct] and L {yy). (ii) log(z, y, o x)) log P((z, y (2) + R(x, 2, 9)33, where we assume that P(o = 1 x, 2, y) exp( R(x, z, v)/B) with & = {1}. We employ PPO
4. Experiments
In this section, we systematically demonstrate that our uni- fied algorithm enhances the reasoning capability of LLMs.
4.1. Experimental Setups
Tasks and Datasets. To evaluate mathematical reason- ing capabilities, we conduct experiments on two promi- nent benchmarks: GSM8K (Cobbe et al., 2021) and MATH (Hendryeks et al., 2021). GSM8K contains 1,319 high- quality grade school math problems requiring multi-step reasoning, ranging from basic arithmetic elementary alge- bra. The MATH dataset comprises 5,000 competition-level problems covering advanced topics such as algebra, ge- ometry, number theory, probability, and calculus. These problems require more sophisticated problem-solving strate- gies and formal mathematical reasoning. The training sets of GSM8K and MATH datasets each contain approximately 7,500 data points. Each data point includes a question x, along with human-annotated rationale 2n and the correct final answer y"
(Schulman et al., 2017) or GRPO (Shao et al., 2024) to optimize this MDP.
2. BRiTE first obtains Q in (3.4) through RL, then updates in (3.5) using SFT. The SFT data consists of three components: problems x, thinking processes 20(2)) gen- erated by Q. along with the ground truth answers y".
3. We implement BRiTE-DPO, which consists of 3 itera- tions. During the t-th iteration, we first learn the distri- bution Q according to (3.4). Then for each prompt we leverage Q to generate N = 30 reasoning process and output pairs {( )a)) Subsequently, we evaluate the correctness of each 3/2 to identify the best and worst latent reasoning processes and construct response pairs (= vē. = for use in DPO training (3.6).
Base Models. We use several open-source instruction- tuned LLMs as base models, including Gemma-2-9b-it (Team et al., 2024b), Gemma-1.1-7BB (Team et al., 2024a), Dextral-7B-Instruct--oo (Jiang et al., 2023) and Llama-3- 8B-Instruct (Touvron et al., 2023).
Baselines. Our first baseline is rejection sampling (RS) methods (Neal & Hinton, 1998; Dong et al., 2023a; Yuan al., 2023; Zelikman et al., 2022). For each problem x, we generate N- 30 candidate rationales and select one (2r) containing the correct answer y". The model is fine-tuned on these problem-rationial aaationll--answer tuples {(z, 2is: }² Impor- tantly, this method does not use numan-annotated rationales z", making directly comparable to our approach. We also test the performance of supervised fine-tuning (SFT) on datasets with human-annotated rationales {[x, 2", ",,"" yy)] As baseline learning from preference data, we implement the iterative DPO (Xiong et al., 2024a; Pang et al., 2024), ini- tialized with instruction-tuned models. The training process consists of 3 iterations. In each iteration, we: (1) use the current model to generate 30 CoT and responses {(2, y)} per prompt; (2) select the best and worst response pairs
4.2. Experimental Result and Analysis
1. BRiTE Significantly Improves Existing Rejection Sam- pling Fine-Tuning Algorithms. We begin by comparing BRiTE-SFT with rejection sampling EM-type algorithmss, both of which aim to enhance the reasoning capabilities of LLMs by bootstrapping the thinking process. The key distinction is that BRiTE-SFT uses RL for this bootstrap- ping, while rejection sampling methods rely on sampling techniques. From Table 1, we observe that BRITE-SFT con- sistently outperforms rejection sampling- based algorithms across all models, achieving concrete accuracy improve- ment of 1-10 points. These improvements are primarily attributed to BRiTE-SFT's ability to generate higher-quality thinking processes compared to rejection sampling, high- lighting the potential of RL-driven approaches to advance
7

---

BRiTE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Mistral-7B-Innttuut--bb
Gemma-1.1-7B-it
Gemma-2-9B-it
Llama-3-8B-Instrud
Algorithm
GSM8K
MATH
GSM8K
MATH
GSM8K
MATH
GSM8K
MATH
41.8
9.8
49.0
18.8
81.3
37.3
79.2
28.3
SFT
52.8
13.6
57.5
19.6
80.1
41.5
72.6
27.1
RS
47.7
10.3
58.4
18.7
87.6
47.5
79.5
28.9
BRiTE
52.2
11.2
59.2
23.7
89.7
50.5
81.0
30.0
Table A comparison the performance three algorithms: BRİTE, rejection sampling (RS) type algorithms, and SFT using human- annotated data.
LLM reasoning through more effective bootstrapping of the thinking process.
BRiTE Matches or Even Enhances the Performance of SFT with Human-Annotated Thinking Process. To further validate the effectiienesss of BRiTE's RL-based boot- strapping mechanism, compare its performance against SFT using uuman-annotated thinking process data. Human annotations are widely regarded as the gold standard for training LLMs, as they include explicitly crafted reasoning pathways to ensure high-quality outputs. However, as shown Table BRiTE achieves performance on par with, and in some cases surpasses, that of based fine-tuning in downstream tasks. This highlights remarkable outcome: RL-generated thinking processes can achieve a quality comparable to or even superior to human- derived reasoning. This somewhat surprising result under- scores the value of BRiTE as cost-effective alternative to labor-intensive and timeconnumnng human annotation pro- cesses. By reducing reliance manual annotation, BRiTE sheds light on mitigating the bottleneck associated with creating high-quality datasets, particularly for complex rea- soning tasks where human annotations can be prohibitively expensive or inconsistent.
Figure 2. Comparison between BRITE and iterative DPO.
HvmanEval
BCB (Instruct)
Algorithm
Basic (%)
Plus (%)
Hard (%)
Full (%)
reasoning-
78.0
70.7
10.1
35.5
SFT
78.0
67.7
11.5
37.2
RS BRITE
79.3
73.2
11.5
35.6
81.7
72.6
15.5
36.3
Table 2. Results of BRiTE on coding generation task using the deepseek-cnder--6795l -ebiiir-.7b-insttrucc madel.
ity. Finally, we extend the evaluation of BRiTE beyond mathematics tasks to assess its performance on code gen- eration. The results, presented in Table demonstrate consistent trend observed in mathematics tasks: BRiTE outperforms rejection sampling-basee -bsed algorithms and even surpasses SFT using human-annotater answers. This high- lights BRiTE's versatility and effectiveness across diverse problem domains, showcasing its ability to generalize to a wide range of reasoning tasks. We also remark that rejection sampling- based algorithms require the code datasets be equipped with correct unit tests, which unnecessary for BRiTE. For detailed description of the experimental setup and configuration, refer to Appendix E.2.
BRiTE Further Enhances the Reasoning Capacity in RLHF Stage. In addition to advancing reasoning algo- rithms using (uesttion(raatonnal))) strates the potential to enhance RLHF algorithms that rely on preference data. As shown by the experimental results in Figure 2, BRiTE consistently outperforms iterative DPO across multiple benchmarks, highlighting its effectiveness in the RLHF stage. This superior performance is attributed BRiTE's ability to facilitate more structured and contex- tually nuanced reasoning processes, which, in turn, produce higher-quality preference data and enable more robust pol- icy refinement. These findings not only underscore the versatility of BRiTE but also affirm its value in optimizing RLHF-bbased fine-tuning pipelines, further cementing its role as a generic algorithm for enhancing LLM reasoning in the post-training stage.
data, BRiTE demon-
4.3. Enhanced Math Reasoning via Expanded Dataset and Advanced Base Model
In the previous subsection, BRiTE's performance did not demonstrate significant advantages over other algorithms like rejection sampling. This was primarily due to two factors: the instruct models had already undergone post- training, and the dataset size was limited, To overcome
4. BRITE Can Also Improve Code Generation Abil-
8

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Method
MATH500
Minerva Math
OlympiadBench
AIME24
AMC23
GPQA Diamond
44.1
12.9
16.1
0.9
10.1
25,9
RS
54.3
21.0
23.1
5.6
31.6
26,9
BRiTE (v-upddate
79.1
35.0
35.7
14.3
57.7
28.5
BRiTE (f-update)
76.9
40.6
37.0
14.4
57.1
29.8
BRiTE-iter-2 (1j-update)
80.6
41.3
37.3
14.3
57.9
29.9
BRiTE-iter-2 (8-update)
78.2
39.8
37.9
15.3
56.4
30.1
Table Performance comparison across different reasoning benchmarks.
these constraints, we have implemented larger training dataset based on the more advanced Qwen base model.
Specifically, we perform BRiTE on the model Qwen2.5-7B (Team, 2024) with mixed dataset (the size is around 40K) of RUC-AIBOX/
MATH, and historical AIME problems (excluding AIME 2024). We evaluated models trained using the RS bascline, the vi-update and 8-update of BRiTE on diverse set of reasoning benchmarks, including six challenging math and science reasoning benchmarks: Math-500, Minerva Math, Olympiad Bench, AIME24, AMC23, and GPQA Diamond. Here MATH500 contains 500 competition- math problems drawn from the MATH dataset (Hendrycks et al., 2021), spanning subjects from algebra to precalculus and accompanied by solutions. Minerva Math comprises 272 undergraduate--ever quantitative problems in physics, chemistry, biology, economics, and mathematics, designed to test advanced multi-step reasoning (Lewkowyez et al., 2022). OlympiadBench (He et al., 2024) is a collection of 8,952 Olympiad-caliber problems in math and physics (with 57% including diagrams) provided in both English and Chinese, each with a detailed solution. Drawing from its extensive entries, we selected 674 open-ended, competition--stlee, text-only problems from Olympiad- Bench. AMC23 (40 problems) and AIME24 (30 problems) are benchmarks based on recent AMC 12 (2023) and AIME (2024) contests (Mathematical Association of America, 2023; 2024), representing high school-level competition questions of moderate and high difficulty, respectively Finally, GPQA Diamond (Rein et al., 2024) consists of 198 graduate-level scientific questions verified by experts. Each of these datasets challenges models with different profile of difficulty and domain coverage, collectively evaluating broad spectrum of mathematical reasoning capabilities. We evaluate the pass@) @1 accuracy (64 sampling times for AIME24 and AMC23 and 8 sampling times for the others) for each benchmark.
Figure 3. Left: Training dynamics of BRiTE with an external veri- fier and rejection sampling. Right: Mean accuracy of benehmark scores of models trained by BRİTE with an external verifier and reject sampling during the training process.
strong impact RL-bootstrapped rationales. Moreover, we also show that the iterative training version of BRiTE leads to some improvements, though the training in iteration I reaches plateau that may limit further gains. Figure 3 (left) shows that BRiTE exhibits faster and more stable training loss reduction compared to rejection sampling, indicating more effective use of feedback during optimization. Fig- ure (right) further confirms this trend across benchmarks: BRiTE consistently achieves higher mean accuracy through- out training. These results indicate that BRiTE not only improves final performance, but also accelerates the acquisi- tion of reasoning capabilities during training.
5. Conclusion
In this work, we explore methods for enhancing language model reasoning through the automated generation of high- quality thinking processes. We present a unified probabilis- tic framework that characterizes both the reasoning process and evaluation signals. Within this framework, we develop the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which advances automated reasoning generation through reinforcement learning during inference and in- corporates improved reasoning processes into post-training phases. Furthermore, we demonstrate that BRiTE possesses provable convergence property and unifies various exist- ing learning paradigms and algorithms. Extensive empir- ical results on mathematics tasks show that our approach surpasses traditional -fthoughtt prompting while en- hancing existing supervised/rejeccioon sampling fine-tuning and reinforcement learning from human feedback methods.
As shown in Table 3. BRITE with an external verifier can improve reject sampling (RS) significantly. Specifically, MATH500, Minerva Math and AMC23, BRiTE improve upon RS by over 15 points in accuracy, showcasing the
http:::::///uuggggaaaeeeeeee/aee RUC-ATBOX/STILL-33 Ptw.rrt---------ei---ee-- Preview -revieeRLL-Date RL-Data

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Impact Statement
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. 2021.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
arXiv preprint arXiv:2
Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (method- ological), 39(1):1-22, 1977.
References
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06700 2023a.
Anthropic.
Introducing claude. 2023.
URL
https: intr introducing-claudee
wwt.ccooiiiin index/
Dong, Y., Wang, Z. Sreedhar, M. N., Wu, X., and Kuchaiev, O. Steerlm: Attribute conditioned sft as (user-steerable) alternative to rlhf. arXiv preprint arXiv:2310.0534 2023b.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant. A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08990 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(344):324-345, 1952.
Guo, H., Tan, B., Liu, Z., Xing, E. P., and Hu, Z. Efficient (soft) q-learning for text generation with limited good data. arXiv preprint arXiv:2106.00700 2021.
Bubeck, S. et al. Convex optimization: Algorithms and com- plexity. Foundations and Trends@ in Machine Learning,
8(3-4):231-357) 2015.
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L.. Shen, J.. Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, I.., Liu, Z.,, and Sun, M. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
Cai, Q. Yang, Z., Jin, C., and Wang. Z. Provably efficient exploration in policy optimization. In International Con- ference on Machine Learning, pp. 1283-1294. PMLR, 2020.
Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang. S., Schuurmans, D., Chi, Y., and Dai, B. celentincentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint vrXiv:2405.19320 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math datasct. arXiv preprint arXiv:2103.0388 2021.
Chen, H., Feng, Y., Liu, Z., Yao, W. Prabhakar, A., Hei- necke, S., Ho, R., Mui, P., Savarese, S., Xiong, C., al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:2411.0428 2024.
Hoffman, M. D., Phan, D., Dohan, D., Douglas, S., Le, T. A., Parisi, A., Sountsov, P., Sutton, C., Vikram, S., and A Saurous, Training chain-of- via latent- variable inference. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Chen, X., Zhong, H., Yang, 7., Wang, 7.,, and Wang, I.. Human-in -nohi--oopo Provably efficient preference-bart reinforcement learning with general function approxima- tion. In International Conference on Machine Learning. pp. 1377-3379 PMLR, 2022.
Hu, F. J., Shen, Y., Wallis, P., Allen-Zhu, 7.,, Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint 1XXiv:210600966 2021.
Christiano, P. F., Leike, J., Brown, T., Martic, M.. Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Hu, E. J., Jain, M., Elmoznino, E., Kaddar, Y., Lajoie, G., Bengio, Y., and Malkin, N. Amortizing intractable inference in large language models. arXiv preprint WxXiv2230.04330 2023.
10

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Huang, S., Cheng, T., Liu, J. K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J., Zhang, C., Chai, L.,, al. Opencoder: The open cookbook for top-tier code large language mod- els. arXiv preprint 24X::::441.009000 2024.
Nemirovskij, A. S. and Yudin, D. B. Problem complexity and method efficiency in optimization. 1983.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
OpenAl. Introducing openai ol. https://opent: com/ol/ 2024.
Ouyang. I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 15:27730-277744 2022.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer. E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solle T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Pacchiano, A., Saha, A., and Lee, J. Dueling rl: reinforce- ment learning with trajectory preferences. arXiv preprint arXiv:2111.04855 2021.
Pang. R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733 2024.
Liu, G., K., Zheng. R., Wu, Z., Dun, C., Gu, Q., and Yan, L. Enhancing multi-step reasoning abilities of language models through direct q-function optimization. arXiv preprint ar arXiv:2410.000 09302, 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Rein, D., Hou, L.., Stickland, C.,, Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proee poggle-rooff q&a benchmark. In First Conference on Language Modeling, 2024.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. 16436, 2024b.
Rush, S. and Ritter, D. Speculations on test- time scaling. attps::/srusse githh.githhhi tutb...... io/ catecmee-ol/l-ttttee pdf, 2024.
Lu, X., Welleck, S., Hessel, J., Jiang, L.,, Qin, L.,, West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Mathematical Association of America. American mathe- matics competitions ame 12, 2023, 2023. URL https: //maa org wwtp:///aatccoteettttii....eeeee
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song. J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Mathematical Association of America. American invitational mathematics examination aime i &
ii. 2024, 2024. URL pettps:/mma...e
Singh, A., Co-Reyes, D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06558 2023.
maa - www.iiataiiillloommeeiiiiiee
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-frrr freee reward. arXiv preprint arXiv:2405. 14734, 2024.
Neal, R. M. and Hinton, G. E. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pp. 355-368. Springer, 1998.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint 4XXi:2408.00314 2024.

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. A., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.057749 2024.
Xiong, W., Dong, H., Ye, C., Wang, Z.. Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-conntraint. In Forty-first International Con- ference on Machine Learning, 2024a.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.cm 08295, 2024a.
Xiong, W., Shi, C., Shen, J., Rosenberg, A., Qin, Z.., Ca- landriello, D., Khalman, M., Joshi, R., Piot, B., Salch, M., et al. Building math agents with multi-turn iterative preference learning. arXiv preprint PrXiv:2409.02555 2024b.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001188 2024b.
Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Chen, J. sewards-in-contedd Multi-objective alignment of foundation models with dynamic preference adjust- ment. arXiv preprint arXiv:2402. 10207, 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL htttps://wwenmm.
Touvron, H., Lavril. T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.,, Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 02.000..19971 13971, 2023.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Wang, H., Hao, S., Dong, H., Zhang, S., Bao, Y., Yang, Z., and Wu, Y. Offline reinforcement learning for IIm multi-step reasoning. arXiv preprint 1111:22412.16155. 2024.
Yuan, Z., Yuan, H., Li, C.,, Dong, Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint arXiv:2308. 01825, 2023.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
Yue, Y., Broder, J., Kleinberg. R., and Joachims, T. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5): 1538-155, 2012.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476-15488, 2022.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le. Q. V., Zhou, D., et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., and Goodman, N. D. Quiet-star: Language models can teach themselves think before speaking, arXiv preprint arXiv:2403 0.009,,9999..... 2024.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. A survey of preference-basee reinforcement learning meth- ods. Journal of Machine Learning Research, 18(136): 1-46, 2017.
Zhang, S., Yu, D.,. Sharma, H., Zhong, H., Liu, Z., Yang, Z., Wang. S., Hassan, H.,, and Wang, Z. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprini arXiv:2405. /9332, 2024.
Wu, T., Lan, J. Yuan, W., Jiao, J., Weston, J., and Sukhbaatar, S. Thinking Ilms: General instruction following with thought generation. arXiv preprint arXiv:2410. 10630, 2024.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. /0425, 2023.
Xie, T., Foster, D. J., Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit a approxxxiiaioo for sample- efficient rlhf, arXiv preprint arXiv:2405.2 2024.
Zhong, H. and Zhang, T. A theoretical analysis of optimistic proximal policy optimization linear markov decision processes. Advances in Neural Information Processing Systems, 36, 2024.
12

---

BRiTE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Zhong, H., Feng. G., Xiong, W., Zhao, L.,, He, D., Bian, J., and Wang, L. Dpo meets ppo: Reinforced token optimization for rihf. arXiv preprint arXiv:240411892, 2024.
Zhou, D., Schärli, N., Hou, L., Wei, Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint 2202: 2022.
Zhuo, T. Y., Vu, M. C.,, Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
Ziegler. D. M., Stiennon, N., Wu, J.. Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08599 2019.
13

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
A. Author Contributions
This work stems from the valuable contributions and close collaboration of all authors. The first seven authors and ZW are the core contributors this project, all participating in discussions of problem formulation and theory, as well as experimental implementation. In particular, HZ and ZW lead the project, primarily propose the methodology. derive the theoretical results, guide experimental progress, contribute to early baselines, and write the paper. YY, SZ, and XX mainly contribute to the initial implementation of BRiTE without verifier reward for Llama, Gemma, and Mistral models, while YL. YZ, and ZL mainly contribute to BRiTE with verifier reward for larger datasets and Qwen base models. In addition, YL, YZ, and ZL make key contributions to direct preference learning, rejection sampling, and code generation, respectively, Other authors also make significant contributions to this work, providing computational resources and offering suggestions for theoretical analysis, experiment design, and paper writing.
B. Additional Related Works
RLHE RLHF (Christiano et al., 2017; Ziegler et al., 2019), also known the dueling RL (Yue et al., 2012; Pacchiano et al., 2021) or preference-based RL (Wirth et al., 2017: Chen et al., 2022), has emerged as breakthrough technique in language model development, playing a pivotal role in the success ChatGPT (Ouyang et al., 2022) and subsequent large language models by effectively aligning model outputs with human preferences. the standard RLHF pipeline, the first stage involves learning reward function from human preference data, followed by optimization using proximal policy optimization (Schulman et al., 2017). However, this approach demands substantial computational resources and requires careful tuning of numerous hyperparameters. Direct preference learning (e.g.. Zhao et al., 2023; Rafailov et al., 2024; Azar et al., 2024; Tang et al., 2024; Meng et al., 2024; Liu et al., 2024b; Cen et al., 2024) offers an alternative by learning the desired policy directly, circumventing the need for explicit reward learning and optimization. Recent work by Xiong et al. (2024a); Xie et al. (2024); Cen et al. (2024); Zhang et al. (2024) extends these offline approaches through online iterative learning to gather on-policy data for enhanced performance. However, these algorithms mainly focus on the chat task and do not explicitly model the latent reasoning process, thus not fully extracting the reasoning power of LLMs. While Pang et al. (2024); Wu al. (2024) incorporate reasoning processes into DPO training using CoT prompting, our approach distinctively generates reasoning processes automatically through RL, demonstrating significant improvements in model reasoning capacity. Some work (e.g., Xiong et al., 2024b) combines iterative DPO with tool use to improve reasoning capabilities, which is beyond the scope of our current work.
C. Missing Proofs in the Main Paper
C.1. Proof of Lemma 3.1
Proof of Lemma 3.1. For any g(-) e A(W), we have
Pwr
E Ewwsg() (log Pu -logg(w)) - log Σ P: Frwng()))
log
log ((w) KT.(alp)) s O.
Eweew ew Pw
WEW
where ÿ is the distribution defined as p(w) Pwu Pwr), and the equality is achieved when q P. Hence, we have finished the proof of Lemma 3.1.
C.2. Proof of Proposition 3.7
Proof of Proposition 3.7. The first equation follows from the deterministic transition. We will now focus on proving the second propositional relationship. According to equation (2.3), we have:
T T'(a; (s:) exp((Q"(s;; az) V (s:))//)),
Whsish HI
14

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
which implies that
3 log x (a: 9:) =
Σ (s; -v((ss)
1=h
(r) + - + (r(sn, - (*(s))
Σ r(si, az) (*)),,
(C.1)
where the second equality uses the fact that aи EOS. Hence, we have
H = exp 3 E H [arl:)))))
r(s),
H
& exp (B i=h
Erss,ai))
exp(V*
k=h
where the last step is obtained by the fact that 8h is fixed state, independent of as U (:)
D. Convergence Results
D.1. Proof of Theorem 3.3
Before starting the proof of Theorem 3.3, we present two technical lemmas.
Lemma D.1. For any (0,0)) 0') and fixed x E X, we have
(1, (r.s x2) = A(z, A(x, (VA(x,)). )
Proof of Lemma D.I. By the definition of KL-divergence, we have
P(z, y x, 0) log P2, y x
(L x, (.. 12, =
Σ
P(z, y x,
(x
Σ
P(z, y x. [o, 2,8) - for (x,, 2. A(x,0' - A(s,
(x))
A(x, 0') A(x, 0) (B(=..)))P 1z.0) [K((,, 2, v), }), fo for),
(D.I)
where the last equality uses (i) Σ ((a)) P( 12, xr, 1; and (ii) the assumption that ʃo E (Assumption 3.2).
Meanwhile, by Assumption 3.2, we have
A(x, 0) log Σ (x,))xZxy
exp(fo(z,,,.)) log Σ
(x.))
which implies that
A((x0))=
Exewezzyyepp((((( (x)(((()))(((())))))))))))))
exp((X((xx 2, y). fol)
= Σ
[((x,,,)))
Σ (ZaDEEzY exp (K( v),
(x.))2x))
Σ P(z,y| xx K[( 2, 2). (x.))
(D.2)
where V is functional gradient. Combining (D.1) and (D.2), have
(L x, () 0')) A(x, A(x,0) (VA(z,0),f) )
which finishes the proof of Lemma D.I.
15

---

BRiTE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Lemma D.2. It holds that
Vlog P(z E ,IEEW,EE
,0E0 0 x,0) =
vA(x,
Loco P(3.0oo)))))))))
where Q2) ( ((ag0
Proof. By definition, have
Vlog (.1/2eEE,,0 2, 0) V log
Σ P(2,8.0|2;0)
E Vf(z,y.0|z,1
E P0 (2,
E(exooeexxxx (f0l / 2,,0)
(D.3)
E x 0
where the last equality follows from (3.1). Meanwhile, by Assumption 3.2, we further have
vP(zzyl x, V Ve 0))
vA(o,
P(2,8/2,0) (K((z,,,,),)) Av(,, 0)).
(D.4)
Combining (D.3) and (D.4), we have
(0 (P(o) V x,
Vlog P(z E 2, E 2 00000000 (000)100)
Eoco Pl y,
Σ
[K((),,,))))) -AA)))))))))((()
(2.))xx E(t.,))eX' P(z,
(a (1.
2),, 21(x,,,)))) VA(x,
(D.5)
This finishes the proof Lemma D.2.
Now we start the proof of Theorem 3.3.
Proof of Theorem 3.3. By the updating rule of 0c+1 in (3.5), we have
0.41 argmax
DggF(z,l/2,00 (,,0)))(2e,,, 01) (f.Mo)E
Σ
6
Σ
logP(z. a2 v x, 0.)
argmax
0
(x
where we use the notation Q(z 02) = Eoco QF 0.). Furthermore, we have
0++1 = argmax
Σ
log P(z, y n 0) Q2 y x. 02)
9
(#w)) x
Pl y x. @(2,8/1 0,) P(z, y x 0.)
argmax
log
(#N)) x
(E(s.a))Q6.1aa (....) [x, 2, v), )), So Sa.) A(x, A(2,0)
argmaxx
(D.6)
16

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
where the last equality is implied by
(z, y 0, fo(2,2, 2, fo. (x. + A(x, 00) - A(r,0) P(zz x, x,)
log
(K(((,,z ;) So.) + A(x, A(x,
Combining (D.6) and Lemma D.2, we have
0++1 argmax log By the optimality condition, this implies that
P(2 (VlggI(ze 2.y 30 0l200,) VA(2, -o.) + A(x,0) A(z,0))
(D.7)
vA(0++1)) VA(O.) V VlogP(2 2, 2 ,0 012,0.).
(D.8)
We have
(TA(x,0:41) - VA(z, Oc), fo..)
(VA(x,,))) f.), - fa.) - A(I, 0(+1) + A(x,0))
AV,( 0z), So- - )0,) + A(x, 0²) - -A(x, 0.)
(V,(x, So- - So(n)) A(x, 0-) + A(x 0c+1)
-KL(P(ss 1 0.) 12,55111,,00000 0c+1)) + (L((( 0.)11P(s - KL(P(s 12, 1=,0)).
where the last equality follows from Lemma D.1. This is equivalent
KL (P(:, - x 0.) (,, 0*)) - (F 1x, Oc+ 10+0))1((P (I (x,,
(VA(6.+1) - VA(O,), for f + KL (P(, - xx, (.)) xx, 0.+1))
log y V ,00 0 x, 0c), for fo...) (LL(P( 0,) IP(, 1x, 0c41))
(D.9)
where the last equality follows from (D.8). For the Term (+), we have
(*) v log ((= E 2,y y V/ 0 x. 0c), So- so,n
V log 2. 2 0 x. 0₂), fo- fo.) + (Vlog P(2 E F.yE e V ,00 0 x1 0| x10,), fo. foc4)).
(II
(D.10)
Term (I).
By the local concave assumption, we have
Term (I) (D.10) 2 log P(z E 2, 2 o 0. (0001100)
log E 2. E V.0 0.000 x1),0).
(D.11)
Term (II). By Lemma 3.1, we have
log (p= 2, y E V
000 0 xr, 0c+1)
Σ
Log|P(z,y,00 0-+1) log a2 0.o x, 0.++1) Q(2, y. 00),0,),
(D.12)
xx
where
P(z, y, x 0)
F(z,y.0 0 x
02,,0,012,00
(D.13)
E( xo P( 1/ x)
P(z E F.y V 00 x)
17

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Note that Q(--'- x. 0)) is a distribution over 2 y 0, which means that Exxgooezxxxx Q2,,,,01z,0) Hence, we have
P( 2. 0z
Σ
Q2. 2,0,) log 2. E ,0 ,0 x,,0,)
(2.9.0) x x
Q22 x. 0.) log P(z, x. 0z)
Σ
(D.14)
Q y. x.
(x.
where the last equality is implied by the definition of Q 0. x. 0) in (D.13). Combining (D.12) and (D.14), we have
log P(= E 2, y E y
, E 0 x, log e 2, ,0 oE 0 (x,0,)
22, KL. [Q(s x. (Q((s))))( (2,1) 0e41))
Q(2,v.0/2,00) log F(2,y.0|2,60 (x
Σ
00, 0/ (1,1+1
Σ
01,,0, log
PP 0.)
(=
P(z. y 2z 0(41)
Σ
0,) log
(D.15)
PP(2, y (,,
xx
where the third equality follows from the non-negativity of KL-divergence, and the last equality is implied by (3.1). By Assumption 3.2, we can rewrite the right-handside (D.15) as
Σ
Q(2,y,012,,0) 0s) (for:1 (ar v, fo, (a, 2,1)) + A(x, A(x,0++1)
(2 xM
(B(..))-(..11.11)
2, y) }], fern1 - fee) + A(x, 01) A(x, 0c+1) (....)))(...1..0))))0) [K](( 2, u), -)] - VA(x, So.11 - S0.) - KI. (P(, 1 0.) (.) |x, (c,0)+11)
= (V log P(: E 2 E 3 E 0 xr Jo. - -fo.) - KT. (T, (2r, 0z) PP) xx. 0c+1)),
(D.16)
where the third line is implied by Lemma D.I. and the last equality follows from (D.5). Plugging (D.16) into (D.15), we have
Term (II) in (D.10)
V log F . 0lz,0)), fo. fe...) 2 P( E 2 v, E 0 2|, 01) - log P(: E 2, y 0 01000
11+))
(D.17)
Combining (D.10), (D.11), and (D.17), have
(+) 2 Term (I) + Term (II)
2 log P(= E 2 y E 2 o 0 01 x, log 2 y V . 0 (1,00) logP 2 N V .0 0 x,0,)
- -logP(2 E 2 y E oE 0 x, 0c+1) (L( 12, -
log P(= 2, y E v o 0 x, 0)) P(z E Z. v. E 0 11, 0+11)
- (KL x, (P( - 02+1)).
(D.18)
Putting (D.9) and (D.18) together, we obtain
KL ((: 1 x, IP( xx 0)) -
(2,00))0)))
2 log E F. E 3 o E 0 x, 0*) log P(z E F. E V ,00 0 (1,0(14).
Telescoping this inequality from 0 to T - 1, we obtain that which implies that
(L(P(s) (2. 0*)
min (log P(= E 2. y E V o 0 x. x, - log E 2. y E 3.006 (000000))) x.01) 1SETT
T
which finishes the proof of Theorem 3.3.
18

---

BRITE:
Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
D.2. Additional Theoretical Results
Theorem D.3. Suppose Assumption 3.2 holds. Then we have
min (P 12, (2,00+1) (IP()) 2,) log ISSS
0T+1) - P(z E 2, V ,00 0 x2,0))
T
Proof of Theorem D.3. By the same derivation of (D.17), we have
(L((((: O.)lP(, 1 0c+1)) 2 log P(z E 2, ,0 - P(= E 2, ,0E 0 21,00++1
(VVogg P(z 2, " 0 21 0,), So. sonn)
log P(z E 2, ,0 0 01 (2) log P(= E 2, E , 0 x. 0c+1)
(VA((O++) A(O+)) So. So..1)
(D.19)
where the last equality is implied by the optimality condition in (D.8). By Lemma D.I, we have
KL (P 1x, 0.)][I((s 0.+1))) = A(x, 0c+1) - A(x, 0,) + (VA(z, (e)(,,0)),fe - Sott))
(D.20)
Combining (D.19) and (D.20), we have
Aar, 0++1) - Alrr (VA(O++)), - -V
2 log P(2 E 2,y y E 2 e 0 012, 01) - log Poz &, . y ,0 00 01
Together with Lemma D.1, we have
log P(= E 2, E 2 , (,, log 2 2, 20ye,,00
0E 1 12,0,)
2 A(x,0,) A(x, 0(+1) + (VA(O++1),, - fe.) (L(P(( 12, 1c 0.)).
Telescoping this inequality across t [T], we have
[L 12, 0++1) 12,
min (PL 12, (.41) (2)) (E[2]
T
log P( 2,y E V .0 0
x.
GT 1) - P(z 2, 2,y E 3 E 0 xz 01)
T
which finishes the proof of Theorem D.3.
E. Implementation Details
E.1. Implementation Details for Math Task
Implementation of BRITE. The BRITE algorithm run on 4 NVIDIA H100 during training. We leverage the PPO pipeline (Schulman et al., 2017) to learn the sampling policy Q (3.4) with a learning rate 5e 7 and batch size of 1. For the subsequent SFT on rationales sampled by Q. we the learning rate 5e and the batch size 2. We adopt the LoRA (Hu et al., 2021) training for both steps, where the is set to 32 and lora alpha is set to 128.
Implementation of RS. For rejection sampling, we the temperature of the model's generation 1.0, sample N 30 candidate rationales for each problem z, and select the best rationales 4s. filter generations by selecting those that produce the correct final answer. The model is then fine-tuned on these problem-rationall--nwww tuples {a, Ers v²} using the same learning rate and LoRA parameters as in BRiTE.
Implementation of SFT. For SFT on original datasets with human-annotated rationales, we use the same LoRA parameters as BRITE. The learning rates for -attral--BIInttuuu
and Ce are set to 5e-5. For Gemma- - 2----bbb- 2-9biit, t, the learning rate is 5e 6, and for Lma - 3-88---B----- GBB-Instucc,, it is Se-7 These smaller learning rates are chosen to mitigate overfitting and ensure optimal performance.
19

---

BRITE: Bootstrapping Reinforeed Thinking Process to Enhance Language Model Reasoning
Implementation of RLHF Algorithms. For both BRiTE DPO and iterative DPO, we select 6k data points per iteration from the RLAF=MATH/prrmmp itpettiteerrl i dataset² over three iterations. The learning rates for DPO training
configured follows: Mistral- II-Instruct vv. 2 uses learning rate of 2e 7, Gemma 1 11-B--it it employs 4e 7, while Gemma-2-9B-15 and 1-aam-3-8B- B----BIInttuuee are set to 5e - 7. The generation temperature is set to 1.0, and each prompt sample N = 30 responses.
Evaluation. We utilize the evaluation function provided in the Qwen2. 5 S-Math repository" to assess the models' performance on the test sets of GSM8K and MATH.
E.2. Implementation Details for Coding Generation Tasks
Datasets. For the code generation task, We choose the first 4000 rows of the educational instruct split of the dataset OpenCoder
et al., 2024) as the training dataset, which comprises (instruction, code, test case) uples. The entire dataset is generated with an algorithmic corpus as seed and validated through a Python compiler.
Models. We choose the language model deepseek ai deepseek -oweseeeee--cceeeeeeeeeeee billion active parameters as the base model, which is especially pretrained and fined-tuned the code corpus for better code generation performance.
coder- 6. 7b -innsrutttt with around seven
Baselines. Similar to the mathematics tasks, we use Reject Sampling (RS) and SFT as the baselines. We follow a rejection sampling approach similar to prior math section. The model generates candidate rationales with a temperature of 1.0, sample = 30 rationales for problem x. The best rationales, 2ns is then selected. The model is then fine-tuned on these rationale-answer tuples (x, Z w). Notably, unlike the mathematical setting, filtering correct generations the code generation task requires candidates to pass all unit tests and receive positive compiler feedback.
Benchmarks. We evaluate the trained models on the HumanEval task from Evalplus (Liu et al., 2023) and the instruct split of the BigCodeBench (Zhuo et al., 2024). These popular benchmarks evaluate how the language models complete the partial code snippet and generate full code snippet according to natural language instructions.
Training Details. We use 4 NVIDIA A100 GPUs for all the training. We leverage the PPO pipeline (Schulman et al., 2017) to learn the sampling policy Q (3.4) with learning rate of 5e 7 and a batch size of 1. With the rationales sampled from Q of BRiTE, we set the learning rate as 5e - 5 and the batch size as 2 in the SFT stage of BRİTE. For the baselines, we also set the learning rate as 5e 5 and the batch size as 2 in the SFT algorithm. For the fine-tune stage of reject sampling, we use the same learning rate as in BRiTE. To save the computation budget, we adopt the LoRA (Hu et al., 2021) training for all the optimization phases, where the is set to and lora alpha set to 128.
Evaluation Details and Results. We report the pass@1 scores (the success rate of the first attempt) with the greedy decoding for the base model, SFT algorithm, Reject Sampling (RS) algorithm, and our proposed algorithm BRiTE-SFT in Table 2. Results show that our proposed algorithm BRiTE can help LLMs improve their code generation ability and demonstrate the performance gain of BRiTE compared with the baselines. We also remark that rejection sampling-bans based algorithms require the code datasets to be equipped with correct unit tests, which are unnecessary for BRİTE
http://r/////ttttoooooo///////// 5-
hhttps://github.com httpa//....oo//eee///////e/
coder--- ib-instruct.
20


Reviews:
[{"review_id": "IQfAH3191C", "reviewer": "Reviewer_UuED", "summary": "This paper works on the reasoning process generation problem. The authors propose formalizing the reasoning problems as a probabilistic graphical model involving input, latent rationals, answer, and evaluation signal. Then they presented the BRiTE algorithm with a theoretical guarantee on convergence. The proposed method outperforms rejection sampling and SFT on math and coding benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness:**\n(1) Reasoning task diversity is limited. (2) The performance of BRiTE is not significantly better than other efficient methods.\n\n**Response:** We perform BRiTE on the model *Qwen/Qwen2.5-7B* with a mixed dataset (the size is around 40K) of *RUC-AIBOX/STILL-3-Preview-RL-Data*, MATH, and historical AIME problems (excluding AIME 2024).  We evaluate models trained using RS baseline and BRiTE on a diverse set of reasoning benchmarks, including Math-500, Minerva Math, Olympiad Bench, AIME24, AMC23, and GPQA Diamond. The pass@1 accuracy (64 sampling times for AIME24 and AMC23 and 8 sampling times for the others) for each benchmark is reported in the table below. \n|    Method           | Math500 | Minerva_math | Olympiadbench | Aime24 | Amc23 | Gpqa_diamond |\n|----------------|---------|--------------|---------------|-----------|----------|--------------|\n| Base Model          | 44.1    | 12.9         | 16.1          | 0.9       | 10.1     | 25.9         |\n| Reject Sampling| 54.3    | 21.0         | 23.1          | 5.6       | 31.6     | 26.9         |\n|BRiTE| **76.9**    | **40.6**         | **37.0**          | **14.4**      | **57.1**     | **29.8**        |\n\nEvaluation results show that BRiTE can surpass the RS baseline with a stable margin on these bechmarks, which demonstrates the power of BRiTE in the face of **large datasets (40k data)** and boosts the performance of the state-of-art model such as *Qwen/Qwen2.5-7B*. \n**Question 1:**\nCould the authors discuss the algorithm in terms of efficiency? \n\n**Response:** Thank you for your question. We will discuss data efficiency and computational efficiency as follows:\n- *Data Efficiency:* (i) We want to emphasize that our algorithm only requires data without human annotation, making it more efficient than supervised fine-tuning (SFT) that relies on annotated data, which can be harder to collect; (ii) compared to other methods, such as rejection sampling, which also do not require data for the thinking process, our algorithm demonstrates greater efficiency since we have controlled for the same amount of data in all our experiments.\n- *Computational Efficiency:* As mentioned earlier, our algorithm does not need data with a thinking process, so it is not fair to compare our algorithm (or other methods like rejection sampling) with SFT, which relies on human-annotated, step-by-step thinking data. Moreover, both our algorithm and rejection sampling (RS) share the same goal: generating a high-quality thinking process (through RL or RS) and then fine-tuning LLMs based on that. The generation process is the main contributor to computational costs, a factor that is unavoidable for all these methods. Specifically, the first stage of our method (rollout + RL updating) and the RS filtering process (rollout + filtering) both require approximately 30 hours. This main cost is attributed to the rollout of the Qwen2.5-7B model, generating outputs for 40.6K data points with 8 samples per point. Since RL provides more concise/shorter responses, the overall computational time (rollout + RL updating) is comparable to the RS filtering process. Conversely, the training phase (SFT for RS or $\\theta$-updating in BRiTE) is significantly less demanding, requiring only 3.3 hours for 9K data points from RS, or 9 hours for 24K data points from RL, for four epochs. our method achieves significantly better performance without substantially increasing computational costs.\n\n\n**Question 2:**\nIt would be very helpful if the authors could explain the algorithm in steps. In my current understanding. The main contribution is to provide a new loss to optimize. \n\n**Response:** Regarding the implemented algorithm, the main contribution is providing a new approach to generating high-quality latent thinking processes through RL objective with a reward function (as detailed in Section 3.4). Specifically, inspired by our theory, the learning algorithm guarantees provable convergence by iteratively updating equations (3.4) and (3.5). Equation (3.5) is relatively easier to implement, while the key challenge lies in solving equation (3.4). Guided by Proposition 3.7, solving the intractable distribution in (3.4) is equivalent to addressing an entropy-regularized Markov Decision Process (MDP) with the reward function $\\log P(z, y, o\\mid x)$. This can be further specified using the graphical model; for instance, if we let $P(o = 1 | z, y, x) = \\exp(\\beta \\cdot R(x, z, y))$, we can define the reward function as $\\log P(z, y | x) + \\beta \\cdot R(x, z, y)$ with a tuning parameter $\\beta$. This can be attributed to each token through the autoregressive policy property, allowing us to assign the reward to the last token.\n\nBeyond the algorithmic contributions, our main contribution includes a **new unified framework, theoretical guarantee, superior performance, and the demonstration of RL-guided reasoning's potential**. Due to space constraints, we refer you to our detailed response to Reviewer afMW.", "claims_and_evidence": "Mostly yes. \n1. The authors provided a detailed explanation of their reasoning framework and proved the theoretical guarantee.\n2. They mainly experimented with four models on GSM8K and MATH. BRiTE is consistently better than rejection sampling and is only slightly worse than SFT when using the Mistral model. \n3. In the results, when the deepseek model was used on coding generation tasks. BRiTE-SFT shows better performance with only one exception.", "methods_and_evaluation_criteria": "Yes, the two benchmarks are widely used, and the evaluation metrics are reasonable.", "theoretical_claims": "I went over the proofs of convergence guarantee briefly. I did not find any issues.", "experimental_designs_or_analyses": "The paper compared their algorithm with rejection sampling, SFT, and iterative DPO. I think the experiment is lightweight but enough for claims. But definitely more datasets and benchmarks would increase the soundness.", "relation_to_broader_scientific_literature": "The proposed framework incorporated the popular CoT idea, rejection sampling. And can be applied in RLHF and SFT scenarios.", "essential_references_not_discussed": "not found", "other_strengths_and_weaknesses": "Strengths:\n1. Originality: new generalizable reasoning frame, method, and theoretical analysis.\n2. Presentation is clear with minor typos\n\nWeakness:\n1. Reasoning task diversity is limited\n2. BRiTE could introduce heavy computational overhead, but the performance improvement is not significantly better than other efficient methods.", "other_comments_or_suggestions": "no", "questions_for_authors": "Hi, \n1. Could the authors discuss the algorithm in terms of efficiency? Including data efficiency and computation efficiency.\n2. It would be very helpful if the authors could explain the algorithm in steps. In my current understanding. The main contribution is to provide a new loss to optimize. Please let me know if my understanding is incorrect.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pPtws6OS4P", "reviewer": "Reviewer_afMW", "summary": "The authors propose a unified graphical model to incorporate latent thinking processes and evaluation signals. Drawing inspiration from the EM approach, they unify current post-training methods into the same framework, such as SFT, RS and DPO.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments on \"Claims and Evidence\":**  In reality, datasets like MATH have very careless annotations, so the performance of SFT may not be as effective as LLMs generating their own data. This is a point that has been widely discussed in many papers and is not necessarily an interesting finding.\n\n**Response:** Thank you for bringing this to our attention. If a widely used dataset like MATH still struggles with careless annotations, it strengthens our motivation for research, as obtaining data that accurately reflects a high-quality thinking process is challenging. Additionally, our algorithms demonstrate their effectiveness by comparing with other algorithms, showcasing their ability to learn without relying on high-quality step-by-step data.\n\n\n**Comments on \"Methods And Evaluation Criteria\":** It would be better to include a broader range of tasks to evaluate its general applicability.\n\n**Response:** We perform BRiTE on the model *Qwen/Qwen2.5-7B* with a mixed dataset (the size is around 40K) of *RUC-AIBOX/STILL-3-Preview-RL-Data*, MATH, and historical AIME problems (excluding AIME 2024).  We evaluate models trained using RS baseline and BRiTE on a diverse set of reasoning benchmarks, including Math-500, Minerva Math, Olympiad Bench, AIME24, AMC23, and GPQA Diamond. The pass@1 accuracy (64 sampling times for AIME24 and AMC23 and 8 sampling times for the others) for each benchmark is reported in the table below. \n|    Method           | Math500 | Minerva_math | Olympiadbench | Aime24 | Amc23 | Gpqa_diamond |\n|----------------|---------|--------------|---------------|-----------|----------|--------------|\n| Base Model          | 44.1    | 12.9         | 16.1          | 0.9       | 10.1     | 25.9         |\n| Reject Sampling| 54.3    | 21.0         | 23.1          | 5.6       | 31.6     | 26.9         |\n|BRiTE| **76.9**    | **40.6**         | **37.0**          | **14.4**      |**57.1**     | **29.8**        |\n\nEvaluation results show that BRiTE can surpass the RS baseline with a stable margin on these bechmarks, which demonstrates the power of BRiTE in the face of **large datasets (40k data)** and boosts the performance of the state-of-art model such as *Qwen/Qwen2.5-7B*. \n\n**Weaknesses:** I did not find enough novel approaches that significantly contribute to improving LLM reasoning abilities. Instead, the paper primarily re-expresses previous methods within a unified framework.\n\n**Response:** Thank you for recognizing that our proposed framework is general, encompassing many previous methods as special cases. We also want to highligh our four contributions:\n1. **Novel Framework and Algorithm:** The unified framework itself, which innovatively utilizes a graphical model to incorporate both thought $z$ and the optimality $o$ in the explanation of LLM reasoning, is an important contribution to theoretically characterize LLM reasoning. Furthermore, the two-stage optimization algorithm BRiTE, derived from this framework, includes various RL paradigms as special cases and introduces new learning algorithms, demonstrating the framework's utility. \n2. **Unified Theoretical Guarantee:** We provide a unified theoretical guarantee for BRiTE, thereby offering a theoretical foundation for a broad range of algorithms (for the first time) within this framework.\n3. **Superior Empirical Performance:** Our method significantly outperforms previous methods on bootstrapping thinking process (please see the results in our paper and the updated results in the table below).\n4. **RL-Guided Reasoning Paradigm:** Our work demonstrates that RL-guided reasoning paths are effective for generating high-quality thinking processes. This aligns with the concurrent DeepSeek-R1 project, which also employs RL training to generate SFT data.\n\n**Additional Question:** Has anyone evaluated the quality of the generated data? Is there any intuition that explains why the new data is of higher quality?\n\n**Response:** \nWe evaluated the perplexity (the mean negative log probability of the model on the responses) of the base model (Gemma-2-9b-it) using the responses from both the human-annotated data in the corresponding dataset and RL-guided data (generated by model after $\\psi$-update). The results demonstrate that the RL-guided data exhibits significantly lower perplexity, which means models are easier to learn the reasoning path from the RL-guided data.\n\n| Perplexity             |GSM8K | MATH |\n|----------------------|------------|------------|\n| Human-annotated data | 4.14       | 2.84       |\n| RL-guided data  | 1.64       | 1.40       |\n\nDue to the page limit of the rebuttal, we could not provide a concrete comparison between the RL-guided response and the human-annotated response, which shows that the RL-guided reasoning path can provide more detailed and clear steps for thinking and is easier for the model to learn and derive the correct answer. If you are interested, we are pleased to provide this comparison in the new round of rebuttal.", "claims_and_evidence": "I believe the authors may have exaggerated their claims, such as in the abstract where they state that their method \"can even match or exceed human-annotated data.\" In reality, datasets like MATH have very careless annotations, so the performance of SFT may not be as effective as LLMs generating their own data. This is a point that has been widely discussed in many papers and is not necessarily an interesting finding.", "methods_and_evaluation_criteria": "Regarding the evaluation benchmark, currently only tasks like code and math, which are easier to evaluate based on final answers, have been used. However, as mentioned in Section 3.4, the method should be effective for any token sequence. It would be better to include a broader range of tasks to evaluate its general applicability.", "theoretical_claims": "I checked it briefly and did not find any issues.", "experimental_designs_or_analyses": "The improvement in the experiments is too small, and the baseline scores are somewhat low. For instance, for Llama-3-8b-Instruct, according to the [Meta-Llama-report](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), the current method only matches the performance of the 4-shot approach on MATH. Given that the method still requires SFT and RL training, the improvement is quite limited.", "relation_to_broader_scientific_literature": "The author mentions the recent paper in related works.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**  \nThe paper introduces an innovative graphical model to represent the LLM post-training process.\n\n**Weaknesses:**  \nI did not find enough novel approaches that significantly contribute to improving LLM reasoning abilities. Instead, the paper primarily re-expresses previous methods within a unified framework.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Has anyone evaluated the quality of the generated data? Is there any intuition that explains why the new data is of higher quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9EHalCPrfE", "reviewer": "Reviewer_8vbs", "summary": "Authors propose a new approach to incorporate latent thinking processes and evaluation signals in the model training. Authors propose two-step approach: 1. generate high-quality rationales by approximating desired posterior of thought given the question-answer pair; this approach relies on reinforcement learning, using a novel token-level reward function; and 2. enhance the base LLM by maximizing the joint probability of rationale generation with respect to the model’s parameters. Importantly, this approach does not require reference chain-of-thought data, but instead relies on model to generate rationales. Authors provide extensive theoretical background for the proposed approach. Experiments on math and code generation tasks show consistent improvements across five LLM, including Gemma, Llama, Mistral, and DeepSeek sizes ranging from 1B to 8B.\n\n## update after rebattle \nI read through other reviews and responses. I think the paper has merits and leaning towards acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comments on \"Experimental Designs Or Analyses\":**  Authors test their approach on math and code reasoning tasks. It would be interesting to see how this approach will work on other types of reasoning tasks (for ex, some popular benchmarks include MMLU, GPQA, BIG-Bench Hard, etc).\n\n**Response:** We perform BRiTE on the model *Qwen/Qwen2.5-7B* with a mixed dataset (the size is around 40K) of *RUC-AIBOX/STILL-3-Preview-RL-Data*, MATH, and historical AIME problems (excluding AIME 2024).  We evaluate models trained using RS baseline and BRiTE on a diverse set of reasoning benchmarks, including Math-500, Minerva Math, Olympiad Bench, AIME24, AMC23, and GPQA Diamond. The pass@1 accuracy (64 sampling times for AIME24 and AMC23 and 8 sampling times for the others) for each benchmark is reported in the table below. \n|    Method           | Math500 | Minerva_math | Olympiadbench | Aime24 | Amc23 | Gpqa_diamond |\n|----------------|---------|--------------|---------------|-----------|----------|--------------|\n| Base Model          | 44.1    | 12.9         | 16.1          | 0.9       | 10.1     | 25.9         |\n| Reject Sampling| 54.3    | 21.0         | 23.1          | 5.6       | 31.6     | 26.9         |\n|BRiTE| **76.9**    | **40.6**         | **37.0**          | **14.4**      | **57.1**     | **29.8**        |\n\nEvaluation results show that BRiTE can surpass the RS baseline with a stable margin on these bechmarks, which demonstrates the power of BRiTE in the face of **large datasets (40k data)** and boosts the performance of the state-of-art model such as *Qwen/Qwen2.5-7B*. \n\n**Weakness on Training Cost:** Proposed approach for BRiTE-SFT involves two stages of training, while traditional SFT methods authors compare with involve 2 stage of training. Authors to not provide any theoretical or actual cost estimates required to train with BRiTE. Given this process is two-staged, I assume it would use more GPU days and - depending on how much more - might not be directly comparable with direct SFT. Please include theoretical or actual cost estimates required to train with BRiTE-SFT, and add comparison with traditional SFT you used for Table 1.\n\n**Response:** Thank you for your question. We will discuss data efficiency and computational efficiency as follows:\n- *Data Efficiency:* (i) We want to emphasize that our algorithm only requires data without human annotation, making it more efficient than supervised fine-tuning (SFT) that relies on annotated data, which can be harder to collect; (ii) compared to other methods, such as rejection sampling, which also do not require data for the thinking process, our algorithm demonstrates greater efficiency since we have controlled for the same amount of data in all our experiments.\n- *Computational Efficiency:* As mentioned earlier, our algorithm does not need data with a thinking process, so it is not fair to compare our algorithm (or other methods like rejection sampling) with SFT, which relies on human-annotated, step-by-step thinking data. Moreover, both our algorithm and rejection sampling (RS) share the same goal: generating a high-quality thinking process (through RL or RS) and then fine-tuning LLMs based on that. The generation process is the main contributor to computational costs, a factor that is unavoidable for all these methods. Specifically, the first stage of our method (rollout + RL updating) and the RS filtering process (rollout + filtering) both require approximately 30 hours. This main cost is attributed to the rollout of the Qwen2.5-7B model, generating outputs for 40.6K data points with 8 samples per point. Since RL provides more concise/shorter responses, the overall computational time (rollout + RL updating) is comparable to the RS filtering process. Conversely, the training phase (SFT for RS or $\\theta$-updating in BRiTE) is significantly less demanding, requiring only 3.3 hours for 9K data points from RS, or 9 hours for 24K data points from RL, for four epochs. our method achieves significantly better performance without substantially increasing computational costs.\n \n \n**Additional Question:** How do you select the best and worse response based on the correctness of y? Given 30 CoT, k will have correct answer, and (30-k) incorrect. How do you select best of k and worst of (30-k)? If it is not mentioned in the paper, please add elaboration.\n\n**Response:** We randomly select one correct answer $A^+$ from the $k$ available correct answers and one incorrect $A^-$ answer from the $(30-k)$ incorrect answers. The pairs $\\{(A^+, A^-)\\}$ are then used to create a pair for DPO training. We will add this clarification in the revision.", "claims_and_evidence": "Yes. Authors propose a new method to enhance reasoning abilities of the models, provide extensive theoretical investigations and compare performance of five models on four tasks showcasing consistent improvements.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I checked the correctness of proofs and theoretical claims, however I'm not completely confident in my evaluations.", "experimental_designs_or_analyses": "Yes. I think authors included solid variety of the models to show the generalization ability of their approach. However, I would highlight a few issues:\n1. Authors test their approach on math and code reasoning tasks. It would be interesting to see how this approach will work on other types of reasoning tasks (for ex, some popular benchmarks include MMLU, GPQA, BIG-Bench Hard, etc).\n2. Proposed approach for BRiTE-SFT involves two stages of training, while traditional SFT methods authors compare with involve 2 stage of training. Authors to not provide any theoretical or actual cost estimates required to train with BRiTE. Given this process is two-staged, I assume it would use more GPU days and - depending on how much more - might not be directly comparable with direct SFT.", "relation_to_broader_scientific_literature": "This work further advances popular RL-based methods used for LLM post-training.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This work is novel and provides extensive theoretical and practical details. In my opinion, the main weakness of the paper lies in its experimental part. Proposed approach for BRiTE-SFT involves two stages of training, while traditional SFT methods authors compare with involve 2 stage of training. Authors to not provide any theoretical or actual cost estimates required to train with BRiTE. Given this process is two-staged, I assume it would use more GPU days and - depending on how much more - might not be directly comparable with direct SFT.", "other_comments_or_suggestions": "Page 3, line 113: Initially, an initial state  -> An initial state \nPage 3, line 126: .. l framework fro LLM reason and unified ... - not sure what you mean here", "questions_for_authors": "1. p. 7 line 383: How do you select best and worse response based on correctness of y? Given 30 CoT, k will have correct answer, and (30-k) incorrect. How do you select best of k and worst of (30-k)? If it is not mentioned in the paper, please add elaboration.\n\n2. Please include theoretical or actual cost estimates required to train with BRiTE-SFT, and add comparison with traditional SFT you used for Table 1.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **summary**:  
This paper introduces BRiTE, a framework for reasoning process generation that formalizes latent rationales using a probabilistic model and optimizes them through reinforcement learning without needing labeled chain-of-thought data. The method is theoretically grounded and shows consistent, albeit modest, improvements across math and code reasoning tasks with various language models.

**Pros**  
1. Proposes a novel and theoretically sound framework incorporating latent rationales and evaluation signals.  
2. Demonstrates consistent improvements across multiple models on math and code generation tasks.  
3. Does not require human-annotated rationales, making it more scalable for training.

**Cons**  
1. Experimental scope is limited to math and code tasks, with no evaluation on broader reasoning benchmarks.  
2. Performance gains over strong baselines are relatively small.  
3. BRiTE involves a multi-stage training process that may introduce significant computational overhead without clear cost analysis.
Decision: Accept (poster)