Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding

## Abstract

Abstract Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. While prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at inference time, optimizing reasoning capabilities during training remains challenging. We introduce LaT ent R easoning O ptimization ( LaTRO ), a principled framework that formulates reasoning as sampling from a latent distribution and optimizes it via variational approaches. LaTRO enables LLMs to concurrently improve both their reasoning process and ability to evaluate reasoning quality, without requiring external feedback or reward models. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of 12.5% over base models and 9.6% over supervised fine-tuning across Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that pre-trained LLMs possess latent reasoning capabilities that can be unlocked and enhanced through our proposed optimization approach in a self-improvement manner. The code of LaTRO is available at https://github.com/SalesforceAIResearch/LaTRO .

## 1 Introduction

The development of large language models (LLMs) with enhanced reasoning capabilities has emerged as a crucial area of research. Despite their impressive advances, the inherent next-token prediction mechanism of LLMs makes it challenging for these models to solve complex problems requiring multiple reasoning steps (Wang et al., ; Huang et al., ). For instance, LLMs often struggle to directly provide accurate solutions to mathematical problems or even simple puzzles like counting specific letters in a word.
Consequently, researchers have explored various prompting strategies that guide LLMs to generate reasoning trajectories or rationales—sequences of tokens that build a step-by-step progression toward an answer. Techniques such as Chain-of-Thought (CoT) (Wei et al., ), Tree-of-Thought (ToT) (Yao et al., ), and Program-of-Thought (PoT) (Chen et al., ) prompting methods exemplify this approach.

Figure: Figure 1: Overview of LaTRO with an example question from GSM8K (Cobbe et al., ). LaTRO treats reasoning trajectories as latent variables and optimizes the underlying distribution through self-rewarding. Given a question, the language model generates multiple reasoning rationales, evaluates their likelihood of producing the correct answer, and updates its parameters to favor high-quality rationales. This iterative process allows the model to improve both its ability to generate good reasoning paths and to evaluate the quality of those paths.

Recent progress has also focused on inference-time techniques to enhance the reasoning abilities of LLMs (Wu et al., ; Brown et al., ), as observed in the OpenAI o1 model (OpenAI, ). These methods have demonstrated remarkable performance in diverse reasoning tasks, including mathematics (Cobbe et al., ; Trinh et al., ; Luo et al., ), coding (Jimenez et al., ; Guo et al., ; Zhang et al., ), and scientific problem-solving (Rein et al., ). Notable inference-time methods, such as CoT with Self-Consistency (CoT-SC) (Wang et al., ) and CoT-Decoding (Wang & Zhou, ), extend the CoT approach by generating multiple reasoning paths and selecting the most consistent one. Additionally, techniques like ReAct (Yao et al., ) and Reflexion (Shinn et al., ) integrate reasoning into LLM agent loops, further enhancing their problem-solving capabilities.

Despite the promising results at inference time, improving the reasoning abilities of LLMs during their training phase remains a challenging problem. Several obstacles impede progress in this area. Firstly, there is a scarcity of high-quality reasoning data for complex problems, limiting the applicability of traditional supervised fine-tuning (SFT) approaches (Zelikman et al., ). Moreover, when such data is available, SFT on deterministic reasoning paths may result in a lack of diversity in problem-solving strategies, potentially causing over-confidence issues and performance degradation (Cobbe et al., ), especially in domains needing multiple valid approaches, such as mathematical proofs and coding.
Alternatively, improving reasoning through reinforcement learning from human feedback (RLHF) presents its own challenges (Havrilla et al., ; Luo et al., ). Developing a reward model that accurately evaluates the quality and validity of reasoning paths is a formidable task, susceptible to distribution shifts and biased evaluations.

Self-improvement approaches like STaR (Self-Taught Reasoner) (Zelikman et al., ) and Quiet-STaR (Zelikman et al., ) have shown promise in enhancing language models’ reasoning capabilities without external feedback.
However, STaR relies on task-specific few-shot examples to bootstrap its reasoning process, which can limit its generalizability across diverse tasks. While Quiet-STaR attempts to overcome this by inferring implicit rationales across arbitrary text, it does not directly optimize the reasoning process itself. Through these findings, we observe that pretrained LLMs already possess innate reasoning capabilities but just have not been fully activated or utilized, inspiring us to propose our approach.

Our proposed method, LaTent Reasoning Optimization (LaTRO), addresses the limitations of previous approaches by formulating reasoning as sampling from a latent distribution and optimizing it through a principled variational framework. As illustrated in Fig. 1, LaTRO enables language models to concurrently improve both their reasoning process and ability to evaluate reasoning quality, without requiring task-specific few-shot examples or external reward models.
Key contributions of LaTRO include:

- 1.
A theoretical formulation connecting LLM reasoning optimization to latent variable models;
- 2.
A self-rewarding mechanism leveraging the model’s own probability estimates;
- 3.
Significant performance gains across multiple model architectures and reasoning tasks, demonstrating LaTRO’s effectiveness in unlocking latent reasoning capabilities of language models.

Our findings suggest that pre-trained LLMs are not only capable reasoners but also possess the potential to act as explicit reward models for evaluating reasoning paths. We term this approach of utilizing explicit reward functions induced by LLMs themselves as "self-rewarding." Empirically, LaTRO outperforms both baseline models and supervised fine-tuning approaches on reasoning tasks like GSM8K, while also demonstrating the capacity to compress reasoning processes and shift computational burdens from inference to training time.

## 2 Related work

#### Prompt-based LLM Reasoning

Prompt-based reasoning methods prove to be effective across various domains,
such as math problem-solving (Polu & Sutskever, ; Hendrycks et al., ; Cobbe et al., ), logical reasoning (Sprague et al., ) and agentic tasks (Yao et al., ; Shinn et al., ; Yao et al., ). Chain-of-Thoughts or CoT (Wei et al., ) is the pioneering work that prompts LLMs to decompose challenging tasks into smaller reasoning steps. After that, two primary research directions further improved reasoning capabilities during inference. One direction searched over the reasoning trajectories against a process-based verifier, or reward model (Yao et al., ; Besta et al., ; Lightman et al., ). For example, tree-of-thoughts (Yao et al., ) explored over thoughts by depth-first search (DFS), breadth-first search (BFS) or beam search. The other approach used a critic model to provide verbal feedback, iteratively refining the responses with that feedback (Saunders et al., ; Shinn et al., ; Yao et al., ; Madaan et al., ).

#### Self-Rewarding for LLM Reasoning

Reasoning capabilities in LLMs can be enhanced in post-training through self-rewarding and reinforcement learning. The Self-Taught Reasoner, or STaR (Zelikman et al., ) introduced a bootstrapping technique that allows LLMs to generate rationales and fine-tune itself with self-generated reasoning paths. Quiet-STaR (Zelikman et al., ) extended this by training LLMs to infer implicit rationales across arbitrary text, enhancing both reasoning and predictive abilities without task-specific fine-tuning. Reinforced Fine-Tuning, or ReFT (Trung et al., ) took this further by leveraging reinforcement learning to improve generalization in reasoning tasks like math problem-solving, enabling LLMs to learn from multiple reasoning paths. Self-correction capabilities in LLMs can also be reinforced through self-generated data (Kumar et al., ). Lastly, Hoffman et al. (); Hu et al. () formulated the reasoning process as latent variable models, aligning LLMs towards more accurate reasoning with fewer annotated data.

## 3 Background anMotivulation

We start by briefly introducing reasoning techniques (e.g., chain-of-thought (Wei et al., ), ReAct (Yao et al., ), etc).
Given a user query $\boldsymbol{x}$, the standard procedure to sample the response $\boldsymbol{y}$ is to leverage an autoregressive pretrained LLMs $\pi_{\theta}$ (parameterized by $\theta$): $\boldsymbol{y}\sim\pi_{\theta}(\cdot~{}|~{}\boldsymbol{x})$.
As for prompt-based reasoning techniques such as chain-of-thought (Wei et al., ),
the LLM $\pi_{\theta}$ is firstly asked to generate thoughts (a.k.a reasoning rationale) before generating the answers to the response:

$$ $\displaystyle\boldsymbol{x}^{\prime}:=\texttt{Reason}(\boldsymbol{x})$ $\displaystyle=\boldsymbol{x}\oplus\texttt{Prompt}~{}\texttt{Template}~{} \texttt{of}~{}\texttt{Thought},$ $\displaystyle~{}~{}~{}\boldsymbol{z}\sim\pi_{\theta}$ $\displaystyle(\cdot~{}|~{}\boldsymbol{x}^{\prime}),~{}~{}~{}~{}\boldsymbol{y} \sim\pi_{\theta}(\cdot~{}|~{}\boldsymbol{x}^{\prime}\oplus\boldsymbol{z})\,,$ $$

where $\boldsymbol{z}$ is the thought or the reasoning rationale path, $\oplus$ indicates the concatenate operator, and the prompt template of the thought can be some hint prompt such as “Let’s think s
tep by step” (^1^11We omit the difference between $x^{\prime}$ and $x$ for convenience in the latter notation.).Empirically, people observe that there is a higher chance for the LLM $\pi_{\theta}$ to generate the desired answer $\boldsymbol{y}$ following the above procedure than directly sampling the response $\boldsymbol{y}\sim\pi_{\theta}(\cdot~{}|~{}\boldsymbol{x})$.
From a statistical perspective, we hypothesize that good reasoning rationales can significantly improve the probability of generating good answers $\boldsymbol{y}$: $~{}\exists~{}\boldsymbol{z},~{}~{}\textit{s.t.}~{}\pi_{\theta}(\boldsymbol{y}|
~{}\boldsymbol{x}\oplus\boldsymbol{z})\gg\pi_{\theta}(\boldsymbol{y}|~{}
\boldsymbol{x})$.

Figure: Figure 2: Average negative log probabilities of LLMs to generate correct responses.
Refer to caption: x1.png

To validate the hypothesis, we check the probability of the correct answers $\boldsymbol{y}$ on pretrained LLMs with or without reasoning rationales.
In Figure 2,
we visualize the negative log probability of the correct answers on three different LLMs on GSM8K dataset (Cobbe et al., ).
We have observed that when the LLMs are conditioned on the reasoning rationales, the probability of the correct answer is remarkably larger than without reasoning rationales.
This suggests that good reasoning rationales help the LLMs generate desired answers for complex tasks by increasing their probability, giving them a higher chance of generating the correct answers.

The above observation inspires us that we can potentially further improve the reasoning ability of existing LLMs.
One may find some surrogate objective to enhance the quality of the reasoning rationales or improve the ability of LLMs to leverage good reasoning rationales.
In the following Proposition 1,
we show that Self-Consistency Chain-of-Thought (CoT-SC) (Wang et al., ), which takes a majority vote of multiple reasoning rationales to improve the reasoning ability,
approximates some surrogate objective.

###### Proposition 1 .

Denote the user query, model response, and reasoning rationale by $\boldsymbol{x},\boldsymbol{y},\boldsymbol{z}$, respectively.
The distribution of the majority vote answer of the $K$ reasoning rationales obtained by CoT-SC approximates $p_{M}(\boldsymbol{y}|\boldsymbol{x}):=\mathbb{E}_{\boldsymbol{z}\sim
\operatorname{\pi_{\theta}}(\cdot|~{}\boldsymbol{x})}[\operatorname{\pi_{
\theta}}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})]$, as $K\rightarrow\infty$.

###### Proof.

Given a user query $\boldsymbol{x}$, CoT-SC essentially follows the procedure: 1) Sample $K$ i.i.d reasoning rationales together with model responses:
$(\boldsymbol{z}_{k},\boldsymbol{y}_{k})\sim\pi_{\theta}(\cdot|\boldsymbol{x}),
~{}1\leq k\leq K$.
2) Take the majority vote of $(\boldsymbol{y}_{1},\ldots,\boldsymbol{y}_{K})$.
For a specific response $\boldsymbol{y}$, its frequency can be calculated as $F(\boldsymbol{y}):=\frac{1}{K}\sum^{K}_{k=1}\mathbbm{1}\{\boldsymbol{y}_{k}=
\boldsymbol{y}\}$, where $\mathbbm{1}$ is the indicator function. Then the expectation of $F(\boldsymbol{y})$ is

$$ $\begin{split}\mathbb{E}_{\boldsymbol{y}_{1},\ldots,\boldsymbol{y}_{K}}F( \boldsymbol{y})&=\frac{1}{K}\sum^{K}_{k=1}\mathbb{E}_{\boldsymbol{y}_{i}} \mathbbm{1}\{\boldsymbol{y}_{i}=y\}=\frac{1}{K}\sum^{K}_{i=1}\mathbb{P}_{ \boldsymbol{y}_{i}\sim\operatorname{\pi_{\theta}}(\cdot|\boldsymbol{x}\oplus \boldsymbol{z}_{i})}[\boldsymbol{y}_{i}=\boldsymbol{y}]\\ &=\frac{1}{K}\sum^{K}_{i=1}\operatorname{\pi_{\theta}}(\boldsymbol{y}| \boldsymbol{x}\oplus\boldsymbol{z}_{i})\overset{K\rightarrow\infty}{ \longrightarrow}\mathbb{E}_{\boldsymbol{z}\sim\operatorname{\pi_{\theta}}( \cdot|\boldsymbol{x})}\operatorname{\pi_{\theta}}(\boldsymbol{y}|\boldsymbol{x }\oplus\boldsymbol{z}).\end{split}$ $$

∎

CoT-SC essentially leverages $p_{M}(\boldsymbol{y}|\boldsymbol{x}):=\int\operatorname{\pi_{\theta}}(
\boldsymbol{z}|\boldsymbol{x})\operatorname{\pi_{\theta}}(\cdot|~{}\boldsymbol
{x}\oplus\boldsymbol{z})d\boldsymbol{z}$
to obtain reasoning rationales and produce final correct answers.
Inspired by the conclusion,
we could leverage surrogate objectives like $\mathbb{E}_{\boldsymbol{z}\sim\operatorname{\pi_{\theta}}(\cdot|~{}\boldsymbol
{x})}[\phi(\operatorname{\pi_{\theta}}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus
\boldsymbol{z}))]$ to further enhance the reasoning ability of LLMs, where $\phi$ is some monotonic transform such as logarithm ($\log(\cdot)$).
Further, we could also optimize the parameters of LLMs to enhance the reasoning abilities of existing LLMs during the training, so we can obtain LLMs with better reasoning abilities with the same inference time budget.
In the following sections, we introduce the idea of optimizing LLMs to improve reasoning abilities without external feedback, by proposing a principled framework containing the surrogate objective.

###### Proposition 1 .

###### Proof.

## 4 Optimizing the reasoning process

In this section, we describe how to optimize the reasoning rationale without external feedback. Specifically,
we introduce the objective for optimizing the reasoning rationale in Section 4.1 from a variational perspective of LLM training;
we derive the gradient estimation for the new objective in Section 4.2, and discuss the sampling procedure together with reward shaping in Section 4.3.
We summarize the proposed algorithm, LaTent Reasoning Optimization (LaTRO) in Algorithm 1, and illustrate the overall procedure in Figure 1.

### 4.1 Latent Reasoning Optimization: A Variational Approach

Suppose we are given a golden dataset $\mathcal{D}_{\text{Gold}}:=\{(\boldsymbol{x}_{i},\boldsymbol{y}_{i})\}_{i=1}^{N}$ consisting of $N$ query and answer pairs,
where $(\boldsymbol{x},\boldsymbol{y})$ denotes the query and the answer respectively.
A standard finetuning procedure to fit the LLM $\pi_{\theta}$ to the dataset $\mathcal{D}_{\text{Gold}}$ is by likelihood maximization:

$$ $\displaystyle\max_{\theta}\mathbb{E}_{(\boldsymbol{x},\boldsymbol{y})\sim \mathcal{D}_{\text{Gold}}}\left[\log\pi_{\theta}(\boldsymbol{y}~{}|~{} \boldsymbol{x})\right]\,,$ (1) $$

where $\theta$ are the parameters of the LLM $\pi_{\theta}$ to optimize.
Based on the discussion in Section 3, it is more feasible to optimize $\pi_{\theta}$ with additional reasoning rationale path $\boldsymbol{z}$,
compared with standard finetuning objective in Equation 1.
Hence, we can introduce another “reasoner” $q(\boldsymbol{z}|\boldsymbol{x})$ to sample the latent reasoning rationales that can help the optimization procedure of $\pi_{\theta}$.
This is achievable by optimizing the following lower bound:

$$ $\displaystyle\log\pi_{\theta}(\boldsymbol{y}|\boldsymbol{x})$ $\displaystyle=\log\int\pi_{\theta}(\boldsymbol{y}~{}|~{}\boldsymbol{x}\oplus \boldsymbol{z})\pi_{0}(\boldsymbol{z}~{}|~{}\boldsymbol{x})d\boldsymbol{z}$ $\displaystyle=\log\int\pi_{\theta}(\boldsymbol{y}~{}|~{}\boldsymbol{x}\oplus \boldsymbol{z})\frac{q(\boldsymbol{z}|\boldsymbol{x})}{q(\boldsymbol{z}| \boldsymbol{x})}\pi_{0}(\boldsymbol{z}|\boldsymbol{x})d\boldsymbol{z}$ $\displaystyle\geq\max_{q(\boldsymbol{z}|\boldsymbol{x})}\mathbb{E}_{q( \boldsymbol{z}|\boldsymbol{x})}\big{[}\log\pi_{\theta}(\boldsymbol{y}|~{} \boldsymbol{x}\oplus\boldsymbol{z})\big{]}-D_{\mathrm{KL}}[q(\boldsymbol{z}| \boldsymbol{x})||\pi_{0}(\boldsymbol{z}|\boldsymbol{x})]\,,$ (2) $$

where $\pi_{0}$ is a prior reference LLM that regularizes the “reasoner” $q(\boldsymbol{z}|\boldsymbol{x})$, and the lower bound is achieved via Jensen’s inequality (Higgins et al., ).
Based on the literature of variational Bayes (Kingma, ), one can either learn and optimize $q(\boldsymbol{z}|\boldsymbol{x})$ via variational Expectation Maximization (EM) (Abdolmaleki et al., ; Liu et al., ),
or introduce another parameterized LLM $q_{\phi}(\boldsymbol{z}|\boldsymbol{x})$ and optimize $\phi$ to amortize the cost.
Additionally, from the discussion in Section 3, we know $\pi_{\theta}$ itself can also serve as a naive “reasoner”, since $\pi_{\theta}$ is an autoregressive LLM.

To simplify the learning procedure, we propose to use $\pi_{\theta}$ as the “reasoner” $q(\boldsymbol{z}|\boldsymbol{x})$. As a result,
we can jointly learn one single LLM $\pi_{\theta}$, that is capable of generating good reasoning rationale together with providing correct answers given the query and its own generated reasoning rationale. To be more specific, we can define the learning objective as follows:

$$ $\displaystyle\max_{\theta}J(\theta):=\mathbb{E}_{(\boldsymbol{x},\boldsymbol{y })\sim\mathcal{D}_{\text{Gold}}}\bigg{[}\mathbb{E}_{\boldsymbol{z}\sim\pi_{ \theta}(\cdot|\boldsymbol{x})}\big{[}\underbrace{\log\pi_{\theta}(\boldsymbol{ y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})}_{R_{\theta}(\boldsymbol{z}, \boldsymbol{y},\boldsymbol{x})}\big{]}-D_{\mathrm{KL}}[\pi_{\theta}( \boldsymbol{z}|\boldsymbol{x})||\pi_{0}(\boldsymbol{z}|\boldsymbol{x})]\bigg{] }\,,$ (3) $$

where we specify the reference LLM $\pi_{0}$ to be the original $\pi_{\theta}$ before the optimization.
Furthermore, $\log\pi_{\theta}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})$ in Equation 3 can be viewed as the reward function $R_{\theta}(\boldsymbol{z},\boldsymbol{y},\boldsymbol{x})$ to evaluate the quality of the rationale $\boldsymbol{z}$ given the pair $(\boldsymbol{x},\boldsymbol{y})$, since the reasoning rationale $\boldsymbol{z}$ with higher likelihood $\log\pi_{\theta}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})$ indicates that it would provide a higher probability for the model to answer the question correctly.

Remark    By substituting $\log\pi_{\theta}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})$ with $R_{\theta}(\boldsymbol{z},\boldsymbol{y},\boldsymbol{x})$, Equation 3 exactly recovers the standard optimization objective defined in offline RL (Levine et al., ), RLHF (Ouyang et al., ; Rafailov et al., ) literature.
Though Equation 3 unifies the learning procedure of the “reasoner” $\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})$ and the “reward” function $R_{\theta}(\boldsymbol{z},\boldsymbol{y},\boldsymbol{x}):=\log\pi_{\theta}(
\boldsymbol{y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})$, we can break down these two procedures to analyze them separately.
When we fix $R_{\theta}(\boldsymbol{z},\boldsymbol{y},\boldsymbol{x})$ and optimize the “reasoner” $\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})$,
the procedure can be interpreted as *self-improvement* learning, where we improve $\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})$ on self-generated synthetic reasoning rationale.
When we fix $\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})$ and optimize $R_{\theta}(\boldsymbol{z},\boldsymbol{y},\boldsymbol{x})$, the procedure can be interpreted as *self-reward* learning, where we learn the self-reward function $\log\pi_{\theta}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus\boldsymbol{z})$.
The procedure can also be considered finetuning optimization given the learned reasoning rationale and query.
Fortunately, we can naturally enjoy the benefits of these two self-learning procedures with the new reasoning finetuning objective.

### 4.2 Gradient estimation for LaTRO

From previous RL literature, we know that estimating $\nabla_{\theta}J(\theta)$ in Equation 3 involves the use of policy gradient methods,
which usually suffers from high variances with the naive REINFORCE estimators (Williams, ). Inspired by the recent work on policy gradient for LLMs (Ahmadian et al., ),
we also leverage the REINFORCE Leave-One-Out (RLOO) (Kool et al., ) to optimize the “reasoner” $\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})$,
where we can achieve lower variances of gradient estimation by sampling multiple rationales.
We summarize the empirical gradient estimation for solving LaTRO in Proposition 2.

###### Proposition 2 .

(LaTRO Gradient Estimation)
Suppose we are given a set of training data $\mathcal{D}_{\text{Gold}}:=\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\}_{i=1}^{N}$, we sample $K$ i.i.d reasoning rationales $\boldsymbol{z}_{1}^{(i)},\boldsymbol{z}_{2}^{(i)},\ldots,\boldsymbol{z}_{K}^{(
i)}\sim\pi_{\theta}(\cdot|\boldsymbol{x}_{i})$ for each query and answer pair $(\boldsymbol{x}_{i},\boldsymbol{y}_{i})$. The empirical gradient estimator for $\nabla_{\theta}J(\theta)$ is expressed as

$$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}\nabla_{\theta}\widehat{J}(\theta):=\frac {1}{NK}\sum_{i=1}^{N}\sum_{k=1}^{K}\bigg{(}\nabla_{\theta}\log\pi_{\theta}( \boldsymbol{z}_{k}^{(i)}~{}|~{}\boldsymbol{x}_{i})\cdot A_{k}^{(i)}+\nabla_{ \theta}\log\pi_{\theta}(\boldsymbol{y}_{i}~{}|~{}\boldsymbol{x}_{i}\oplus \boldsymbol{z}_{k}^{(i)})\bigg{)}\,,$ (4) $\displaystyle\text{with}~{}A_{k}^{(i)}=r(\boldsymbol{z}_{k}^{(i)})-\frac{1}{K- 1}\sum_{j\neq k}^{K}r(\boldsymbol{z}_{j}^{(i)})\,,r(\boldsymbol{z}_{k}^{(i)}): =\log\pi_{\theta}(\boldsymbol{y}_{i}~{}|~{}\boldsymbol{x}_{i}\oplus\boldsymbol {z}_{k}^{(i)})-\beta\log\frac{\pi_{\theta}(\boldsymbol{z}_{k}^{(i)}~{}|~{} \boldsymbol{x}_{i})}{\pi_{0}(\boldsymbol{z}_{k}^{(i)}~{}|~{}\boldsymbol{x}_{i} )}\,,$ $$

where $\beta\geq 0$ is the coefficient to control the KL penalty. The proof can be found in Section A.1.

The first gradient term in Equation 4 serves as policy gradient to improve
the ability of the LLM $\pi_{\theta}$ to generate high-quality reasoning rationales,
and $\log\pi_{\theta}(\boldsymbol{y}|\boldsymbol{x}\oplus\boldsymbol{z})$ can be viewed as the evaluator for reasoning rationale, which is further used to calculate the advantages.
The second gradient term in Equation 4,
which is the gradient of supervised finetuning loss, essentially helps the LLM $\pi_{\theta}$ to leverage the reasoning rationales to produce correct answers.

###### Proposition 2 .

### 4.3 Practical Considerations

To reduce computation overhead and better control the sampling of reasoning rationales during training, we limit their maximum token length to $L$. The rationale ends either at the EOS token or at the start of a predefined answer template (e.g., "The answer is"). We then use the truncated rationale $\boldsymbol{z}$, along with the query $\boldsymbol{x}$ and the answer $\boldsymbol{z}$, for further computation.

We also encourage the LLM to finish its reasoning process with $L$ tokens.
Inspired by the implementation of the RLOO trainer in the TRL library (von Werra et al., ) , we introduce a constant penalty for rationales truncated by the maximum token length $L$. This penalty encourages the generation of rationales that fit within the specified token limit.

Figure: Algorithm 1 LaTent Reasoning Optimization (LaTRO)

## 5 Experiments

### 5.1 Setup

We evaluate the performance of the proposed method across two datasets: a mathematical reasoning dataset (GSM8K, Cobbe et al. ()) and a logical reasoning dataset (ARC-Challenge, Talmor et al. ()). The sizes of the datasets are listed in Table 1.

**Table 1: Size of the datasets**
| Name | Training | Evaluation |
| --- | --- | --- |
| GSM8K | 7473 | 1319 |
| ARC-Challenge | 1119 | 1172 |

#### Training.

For each dataset, we fine-tune three base models: Phi-3.5-mini-instruct (Abdin et al., ), Mistral-7B-Instruct-v0.3 (Jiang et al., ), and Meta-Llama-3.1-8B-Instruct (Dubey et al., ), abbreviated as Phi-3.5, Mistral-7B, and Llama-3.1-8B, respectively. We provide two baseline comparisons: the base model and the supervised fine-tuned (SFT) model. For GSM8K, LaTRO fine-tuning excludes golden rationales from the solutions in the training set, while the SFT model is trained using golden rationales. For ARC-Challenge, as suggested in (Zheng et al., ), the model is trained to generate answers to the text of multiple-choice questions rather than selecting labels. Since no golden rationales are available for ARC-Challenge, the SFT model is trained to directly generate answers.

#### Evaluation.

For GSM8K, we evaluate all models with CoT prompting, and for ARC-Challenge, we evaluate the SFT baseline with direct answer generation, while the base model and the LaTRO fine-tuned model with CoT prompting. All evaluations are conducted with zero-shot prompts. We report both greedy decoding (GD) results and self-consistency (with temperature $T=1$) results. We choose a self-consistency sample size $k=8$ (maj@8) in Table 2 after observing that more than 8 samples did not bring further performance improvement (see Figure 3 (b) for details).

#### Implementation.

LaTRO is implemented on the high level as in Algorithm 1, with additional engineering techniques as discussed in section 4.3. LaTRO is implemented using the widely recognized transformers (Wolf et al., ) and TRL (von Werra et al., ) libraries, with PyTorch (Ansel et al., ) as backend. DeepSpeed ZeRO (Rasley et al., ) is used in stage 3, along with Flash Attention 2 (Dao et al., ) to enhance training efficiency. The models were trained on a machine
equipped with 8xH100 80GB GPUs, using bfloat16 precision.

#### Hyperparameters.

AdamW optimizer with a learning rate of $5\times 10^{-7}$, no warm-up steps, and a linear decay strategy is used. The Monte Carlo (MC) sample size $K=16$ and the batch size of the data loader $3$ are predetermined, resulting in an effective batch size of $48$. Gradient accumulation steps and training batch size are subsequently adjusted to prevent out-of-memory errors during training.
A temperature of $T=1$ is used for MC sampling, and a penalty factor $\gamma=2$ is applied for incomplete rationales. The KL penalty is set at $\beta=0.05$ for GSM8K and $0.25$ for ARC-Challenge. Except for the results presented in Section 5.3, the maximum generation length is maintained at $L=500$. We train all models up to six epochs for GSM8K, and 12 epochs for ARC-Challenge. The checkpoint with best test accuracy is chosen.

For the SFT baseline experiments, we use a batch size of $32$ and adjust the learning rate to ensure that the evaluation loss decreases and finally converges. All SFT baselines are trained for a maximum of 12 epochs. The checkpoint with the best test accuracy is selected.

In addition to the main quantitative results, we conduct ablation studies on two factors: 1. The maximum generation length $L$, where we study the effects of tuning $L$ in both training and inference times; 2. The self-consistency samples $k$, where we explore to what extent LaTRO can still benefit from inference-time scaling.

The main quantitative results, qualitative analysis of sample responses, and results of the ablation study are presented in Sections 5.2, 5.4, LABEL: and 5.3, respectively. Additional details on our prompt templates and more samples can be found in Appendices B and C.

### 5.2 Results

In this subsection, we present evaluation results that demonstrate how effectively LaTRO enhances the reasoning abilities of LLMs on downstream datasets. The detailed results are provided inTable 2.

For the GSM8K dataset, LaTRO fine-tuned models outperform all base models by up to $19.5\%$ (Mistral-7B, $47.8\%\rightarrow 67.3\%$) and show an average improvement of $12.5\%$ across the three models examined with greedy decoding. The greatest improvement margin is observed for Mistral-7B, while the smallest is seen for Llama-3.1-8B, consistent with our initial findings in Figure 2, where Mistral-7B exhibited the lowest log probability for directly answering questions and Llama-3.1-8B exhibited the highest. With self-consistency, the improvements are by up to $16.5\%$ (Phi-3.5, $74.0\%\rightarrow 90.5\%$) and the average improvement is $13.1\%$. Furthermore, LaTRO models demonstrate superior performance relative to SFT baselines, with an average improvement of $9.6\%$ for greedy decoding and $13.2\%$ for self-consistency. It is worth noting that for the SFT baseline of Llama-3.1-8B, overfitting on the test set is still observed after tuning the learning rate.

For ARC-Challenge, LaTRO fine-tuned models still outperform the baselines, though with a smaller margin. When using greedy decoding, the improvements over the base models are up to $1.6\%$ with an average increase of $1\%$. We see more increment with self-consistency, where the improvement margins are on average $2.4\%$. Comparing to SFT baslines, we find that all three models are very sensitive when fine-tuning to directly generate the answer of ARC-Challenge questions. They perform even inferior to the unoptimized base models. When using greedy decoding, the improvements of LaTRO fine-tuned models over the SFT baselines are on an average of $5.2\%$, and by up to $6\%$ (Llama-3.1-8B). In the case of self-consistency, LaTRO performs better than the base models by an average of $2.4\%$, and surpasses the SFT models by an average of $8.1\%$.
On the less surprising results compared to GSM8K, we conjecture that for ARC-Challenge, the models are already good at producing the answer either directly or through CoT prompting. Hence, further optimization of the reasoning process did not yield significant improvement.

**Table 2: Zero-shot accuracy (%) comparison between LaTRO and the baselines on GSM8K and ARC-Challenge datasets. The models are fine-tuned on corresponding training datasets. The base model are marked with “N/A” in the training method. GD stands for greedy decoding at inference time and maj@8 stands for self-consistency with 8 samples. The models are evaluated by default using CoT, except that $\dagger$ indicates the direct answer generation is applied during evaluation.**
| Base Model | Training Method | Inference Method | GSM8K | ARC-Challenge |
| --- | --- | --- | --- | --- |
| Phi-3.5 | N/A | GD | 72.9 | 85.1 |
| maj@8 | 74.0 | 86.0 |  |  |
| SFT | GD | 75.8 | 81.0^$\dagger$ |  |
| maj@8 | 77.1 | 80.5^$\dagger$ |  |  |
| LaTRO | GD | 87.6 | 86.4 |  |
| maj@8 | 90.5 | 87.5 |  |  |
| Mistral-7B | N/A | GD | 47.8 | 74.1 |
| maj@8 | 58.2 | 74.1 |  |  |
| SFT | GD | 57.2 | 70.0^$\dagger$ |  |
| maj@8 | 59.9 | 70.6^$\dagger$ |  |  |
| LaTRO | GD | 67.3 | 74.3 |  |
| maj@8 | 73.8 | 78.9 |  |  |
| Llama-3.1-8B | N/A | GD | 76.8 | 81.4 |
|  | maj@8 | 79.7 | 84.4 |  |
| SFT | GD | 73.2 | 77.0^$\dagger$ |  |
| maj@8 | 74.7 | 76.4^$\dagger$ |  |  |
| LaTRO | GD | 80.1 | 83.0 |  |
| maj@8 | 87.0 | 85.3 |  |  |

Figure: Figure 3: Ablation study results on GSM8K with base model Phi-3.5. In (a), the $x$-axis represents various maximum token length $L$ of reasoning rationales, $y$-axis is the accuracy, and the plot shows the zero-shot performance v.s. various maximum token lengths for different methods. In (b), the $x$-axis represents the # of sampled reasoning rationales, the $y$-axis is the accuracy, and the plot shows the zero-shot performance v.s. the # of reasoning rationales used in the majority vote.
Refer to caption: x2.png

### 5.3 Ablation Study

In this subsection, we present our ablation study on the effect of different parameters in LaTRO. For consistency, we fix the base model to Phi-3.5 and the dataset to GSM8K throughout the ablation experiments.

#### How many tokens are enough?

Liu et al. () demonstrated that when the input length is $n$, a transformer model with a hidden size of $O(\log n)$ can solve problems equivalent to Boolean circuits of size $m$, using $m$ CoT steps. However, the empirical determination of sufficient CoT tokens for optimal performance remains underexplored. In this section, we report zero-shot accuracy with generation length $L$ ranging from 200 to 1000 tokens at inference time. Additionally, a Phi-3.5 model is fine-tuned with $L=200$ for comparison. We distinguish two LaTRO fine-tuned models, referred to as LaTRO and $\text{LaTRO}_{200}$. As shown in Figure 3(a) accuracy gains plateau when $L\geq 500$, suggesting 500 tokens might suffice for grade school math problems. In contrast, limiting $L$ to 200 reduces accuracy, unless the model is trained accordingly. Interestingly, LaTRO significantly improves performance under this constraint by training the model to generate more concise rationales. This finding implies LaTRO can also train LLMs to condense rationales for limited inference-time compute. A sample from $\text{LaTRO}_{200}$ is provided in Appendix C for qualitative analysis.

#### Is inference-time scaling still beneficial?

While LaTRO shifts much of the reasoning computation to training time, it can still benefit from the inference-time scaling. To explore this, We evaluate the models again using self-consistency with the number of samples $k$ ranging from 1 to 32. As shown in Figure 3 (b), with $k=8$ rationales and majority voting, LaTRO achieves 90% accuracy on GSM8K. This indicates that LaTRO equipped with self-consistency can further enhance performance, even with LaTRO’s focus on training-time optimization.

Figure: Figure 4: Sample responses of a GSM8K question from Mistral-7B models, the errors are highlighted in red.

### 5.4 Case Study

We take a closer look at the responses generated by the LaTRO fine-tuned models.
We select a question from GSM8K and compare the responses from the base, the SFT model, and the LaTRO finetuned model.
We choose the set of responses from the Mistral-7B models that we evaluated. As can be seen in Figure 4, the base model not only generates a lengthy response, it also makes a logical mistake at the first step, where the correct equation to establish here is “F = 2B + 15”. The SFT model simplifies the answer and makes the first step correct. However, in the second step it first makes a wrong equation, then makes an arithmetic error when evaluating this equation. Further, LaTRO can give a concise and correct answer. We include more sample responses in Appendix C.

## 6 Conclusion

In conclusion, this work introduces LaTRO, a principled framework for optimizing language models’ reasoning capabilities without external feedback or reward models. By formulating reasoning as sampling from a latent distribution and leveraging self-rewarding, LaTRO enables models to concurrently improve both their reasoning process and ability to evaluate reasoning quality. Our extensive experiments across multiple model architectures and tasks demonstrate significant performance gains, with LaTRO outperforming baseline models and supervised fine-tuning approaches. These findings suggest that pre-trained LLMs possess latent reasoning capabilities that can be unlocked through our proposed optimization approach, representing a significant step towards creating more intelligent systems that can self-evolve their problem-solving capabilities.

While LaTRO shows promising results, there are some limitations to consider. The computational cost of sampling multiple rationales during training could be prohibitive for very large models. Future work could explore ways to reduce this computational overhead, such as using more efficient sampling techniques or adaptive rationale generation. Other promising directions include investigating the applicability of LaTRO to a wider range of reasoning tasks beyond math and science problems, and exploring how to
conduct multi-step reasoning learning to to enhance reasoning capabilities further. Despite these limitations, our contributions advance both the state-of-the-art in LLM reasoning capabilities and provide valuable insights into the nature of LLM alignment and its potential for self-improvement.

## Appendix A Additional Details on Our Theoretical Framework

### A.1 Proof of Proposition 2

###### Proof.

We restate the objective as follows:

$$ $\displaystyle J(\theta)$ $\displaystyle:=\mathbb{E}_{(\boldsymbol{x},\boldsymbol{y})\sim\mathcal{D}_{ \text{Gold}}}\bigg{[}\mathbb{E}_{\boldsymbol{z}\sim\pi_{\theta}(\cdot| \boldsymbol{x})}\big{[}\log\pi_{\theta}(\boldsymbol{y}|~{}\boldsymbol{x}\oplus \boldsymbol{z})\big{]}-\beta D_{\mathrm{KL}}[\pi_{\theta}(\boldsymbol{z}| \boldsymbol{x})||\pi_{0}(\boldsymbol{z}|\boldsymbol{x})]\bigg{]}\,,$ $\displaystyle=\mathbb{E}_{(\boldsymbol{x},\boldsymbol{y})\sim\mathcal{D}_{ \text{Gold}}}\big{[}\mathbb{E}_{\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})}[ \log\pi_{\theta}(\boldsymbol{y}|\boldsymbol{x}\oplus\boldsymbol{z})-\beta\log \pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})+\beta\log\pi_{0}(\boldsymbol{z}| \boldsymbol{x})]\,\big{]}\,,$ $$

where $\beta>0$ is a positive coefficient to control the regularization strength.
We take the gradient w.r.t $\theta$ at each sample pair $(\boldsymbol{x},\boldsymbol{y})$, and we get

$$ $\displaystyle\nabla_{\theta}J(\theta;\boldsymbol{x},\boldsymbol{y})$ $\displaystyle:=\nabla_{\theta}\int(\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x}) )(\log\pi_{\theta}(\boldsymbol{y}|\boldsymbol{x}\oplus\boldsymbol{z})-\beta \log\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})+\beta\log\pi_{0}(\boldsymbol{z }|\boldsymbol{x}))d\boldsymbol{z}$ $\displaystyle=\mathbb{E}_{\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})}\left[ \nabla_{\theta}\log\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})\left(\log\pi_{ \theta}(\boldsymbol{y}|\boldsymbol{x}\oplus\boldsymbol{z})-\beta\log\frac{\pi_ {\theta}(\boldsymbol{z}|\boldsymbol{x})}{\pi_{0}(\boldsymbol{z}|\boldsymbol{x} )}\right)\right]$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}+\mathbb{E}_{\pi_{\theta}(\boldsymbol{z}| \boldsymbol{x})}\left[\nabla_{\theta}\log\pi_{\theta}(\boldsymbol{y}| \boldsymbol{x}\oplus\boldsymbol{z})-\beta\nabla_{\theta}\log\pi_{\theta}( \boldsymbol{z}|\boldsymbol{x})\right]\,.$ $$

We further define $r(\boldsymbol{z}):=\log\pi_{\theta}(\boldsymbol{y}|\boldsymbol{x}\oplus
\boldsymbol{z})-\beta\log\frac{\pi_{\theta}(\boldsymbol{x}|\boldsymbol{z})}{
\pi_{0}(\boldsymbol{x}|\boldsymbol{z})}$, and use the fact that $\mathbb{E}_{\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})}[\nabla_{\theta}\log
\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})]=\int\pi_{\theta}(\boldsymbol{z}|
\boldsymbol{x})\frac{\nabla_{\theta}\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x}
)}{\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})}d\boldsymbol{z}=\nabla_{\theta}
\int\pi_{\theta}(\boldsymbol{z}|\boldsymbol{x})d\boldsymbol{z}=0$. we obtain the final gradient as

$$ $\displaystyle\nabla_{\theta}J(\theta;\boldsymbol{x},\boldsymbol{y})=\mathbb{E} _{\pi(\boldsymbol{z}|\boldsymbol{x})}\left[\nabla_{\theta}\log\pi_{\theta}( \boldsymbol{z}|\boldsymbol{x})\cdot r(\boldsymbol{z})+\nabla_{\theta}\log\pi_{ \theta}(\boldsymbol{y}|\boldsymbol{z},\boldsymbol{x})\right]\,.$ $$

And when we use RLOO estimator with empirical samples, we can replace above gradient estimation with empirical samples, which gives us the following result:

$$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}\nabla_{\theta}\widehat{J}(\theta):=\frac {1}{NK}\sum_{i=1}^{N}\sum_{k=1}^{K}\bigg{(}\nabla_{\theta}\log\pi_{\theta}( \boldsymbol{z}_{k}^{(i)}~{}|~{}\boldsymbol{x}_{i})\cdot A_{k}^{(i)}+\nabla_{ \theta}\log\pi_{\theta}(\boldsymbol{y}_{i}~{}|~{}\boldsymbol{x}_{i}\oplus \boldsymbol{z}_{k}^{(i)})\bigg{)}\,,$ $\displaystyle\text{with}~{}A_{k}^{(i)}=r(\boldsymbol{z}_{k}^{(i)})-\frac{1}{K- 1}\sum_{j\neq k}^{K}r(\boldsymbol{z}_{j}^{(i)})\,,r(\boldsymbol{z}_{k}^{(i)}): =\log\pi_{\theta}(\boldsymbol{y}_{i}~{}|~{}\boldsymbol{x}_{i}\oplus\boldsymbol {z}_{k}^{(i)})-\beta\log\frac{\pi_{\theta}(\boldsymbol{z}_{k}^{(i)}~{}|~{} \boldsymbol{x}_{i})}{\pi_{0}(\boldsymbol{z}_{k}^{(i)}~{}|~{}\boldsymbol{x}_{i} )}\,,$ $$

which completes the proof.
∎

###### Proof.

## Appendix B Prompt Templates

In this section, we list all prompt templates used in our experiments. For the GSM8K dataset, we used a minimum prompt template. For the ARC-Challenge dataset, we use a CoT template for the base model and LaTRO fine-tuning, and another template direct answer generation in SFT.

Figure: Figure 5: CoT template for GSM8K

Figure: Figure 6: CoT template for ARC-Challenge

Figure: Figure 7: Direct answer generation template for ARC-Challenge

## Appendix C Sample Responses

Figure: Figure 8: Sample responses of a GSM8K question, from Phi-3.5 models, maximum generation length $L=200$. The base model does not finish the generation, while the LaTRO model generates a short and correct answer.

Figure: Figure 9: Sample responses of an ARC-Challenge question, from Llama-3.1 models, the errors are highlighted in red. The base model shows knowledge about basic physics but makes a logical error on the order.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. Line 290: It is not clear about how the first gradient term in Eq. 4 would lead to optimising the LLM policy to generate higher quality rationales. Further elaboration is needed as to why generating/optimising the likelihood of input question/instruction conditioned on the rationale would lead to better reasoning.
2. Lines 414-415: Please support with examples about why improvements on ARC-Challenge are relatively lesser. The magnitude of improvements is not at all an issue, however, it should be better demonstrated that why better reasoning chains are not leading to higher improvements. Does this make ARC-Challenge ill-suited for this study since it involves limited reasoning scope?
3. Line 430: Why is it needed to generate rationales that are as long as 500 tokens? It would be good to show the usefulness of the information contained in the longer rationales and why does it lead to better performance before saturating.
soundness: 2
strengths: 1. Enhancing the reasoning ability of LLMs without relying on feedback from any external LLM during the training phase is an important research question.
2. Suitability of applying the variational objective is well motivated, explained and justified in Sections 3 and 4.1.
3. Accuracy gains over the base and the SFT version of the LLMs are shown with some ablation studies on greedy decoding vs. self consistency based sampling. Further, the authors show that inference time scaling of number of reasoning samples obtained using LaTRO can additionally enhance the accuracy.
summary: The work focuses on enhancing the reasoning abilities of large language models (LLMs) during the training phase without relying on external feedback. The authors motivate the work by raising an important question on improving the reasoning ability of LLMs during training phase since most prior works have focussed at achieving this at inference time. To do so, the authors propose to sample diverse reasoning rationales from latent distribution and optimize the LLM through a variational framework using the sampled rationales. The intuition behind application of the variational framework is well motivated through the objective of self-consistency based chain-of-thought prompting. Further, the work proposes to usethe  likelihood of generating correct answer conditioned on a rationale as a proxy to explicit reward models to optimise the LLM towards generating better reasoning chains. Results demonstrate that the proposed LaTRO method helps in improving the accuracy achieved for multiple LLMs such as Mistral-7B, Llama-3.1-8B etc. on GSM8K and ARC-Challenge datasets compared to the corresponding base and the SFT versions of the LLMs.
weaknesses: 1. Presentation of the introduction section of the paper needs improvement - It should better motivate the need behind reducing the reliance on external LLMs/feedback for improving a given LLM. Further, it should provide information about why the variational framework is suitable and elaborate some details about the proposed method in the introduction itself.
2. Discussion on related work is very limited. Even though the proposed method is an effort to reduce reliance on external LLMs, it should still discuss and contrast against existing methods that leverage external LLMs (for example, [1,2]) as well as employ self-play based techniques to improve an LLM by itself with requiring help from any other LLM (for example, [3, 4]). The example references are only indicative (but not exhaustive) of the type of citations that should be included.
3. Lack of appropriate baselines - Even though the work claims and focusses at improving the base/SFT version of the LLM through the proposed LaTRO framework, it should still compare against some of the existing self-play trainable methods (by adapting them for 7B LLMs) as baselines (eg. [3] - on datasets where the ground-truth rationale is available and [4]). Such methods improve the rationales generated by an LLM by exploring the generation space and discriminating better rationales from the inferior ones.
4. More reasoning datasets such as CSQA, Hellaswag etc. should be considered for evaluation studies.

[1] Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, Toronto, Canada. Association for Computational Linguistics.

[2] Zephyr: Direct Distillation of LM Alignment. Lewis Tunstall and Edward Emanuel Beeching and Nathan Lambert and Nazneen Rajani and Kashif Rasul and Younes Belkada and Shengyi Huang and Leandro Von Werra and Clementine Fourrier and Nathan Habib and Nathan Sarrazin and Omar Sanseviero and Alexander M Rush and Thomas Wolf. First Conference on Language Modeling, 2024.

[3] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. Zixiang Chen and Yihe Deng and Huizhuo Yuan and Kaixuan Ji and Quanquan Gu. ICML 2024.

[4] Self-Rewarding Language Models. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: see weakness
soundness: 3
strengths: - The writing is clear and easy to follow.
- The discussed topic and motivation are both innovative and significant.
summary: This paper proposesLaTRO, a novel framework aimed at enhancing the reasoning capabilities of LLMs. LaTRO addresses the challenge of improving reasoning during the training phase by formulating reasoning as sampling from a latent distribution and optimizing it through variational approaches. This method allows LLMs to self-improve their reasoning process and ability to evaluate the quality of reasoning without external feedback or reward models. The paper validates LaTRO's effectiveness through experiments on GSM8K and ARC-Challenge datasets, demonstrating significant improvements over base models and supervised fine-tuning approaches.
weaknesses: - Although I'm not familiar with the topic discussed in this article, I believe the experiments presented are too few and not comprehensive enough. Moreover, the datasets considered are limited to only GSM8K and ARC-Challenge, which lacks persuasiveness.
- The number of case study examples is too limited, with only a few instances in Figure 4 and the appendix, which is not convincing.
- The proposed method, LaTRO, especially the stability of the Self-reward component, has not been adequately considered.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. In Proposition 1, $p(y|x) = \int p(y|z,x) p(z|x) dz$ holds for any CoT-based method. Why, then, is CoT-SC introduced here?

2. What is the definition of "golden rationales", and why can’t ARC-Challenge have golden rationales?

3. What can the experimental results on ARC-Challenge demonstrate? The two baselines are too weak, as the SFT baseline did not utilize rationales.
soundness: 2
strengths: 1. The paper offers a novel perspective by framing reasoning as a process of sampling from a latent distribution and addressing it through variational methods.

2. The paper leverages the model's own probability estimates as an implicit reward, unifying the training of the policy and reward models.

3. The paper is well-organized and easy to follow.
summary: This paper introduces LaTRO, a novel approach that formulates Chain-of-Thought (CoT) reasoning as sampling from a latent distribution, optimized through variational techniques. By leveraging the probability of generating the correct answer as an implicit reward, LaTRO unifies the learning of both the policy and reward models, allowing large language models to refine reasoning paths in a self-rewarding manner. The authors demonstrate LaTRO’s effectiveness through experiments on the GSM8K and ARC-Challenge datasets across various model architectures. Their results indicate that latent reasoning capabilities within pre-trained language models can be unlocked and enhanced using this self-improvement framework.
weaknesses: 1. The experimental setup lacks sufficient strong baselines, which are essential for a robust evaluation. Two key baselines to consider are:
  - A stronger SFT Baseline: Fine-tuning the policy model with correct reasoning paths. Given the availability of ground truth answers, multiple reasoning paths could be sampled, retaining only those that align with the ground truth. This baseline would provide a more rigorous comparison for evaluating LaTRO’s effectiveness in reasoning.
  - DPO Baseline: The authors could further fine-tune the policy model using the DPO algorithm, incorporating both correct and incorrect reasoning paths. Actually, the DPO algorithm aligns closely with LaTRO in its approach, as both methods aim to avoid training an explicit reward model. Including DPO as a baseline would highlight LaTRO’s strengths relative to an approach that similarly leverages implicit reward mechanisms.

2. The experimental scope is limited, as only two datasets and small models were tested. 
 - Expanding the experiments to include a wider range of reasoning datasets would better assess the model's reasoning capabilities. Standard practice for evaluating reasoning in large language models includes diverse datasets that cover arithmetic reasoning (only GSM8K is not enough for arithmetic reasoning evaluation), commonsense reasoning, symbolic reasoning, and other reasoning types. Incorporating these would provide a more comprehensive evaluation.
 - Testing across varying model scales, especially with larger models, could provide insights into how the approach scales with model size and whether larger models yield better reasoning performance.

3. Although the authors claim that training did not rely on external feedback, the ground truth answers effectively serve as a form of implicit external feedback or reward.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. About self-evaluation task in LaTRO: how does LaTRO evaluate the probability of each reasoning path producing the correct answer? What does the conditional probability mention in Section 4.1 mean? Is there a task restriction for this evaluation method? What is the accuracy of self-evaluation? The authors did not discuss these questions in the paper, nor did they explore them in depth in the experiments.

2.  The authors only used a formula to explain the use of self-reward signals to achieve parameter updates, so what exactly does this update parameter refer to during the training phase? How is it implemented?  Suggest provide more detailed information.
soundness: 3
strengths: 1. LaTRO regards the reasoning process as sampling from a latent distribution and optimizes it using a variational method. This approach is different from prompt-based methods such as CoT and is closer to unsupervised learning. Besides, the feasibility of LaTRO is verified by mathematical proof.

2. This paper focuses on a very interesting topic, which enables an LLM to improve itself through a self-reward mechanism without external supervision and feedback signals.
summary: This paper introduces LaTRO, a framework that enhances LLM’s reasoning abilities by treating reasoning as sampling from a latent distribution and optimizing it with variational methods. LaTRO allows LLMs to improve their reasoning process and evaluation of reasoning quality simultaneously, without external feedback. Experiments on GSM8K and ARC-Challenge datasets demonstrate the effectiveness of LaTRO compared with the SFT training method.
weaknesses: 1. Request a step-by-step description or flowchart of the LaTRO pipeline. Although a large number of formulas are used in this paper to explain each part，but it lacks a detailed description of the proposed method, which makes it difficult for readers to understand the complete pipeline of the proposed method and reproduce it. 

2. As far as I know, there are some works to gradually enhance the capabilities of LLM (not limited to reasoning task), including prompt-based [1][2][3] and training-based methods [4][5][6], some of which do not use any feedback information to enhance the capabilities of LLM [7][8]. The author should discuss the differences between this work and these works and compare the performance of these works. It is necessary to add a dedicated subsection in the Related Work section discussing these specific works and their methods. If possible, include performance comparisons with these methods in the experimental section.

[1] When can llms actually correct their own mistakes? A critical survey of self-correction of llms

[2] Learning From Mistakes Makes LLM Better Reasoner

[3] Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning

[4] REFINER: Reasoning Feedback on Intermediate Representations

[5] CRYSTAL: Introspective Reasoners Reinforced with Self-Feedback

[6] SELF: Language-driven Self-evolution for Large Language model

[7] Small language modes can self-correct

[8] Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models
 
3. In the experiments, the author only employs SFT training method and the base model as baselines, without comparing the performance with COT-SC mentioned in Section 3 and the self-improvement classic work mentioned above, making it difficult to demonstrate the advantages of the proposed LaTRO. So, please provide a more comprehensive analysis of how LaTRO performs relative to these additional baselines across different tasks and metrics.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: See "Weaknesses".
soundness: 2
strengths: 1. The structure of this paper is clear, and it's easy to follow the main idea and the contribution.
2. The research problem this paper targets is very important to the community.
3. The motivation is sound and the benefits of the approach are clear.
summary: This paper proposes a principled framework, LaTRO, to treat the reasoning process as sampling from latent distribution and enable LLMs themselves as reward models to evaluate the quality of reasoning rationales. The proposed LaTRO outperforms supervised fine-tuning baselines on 2 datasets.
weaknesses: 1. Although modeling the rationales as latent variables to sample is well defined and proposed, the paper lacks a discussion with previously proposed reasoning methods that formulate the reasoning process as latent variables [1, 2] as well. Even though they are cited in the related work. Specifically, [1] proposes to sample diverse reasoning rationales to improve the prediction performance and also proposes an EM-like algorithm to improve the reward model and LLM alternatively. It would be great to have a discussion and comparison with these methods.
2. The paper does not compare with various prompting-based reasoning baselines mentioned in the related work section, such as tree-of-thought[3], and RAP[4], as well as fine-tuning baselines such as STaR[5], which is a missed opportunity to demonstrate its effectiveness. It would be better to compare them with metrics like training / inference computational cost and accuracy.
3. The paper does not provide a confidence interval, leading to unawareness of how the proposed LaTRO is robust to the initialization and randomness. It would be great to report the results from at least 3 times repetitions.
4. As the proposed LaTRO is a general approach, it would be great to evaluate it on more benchmark datasets to verify its effectiveness, such as HumanEval[6] and MBPP [7], which are popular in the current LLM reasoning community.


[1] Hu, Edward J., et al. "Amortizing intractable inference in large language models." arXiv preprint arXiv:2310.04363 (2023).

[2] Hoffman, Matthew Douglas, et al. "Training chain-of-thought via latent-variable inference." Advances in Neural Information Processing Systems 36 (2024).

[3] Yao, Shunyu, et al. "Tree of thoughts: Deliberate problem solving with large language models." Advances in Neural Information Processing Systems 36 (2024).

[4] Hao, Shibo, et al. "Reasoning with language model is planning with world model." arXiv preprint arXiv:2305.14992 (2023).

[5] Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.

[6] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[7] Austin, Jacob, et al. "Program synthesis with large language models." arXiv preprint arXiv:2108.07732 (2021).

Meta-review: This paper aims to enhance the reasoning abilities of LLMs at the stage of training. It formulates the reasoning process as sampling from latent distribution and optimizes the LLM through a variational framework using the sampled rationales.  While the target problem is important and the method is well motivated, main concerns of the reviewers include:  1. discussion on related work is limited; 2. the experiment did not compare with strong baselines,  and only two datasets and small models were tested.    The authors did not provide formal rebuttal, and instead acknowledge that they "did not conduct enough experiments to demonstrate the effectiveness of our algorithm empirically". AC agrees and suggests the authors revise and improve the submission accordingly.

Decision: Reject