Latent Chain-of-Thought for Visual Reasoning
Authors: Guohao Sun
Hang Hua
Jian Wang
Jiebo Luo
Sohail Dianat
MAJID RABBANI
Raghuveer Rao
Zhiqiang Tao
Venue: NeurIPS 2025 poster ()
Keywords: vision-language model
reasoning
variational inference
Primary area: deep_learning

Abstract: Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs). However, existing training algorithms such as SFT, PPO, and GRPO may not generalize well across unseen reasoning tasks and heavily rely on a biased reward model. To address this challenge, we reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference. By leveraging diversity-seeking reinforcement learning algorithms, we introduce a novel sparse reward function for token-level learning signals that encourage diverse, high-likelihood latent CoT, overcoming deterministic sampling limitations and avoiding reward hacking. Additionally, we implement a Bayesian inference-scaling strategy that replaces costly Best-of-N and Beam Search with a marginal likelihood to efficiently rank optimal rationales and answers. We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on four reasoning benchmarks, in terms of effectiveness, generalization, and interpretability.

Latent Chain-of-Thought for Visual Reasoning
Guohao Sun!.2 Hang Hua?*>, Jian Wang"*, Jiebo Luo", Sohail Dianat', Majid Rabbani', Raghuveer Rao", and Zhiqiang Tao
Rochester Institute of Technology. SSnap Inc.,
University of Rochester, *DEVCOM Army Research Laboratory
Abstract
Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs). However, existing training algorithms such as SFT, PPO, and GRPO may not generalize well across unseen reasoning tasks and heavily rely this challenge, we reformulate reasoning in LVLMs as posterior inference and propose scalable training algorithm based amortized variational inference. By leveraging diversity-sccking reinforcement learning algorithms, we introduce a novel sparse reward function for token-level learning signals that encourage diverse, high-likelihood latent CoT, overcoming deterministic sampling limitations and avoiding reward hacking. Additionally, we implement Bayesian inference-seias scaling strategy that replaces costly Best-of-N and Beam Search with a marginal likelihood efficiently rank optimal rationales and answers. We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks, in terms of effectiveness, generalization, and interpretability. The code is available at http:://ggihhuu...o..eeeeeeeeeee com www.co///eoosss/lleeoooooo
biased reward model. To address
Introduction
Chain-of-thought (CoT) reasoning is critical for enhancing the interpretability and reliability of Large Vision- Lanion- Models (LVLMs) [1] 29 [5] [6]. These models combine visual perception and natural language processing to perform intricate reasoning tasks that require explicit, step-by-step rationalization. As LVLMs have expanded into more sophisticated applications, such as visual question answering, commonsense reasoning, and complex task execution, the limitations of current training methods, such as generalization, have become increasingly evident.
To enable visual CoT, mainstream training paradigms, such as Supervised Fine-Tuning (SFT), Proximal Policy Optimization (PPO) [34,, and Group Relative Policy Optimization (GRPO) [12] primarily focus optimizing next-token distributions scalar rewards. While effective for in- distribution tasks, these methods often struggle to generalize across diverse reasoning questions due to their inability to explicitly capture dependencies across trajectories [4] Specifically, SFT heavily depends on teacher-forced log-like lidelhod,,,,d, only to parrot reference traces; meanwhile, PPO and GRPO constrained in exploration their KL penalties enforce proximity to the SFT baseline, making them fall short in finding novel rationales. Additionally, they may cause a reward hacking [36] issue that achieves high scores without genuinely solving the intended problem. To address these limitations, this work adopts a latent variable model to realize visual CoT as probabilistic inference problem 6 over latent variables, allowing us work with rich, expressive probabilistic models that better capture uncertainty and hidden structure, without needing direct supervision.
Part of this work was conducted while Guohao Sun and Hang Hua were interns at Snap Inc 'Hang Hua and Jian Wang contributed equally. Corresponding authors: Guohao Sun (gs4288 @ @rittedu) and Zhiqiang Tao (zhiqiang tao @rit.edu)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Unlike using prompting and in-context learning to generate deterministic reasoning CoT (2 (2), we treat Z as a latent variable sampled from posterior ( P(XZY)/E P(YZ'Y, P(XZY), given a question-answer pair (X, Y) as observation. However, such sampling is intractable due to the normalization term. Existing methods to sample approximately from an intractable posterior include Markov chain Monte Carlo (MCMC) and RL approaches such as PPO (..] Despite good training efficiency, these methods show limited capacity in modeling the full diversity of the distribution [[4]. By contrast, Amortized Variational Inference (AVI) [53, 19 22 23 yields token-level learning through optimizing the Evidence Lower Bound (ELBO), which encourages diverse trajectories and provides a principled way to draw samples from the target posterior distribution (see Fig. DT. One way to implement AVI is given by the generative flow networks (GFlowNets [4] 5l) algorithm: training a neural network to approximate distribution of interest. Despite achieving strong performance in broad text reasoning tasks, prior Gelssweet- approaches [14] have yet to fully address visual reasoning due the long CoT sequence inherent in multimodal tasks(e.g.. 1 tokens).
In this study, we propose a novel reasoning model, namely LaCoT, which enables amortized latent CoT sampling in LVLMs and generalizes across various visual reasoning tasks. To achieve this, we propose a general RL training algorithm (RGFN) with a novel reference-guide policy exploration method to overcome the catastrophic forgetting issue and eliminate the diversity constraint caused by the KL penalty. To improve exploration efficiency, we introduce @ a token-level rewarr approximation method, allowing efficient mini-batch exploration for diverse sampling. Finally, we introduce O a Bayesian inference- scallng strategy (BiN) for optimal rationale- solutionn searching at inference time for any reasoning LVLM. Previous works have provided empirical evidence that Best-of-N (BoN) sampling (7), Beam Search [41), and other heuristic-drive approaches [47] can improve model's performance inference time. However, these methods computationally costly and rely heavily on biased critic models, failing to provide an optimal reasoning chain or answer efficiently. Our inference procedure is grounded in Bayesian sampling principles to eliminate the critic model and improve interpretability. We treat rationales as integration variables and rank answers by a principled, length-normalizee marginal likelihood. Consequently, our method delivers a scalable, probabilistically justified searching strategy, effectively identifying optimal rationales and answers within LVLMs.
Empirically, we develop the proposed LaCoT on two base models, Qwen2.5-VL 3 3B and 7B, where the 7B model achieves improvement of 6.6% over its base model and outperforms GRPO by 10.6%. The 3B model surpasses its base model with 13.9% and achieves better results than larger models, e.g., LLa 1B and -VV-7,, demonstrating the effectiveness of learning to sample latent CoT on reasoning benchmarks.
2 Preliminaries
Generative Flow Networks (GFlowNets) 5 are class of amortized variational inference methods designed to sample complex, structured objects such as sequences and graphs with prob- abilities proportional to a predefined, unnormalized reward function. Unlike traditional generative models that often focus on maximizing likelihood or expected reward, GFlowNets objective, such as Sub-Trajectory Balance (subTB) (27,, hierarchical variational objective [28]. Such that the model is capable of expressing any action distribution and the objective function is globally minimized, then the flow consistency for trajectory T = (z+ ....)))) is
F(2))
II PF(# I 22-1) = F(2)) II PB(=k-1 22) Ami+1
1-m2+-11
by minimizing a statistical divergence between the learned and and the target distributions over trajectories DKL (P& [PF), where (2x) is the in flow at state 2, P=(z)|z-11 )) and P=(2-112)) indicates the forward and backward policy that predicts the probability between states.
In the case of causal LLM, token sequences are autoregressively generated one-by-one from left to right, so there is only one path to each state 2,, and each state has only one parent 2;-1. Given this condition, P=(- = for all states. By modeling (-(-I-) with (o parameterized by 0, the loss function aims to ensure consistency between the flow assigned to all trajectories from one complete rationale (i.e., Z (2)22 ... zn T) = 21.z T). Specifically, for trajectory truncated by a
paired index (i, with 0 < < j < the loss penalizes discrepancies between the flow at state 2,,

---

Visual instruction
P(ZIX.))
Y: "Answer is A"
Canstraned
think reasoning chain better.
PPO policy
Critic Model
Target posterior: P(ZIX,Y)
XX
CHOOSE ANSKER
"Answer
AVI policy (ours)
from the target posterior distributio..
"What answer?"
Figure Comparison of different training algorithms for visual reasoning. PPO implicitly ap- proximates the rationale distribution but tends to under-represent its full diversity due to limited exploration constrained by its reference policy (e.g., the SFT model), and it heavily relies on critic (reward) model. In contrast, AVI explicitly estimates the true target posterior P(Z Y) through latent rationales, which promote diverse trajectories and inherently prevent reward hacking.
scaled by the product of transition prohabilities from 2;-1 to and the flow at 2y:
F(z:) ITk=it1 (0(= 11
SuBTT (2:0)-
Σ
log
F(zy)
OSicjšn
R(21 T) ITk=e4 g0(2x 21xk-1)90 21:5)
Σ
log
R(21: T) qo 21:4)
OSicjEn
T) when z; is the the final state, R( T) is the reward of trajectory where F(z;) R((1s)
( ends at 2g, where represents the terminal state, which is usually an (eos) token LLM.
3 Amortizing Variational Inference for Latent Visual CoT
By leveraging GFlowNets for AVI in LVLM, we formulate visual reasoning as a variational inference problem, as shown in Fig. I That is, given a question-answer pair (X, Y) as observation, the goal is to find the latent visual CoT sequences Z that contribute the most to the conditional likelihood:
P(Y
E P(ZY|X),
(3)
Z-P(Z(X,))
where XXZY|X) denotes the likelihood assigned to concatenated sequence (e.g., ZY) given visual instruction X, and Z a latent CoT supposed to be sampled from posterior distribution P(Z) (Z) X, Y). To approximate such a posterior,
use GLowNets objective derived in Eq. ( variational inference method - to train an autoregressive model qo (x). By minimizing Eq. (2). the policy model learns generate trajectories where the probability of generating a particular trajectory is proportional to its reward (i.e., unnormalized posterior probability), ensuring that higher- likeeooiikelihodd rationales (as determined by are more likely.
amortized
However, Eq. 2 requires token-level reward, which is infeasible in complex reasoning chains with thousands of tokens. 2 Efficient and diverse exploration remains challenging research problem in reinforcement learning, especially when an environment contains large state spaces. Given these research problems, provide our solutions the following sections.
3.1 Token-level Marginal Reward Approximation
The proposed amortized rationale sampler go (Z) shares the same generation process as au- toregressive LVLM: given prefix condition X, and the i-th step, token is sampled from

---

policy model qo (0 X, 21:-1),, which is then appended to the sequence. Consistent sampling autoregressively from the LVLM until a terminal state T is reached gives us one completion of rationale 7 (21
T). As shown in Eq. (2), the objective function incorporates state-level rewards, enabling the model to correctly attribute the contribution of each step to the final reward. By
setting the reward R( T) log P(X 21: TY) & P(=1a T X, Y), we optimize the policy model to sample all trajectories such as T = Z1:t7 from the target distribution at convergence.
By treating each token as a state, such a train- ing algorithm provides clear guidance for the policy on how early actions impact the final out- come, helping reduce variance and improving convergence [27]. However, directly computing the exact reward for all states is computationally expensive during training, especially for a long rationale sequence. natural approximation is to assume local smoothness of reward within small regions. To efficiently estimate interme- diate rewards, we adopt a linear interpolation strategy within segmented regions of length À shown in Fig.
1-
Zn-
Z
Liverr Inlepplation
Livea Iniepolstinn
R(ST)
R(A
T
R(3na=TT) R(4snT)
Terminal
Computed approximated reward
Figure 2: Within complete rationale sequence, we compute the actual reward after each À steps and adopt linear interpolation strategy to estimate the intermediate steps.
The following proposition summarizes our theoretical claim for improving the training efficiency of Eq. ( This approximation leverages the local smoothness of the log-likelihood, significantly reducing computational overhead without substantial loss in accuracy. We empirically evaluate the effectiveness of our claim in the experimental section.
Proposition 1. Let R(21: T) = log P(X Y) be a joint-lkeeihood reward function.
(a) If R(2):) ) and R(21:++x) are true reward and the intermediate rewards within region of length are constantly increment, then we can approximate the reward at step t 4 (where 0 S S X) as
((zue4iT) R(212 T) +)+ i
(R(z14+T)) - R(z1a
(4)
(b) If is short enough, the interpolation reward error stays close to O and the flow between F(Z1:-) and +xx satisfies Eq. ①
Proof. See Appendix A
Substituting the estimated reward R in Eq. (2 gives our modified interpolated sub-trajectory balance (CiSubTB) loss:
R(a II (0( 21: (10 21:5)
LIS (2:0)=
Σ
log
(5)
R(21s go(T 21a)
0sicj5n
where A(21:1 T) is defined pice-wise as:
A(21TT)
if is the index of actual reward,
R 21:4 T)
A(21aT) 1-t (R(EL:+AT) - if < iictt À (estimated).
By computing the sparse rewards and efficiently approximating the intermediate states' rewards, easily apply mini-bated exploration for diverse sampling to improve the generalizability of qe (Z X) by covering the full target posterior.
Reference-Guided GFlowNet Fine-tuning
Previous works 13 26 suggest that exploration can let policy gradient methods collect unbiased gradient samples, escape deceptive local optima, and produce policies that generalize better. However, as shown in Fig- allowing the model to explore without constraint causes the catastrophic forgetting issue, where the model tends to generate meaningless content with high likelihood but low reward. Existing methods, such as KL penalty [33 and clipped surrogate objective (34), control the size

---

of the gradient update. If the resulting policy is too far from the previous policy, the KL penalty constrains it take an overly aggressive learning step. However, such a method limits the exploration and increases the variance of trajectories 45 To address this issue, we propose simple but effective solution by integrating a reference-based mechanism to guide the exploration process
towards generating higher-quality rationales.
During training, we first explore m candidate latent rationales
Loss
Log log.lkellhodd
Lag-reward
1:50
40
{Z1, Zz..... Zm) from the current policy model go (X) and compare them against reference rationale Zref that anchors the search in data-grounded region. Before gradient descent, each candidate Z is associated with a reward R(Z,) log P(XZ,Y) and the ones that underperform the refer- ence rationale are discarded before they reach the gradient, preventing collapse into meaningless but high-prold high-probability trajectory. To achieve candidate filtering, we define an indicator function:
1.25
60
880
-100
0.75
Reference gaiced Natvennatmint
-120
0.25
0.00
50
100
150
50
100
150
50
100
150
Stop
Stop
Stop
Figure Allowing the policy model to explore the state space without constraint causes the catastrophic forgetting issue. The proposed reference-guided exploration effectively addresses this problem.
I(Z2) = 1. if R(Z.) > 6, R(Te))
(6)
0, otherwise
where 6, = Tmaz (Tmon Tminn min(1, 8/ x/50) is the annealing coefficient, 8 the index the current training step. By doing this, we tolerate more exploration the beginning and gradually increase its standard. The acceptance bar tightens only after 50 steps, allowing the model to explore first and then exploit later. Furthermore, by filtering out low-reward trajectories, we back-prop only through teatter-thhn--efeeneee samples, which reduces gradient variance without hand-tuning the gradient clip or KL penalty.
By incorporating the reference- based mechanism into Eq. our final objective function denoted as Snference-Guided Get fine-tuning (RGFN): I(2)) LISATTB (isabTB(Z;;
CRGEN (Z;;))
(7)
3.3 Bayesian Inference over Latent Rationales
Image query,
Inference-scaling method such as Best-of-N (BoN) generates multiple candidate responses and select the best one based on a verifier are widely used in reasoning LVLM [47 46). However, BoN has significant com- putational overhead [18). high dependency on reward model quality (2]. and scalability challenges (30]. To address these limitations, this work introduces probabilistic method, namely Bayesian inference over N la- tent rationales (BIN). Our approach is inspired by recent advancements in amortized variational inference for hierarchical models []. where shared parameters represent local distributions, facilitating scalable inference.
Figure 4:
Inference
pipeline of BiN.
Given input X and a target answer Y, we can sample latent rationales Z from a posterior P(Z Y) that bridges X and Y. forming a joint sequence X ZY. The joint likelihood is denoted as P(XZY). and the marginal likelihood of given X is expressed as
P(Y X)
Σ P(ZY XX) Z-P(Z|X,))
Σ Z-P(Z|X,))
P(Z () P(Y XZ))
(8)
Therefore, However, is infeasible to sample all latent rationales from the P(Z|X, Y we we the policy model qo( ( trained via Eq 7 to approximate the marginal likelihood. Fig. shows the complete inference pipeline where we perform the following steps: (i) Sample N latent rationales {Z, J from the learned policy model: Z; go (Z) (ii) For each sampled rationale Z,,
employ
5

---

X System-mesage (cos)
Auystem-messange (eos)
Xsstem-message (cos)
User: Xymage Xinstruct (cos)
User: Xmaae Xinstruct (cos)
User Xmage iinstruct (cos)
Analyzer: (eos) (Optional)
Assistant X Xinswer (cos)
Assistant (think) Xinikk (/hhink)
Assistant Xanswer (cos)
(conclusion) Xanswer (ccocclusion) (cos)
Pre-trained LVLM
Fine-tuned reasoning LVLM
Latent reasoning LVLM (Ours)
Figure 5: Input sequence of training a reasoning LVLM. We use token to represent learnable parts. Specifically, the fine-tuned reasoning LVLM heavily relies on annotated data during optimization, and the object tokens followed by Assistant enforce reasoning for all instructions. We introduce a new role token Analyzer, so the model can selectively provide reasoning steps.
sample the corresponding answer y(1) from ne XZ,) where TO is a reasoning I.L.M. (iii) Compute the joint likelihood for all pairs (Z,Y;): TT. X)). (iv) Estimate the marginal likelihood by normalizing over sequence length |i as
P(Y x) N Σ
[2,Y, T (2,Y, X).
(V) Select the answer Y₁- with the highest estimated marginal likelihood: i" arg max; P(Y; as the final output. This inference strategy aligns with Bayesian sampling principles by approximating the marginal likelihood P(Y through sampling over latent rationales. The use of amortized variational inference for qo [ X) enables efficient sampling without the need for computationally intensive methods like Markov Chain Monte Carlo (MCMC). By selecting the answer with the highest estimated marginal likelihood, we aim to improve the interoperability of answer selection.
4 Empirical results
4.1
Implementation
Reward model. This work utilizes a fine-tuned reasoning LVLM denoted as To parameterized by + as the reward model R. Efficiently, TT also acts as the starting point of the proposed rationale sampler. The purpose of our reward model is to evaluate the quality of rationales sampled from the policy model (rationale sampler). To make sure that the reward function returns a higher reward for better rationale, we first optimize अ.. by maximizing the likelihood of high-quality, structured examples of rationales (SFT), such as thain-of-thougle thought (CoT) sequences. By learning from these examples, the model gains initial understanding of how to approach complex tasks methodically. For training T&. we consider two pre-trained LVLMs as the base models, including Qwen2.5-VL-3B4 7B Bl and mixture of visual reasoning datasets from LLa
(47) and R1-Onevisioon (48). As shown in Fig. 5, we formulate the instructional data with a new special token Analyzer. We fully fine-tune To using the regular token prediction loss for one epoch.
Rationale sampler. To sample the latent rationale Z from the posterior defined in Eq. we parameterize the policy model as autoregressive model go ( initialized with mę. For training, we optimize the model using LoRA with r = 64 and alpha = 128. We resample 3k visual reasoning sample from the SFT data, where each consists of (image, query, CoT, and answer). To be noted, we use the CoTs generated by teacher models, such as GPT-4o or Deepseck-R.. as our reference rationale Zref in Eq. (6). For the reward approximation defined in Eq. 4 set À 8 for all the experiments. Please refer to Appendix B.2 for the study of X. More hyperparameter settings can be found in Appendix B.5
4.2 Multi-modal Reasoning
Task deseription. Multi-modal reasoning evaluates the visual understanding and reasoning ability of LVLM as it requires step-by-step thinking and correct answer searching. This work proposes reasoning LVLM, i.e., LaCoT, which consists of latent rationale sampler qo and an answering model TT. Specifically, at test time, randomly sample m latent rationales Z for an unseen X with

---

Table 1: Test accuracy (%) on visual reasoning benchmarks. are results based on our reproduced experiments. The best results are bold, and the second-best results are underlined. We choose the reasoning models fine-tuned with SFT and GRPO (RI-Onevision) as baselines. All the base models were prompted by step-by-step reasoning instruction.
Math Vista mini
Math full
Math Verse
MMMU
MMMU-pro vision
MMVet test
MME
Method
vision-only
val
test
GPT-4o
60.0
30.4
40.6
70.7
51.9
69.1
2329
Gemini-1.5-Pre
63.9
19.2
65.8
46.9
64.0
2111
Claudc-3.5-Sonel
67.7
46.3
68.3
51.5
66.0
1920
V1.2-4L.2-4B
58.6
16.5
32.0
47.9
55.7
2046
Qwen2.5-VL-33
3
60.3
21.2
26.1
46.6
22.4
61.4
2134
LaCoT-Qwen-3B
63.2
20.7
40.0
48.8
28.9
69.6
2208
LLaVA-CoT-11B
47
52.5
22.6
64.9
I.
21
63.2
11.1
26.2
48.8
24.1
57.5
1998
MiniCPM-V2.6 44
60.6
17.5
25.7
49.8
27.2
60.0
2348
Intern VL2--B
58.3
18.4
37.0
52.6
25.4
60.0
2210
Qwen2.5-VI-7Bf
63.7
25.4
38.2
50.0
34.6
70.5
2333
R1-Onevision
[88
64.1
23.9
37.8
47.9
28.2
71.1
1111
LaCoT-Qwen-7B
68.4
24.9
43.3
54.9
35.3
74.2
2372
temperature from qo ( X), then the answer model samples m answers from T4 (Y Finally, estimate the marginal likelihood of each answer P(Y|X) using the proposed BiN and return the highest one as the final output, as shown in Eq. (9)
Benchmarks. This work utilizes three mathematical and one general domain reasoning benchmarks: (i) Math 24 a math benchmark designed to combine challenges from diverse mathematical and visual tasks, requiring fine-grained visual understanding and compositional reasoning. (ii) Math Vision [42]: a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. (iii) Math [55]: all-around visual math benchmark designed for an equitable and in-depth evaluation of LVLMs. We report the Vision-Only result on 788 questions, which reveals significant challenge in rendering the entire question within the diagram. (vi) MMMU [51]: a benchmark designed to evaluate LVLM on massive multi-discipline tasks demanding college-leve level subject knowledge and deliberate reasoning. Furthermore, we conduct additional experiments on MMMU-pro [52], MM (50), and MME (9] where MMMU-Pro is a robust version of MMMU, designed assess LVLMs' understanding and reasoning capabilities more rigorously.
Results. This work provides two LaCoT models (3B and 7B). From the results summarized in Table [] models are the best open-source LVLM and nar-
Rational-solution log-likellhood
00.55
38,
38. Sampling w/
row the gap to GPT-4o to less than points while using only 7 billion parameters. The consistent im- provements on Math Vista and MMMU show that LaCoT strengthens general multi-modal reasoning. improves the most, especially 3B, where accuracy jumps 14 points and outper-
0.20
0.5
-0.25
78, =
-0.30
Math
78. T 0.5
forms all 7B models. This advancement indicates that
-0.40
78, Sampling ng w/T 0.5
78, Sampling w/ T = 0.7
LaCoT significantly boosts diagram comprehension and OCR robustness. On the other hand, Math Vision consists of real Olympiad diagrams, which are more varied, and often handwritten or low-resolution, con- ditions that push OCR and visual grounding beyond. Many problems split critical information between text and picture (e.g., tiny angle labels or subtle curve annotations), so single misread propagates through the longer, proof-style reasoning chains, leading to a performance drop.
-0.45
SFT
LaCoT
Rational diversity
Figure 6: Maximum log-likelihood and diver- sity of the sampled rationale. LaCoT model (*) samples higher log-likelihood rationale while maintaining higher rationale diversity than SET (o) model.

---

The LaCoTT model can sample rationales with higher diversity than baseline models, increasing the probability of sampling answers with higher likelihood. To validate this hypothesis, we sample 5 rationale candidates with random temperature (T) for each visual instruction. To measure the semantic diversity of the samples, we compute the average inter-sentence similarity between the candidate and the reference set. As shown in Fig. [ rationales generated by LaCoTT-Qwen33B with = 0.7 have the highest log- ligelikel and diversity. Qualitative results can be seen in Fig. and the supplementary.
4.3 Inference-time Scaling
We compare BiN (ours) with Best-of-N (BoN) using LaCoT-Qwen as the shared policy model. At inference, we sample N ratio- nale-answer pairs, compute a length-normalizer log-likelihood of each answer as the reward, and for BoN select the answer with the highest re- external reward
Table 2: Comparison between two inference-time
scaling methods using LaCoT-Qwen (3B/7B).
Method
Math Varhe
MMMU
MMVet
3B BoN
21.2
57.1
44.7
67.1
3B w/ BiN (ours)
40.0
63.2
48.8
69.6
ward. To ensure fairness, model is used. We evaluate E (5, 10) for both methods and report the best score per method. As shown in Table forms BoN on visual reasoning benchmarks.
7B w/ BoN
26.5
62.2
47.3
71.2
7B w/ BiN (ours)
39.7
68.4
54.9
74.2
2 BiN consistently outper-
4.4 Ablation Studies
Effectiveness of RGEN. As baselines, we consider zero-shot prompting w/o reasoning, supervised fine-tuning on the visual reasoning dataset, and GRPO 35 fine-tuning.
From the results summarized in Table 3 the base model performs well without chain-of-thought rea- soning. While supervised fine-tuning on reasoning data slightly improves performance two bench- marks, it still struggles to generalize to challenging visual reasoning tasks. Fine-tuning with GRPO yields poor performance, partly due to inadequate guidance of the external reward model, i.e., it cannot distin- guish good rationales from bad ones, and limited exploration due to the KL penalty. Such misleading optimization due to misaligned reward is a widely noted issue in RL-based algorithms [57 for LVLM. On the other hand, by matching the target distribution, RGFN avoids collapsing to single mode of the reward, and the reference-guided exploration covers diverse trajectories, leading to better performance on complex examples.
Table 3: Test accuracy (%) on reasoning
benchmarks using Qwen2.5-VL-7B model.
Method MathVista MathVerse MMMU
Zero-shot
63.7
38.2
50.0
SFT
62.7
38.7
50.6
GRPO
62.6
36.8
47.9
RGFN
68.4
43.3
54.9
Study of BiN. To evaluate the scalabil- ity, we apply the proposed inference- scaling method by varying the number of candidates N and temperature T us-
MathVerse-VO
MMMU
48.7
49
Accuracy (%)
48
48.6
37.2
ing T-CCo--wen-3B on the reasoning benchmarks. As illustrated in Fig. 7 test accuracy consistently increases with higher N, and higher T. This indicates that increasing N systemat- ically improves test-time accuracy be- cause it (i) reduces the Monte-Carlo variance of the marginal-likelibnn likelihood estimator standar error scales as thereby stabilizing an- swer rankings; (ii) offers broader pos- terior coverage, mitigating mode-dropping bias inherent in the amortized sampling qo (Z[X); (iii) smooths fluctuations introduced by fungth-nrrmllzztiion yielding more reliable re- re-ieightnng: and (iv) enlarges the candidate answer set, elevating the chance that the correct output observed. Together, these effects drive an exponential decay in the probability of selecting incorrect answer.
37.8
47
36
36.0
46
34
33.7
45
41,7
T=0.5
T=0.7
32.7
31.5
44
Greedy
43
63.6
10
#Rational Candidates
#Rational Candidates
Test accuracy on reasoning benchmarks using LaCo --eoo---..nBB.... We evaluate the impact of #rationale can- didates (N) and random temperature (T).
Figure
8

---

Furthermore, higher N can effectively address the hallucination issue in visual reasoning. As shown in Fig. 7 when the sampled rational size N 1, BiN may produce incorrect or misleading reasoning steps and lead to lower answer accuracy on the MMMU dataset. However, increasing N from I to 5 significantly mitigates hallucination and improves answer accuracy. We provide qualitative results in Appendix B.1
To evaluate the generalizability of BIN, we evaluate the performance of Qwen2.5-VL. 3B & 7B (SFT) with the proposed inference-scaling method. We set N 5 and T 0.7, which gives the most performance boost with relatively shorter inference time. As shown in Table 4 BiN consistently improves the model performance on all benchmarks, indicating the effectiveness of BiN as general inference-scaling method for reasoning LVLMS.
Table 4: Test accuracy of Qwen2.5-VV super- vised fine-tuning on reasoning data.
Method
Math Vista
Verse
MMMU
7B (SFT)
62.7
38.7
50,6
BiN
64.4
38.9
51.6
3B (SFT)
58.7
33.3
43.1
BiN
59.4
35.2
45.0
5 Related Work
Learning-basee Multimodal CoT (MCoT) methods have emerged powerful paradigm for enhancing the reasoning capabilities of LVLMs [40] 39]. Unlike prompt-based or plan-based ap- proaches, learning- based MCoT explicitly embeds the entire reasoning trajectory into the models through supervised learning on rationale-augie aooonaal--ggmentedd datasets. Early studies such as Multimodll-O (6) pioneered this direction by fine-tuning LVLMS to generate visual COT, facilitating a structured reasoning process aligned with human cognitive patterns. From that, methods like MC-CoT [46] further refined this approach by incorporating multimodal consistency constraints and majority voting mechanisms during training. In addition, methods such as PCoT [43] and G-CoT (25) demonstrated that explicitly training LVLMs with structured rationales improves the interpretability and general- izability. These advancements underscore the effectiveness and necessity of embedding structured, rationale-driven reasoning capabilities directly into multimodal models.
Reinforcement Learning-based Language Models have demonstrated significant effectiveness in advancing the reasoning capabilities of LLMs. DeepSeck-PP R1 [12] exemplifies this by activating oong-chai-of-tthugge (long-CoT) reasoning solely through reinforcement learning (RL), achieving improvements over models such as GPT-ol [17] in specific aspects when combined with supervised fine-tuning (SFT) cold starts and iterative self-improvement. This success has spurred further interest in RL-driven models, including Open-R1 (8) and TinyZero (3). To enhance reasoning, generalization, and ensure training stability, several RL algorithms have been developed, such as PPO
344, GRPO [12] and simplified methods like RLHF (58), DPO 32, and SPO (38) Nevertheless, these approaches are heavily dependent on high-quality
data (c.g., human preference labels and scalar rewards) and typically produce policies with limited diversity. To address these limitations, this work proposes an RL algorithm specifically designed to train LVLMs using amortized variational inference, which is capable of generating diverse outputs and supporting probabilistic inference- -imee scaling.
Scaling methods aim enhance reasoning performance during inference by lever- aging high-quality prompts and effective sampling strategies. Plan-based approaches, exemplified by MM-ToT [10) and VA-COT [47), utilize search strategies such as DFS and BFS, including Best-of-N search, sentence-level beam search, and stage-level beam search, to identify optimal rea- soning trajectories. These methods typically assess candidate trajectories using scalar metrics ranging from 0.1 to 1.0. However, such explicit evaluation is computationally expensive, as each candidate requires an additional forward pass through dedicated reward model. To mitigate this computational overhead, our work introduces a learming-based algorithm designed to align the marginal likelihood of generating rationale directly with its reward. This approach enables efficient probabilistic sampling without explicit reward computations during inference.
Inference-time
Conclusion
In a real-world scenario, solving a fixed visual query with different reasoning chains that lead to the correct answer requires nuanced understanding of image, context, logic, and flexibility in

---

Question:
How many baseballs are there?
The image shows four blue buckets, each containing five baseballs. Therefore, the total number of baseballs is calculated by multiplying the number of buckets by the number of baseballs per bucket.
Reasoning:
4 buckets baseballs/bucket 20 baseballs. Final output: There are 20 baseballs.
GRPO
The image shows four blue buckets, each containing five baseballs. determine total number baseballs, we can direclly count the baseballs each bucket. As seen the image, each bucket has five baseballs. Since there are
Reasoning:
buckets and each contains five baseballs, can calculate the by multiplying number bucket:
buckets by the number baseballs
4 buckets x 5 baseballs per bucket 20 baseballs.
There is no need to consider any additional factors or hidden details since the image clearly shows the contents of each bucket.
Higher diversity
Final output: 20
LaCoT
Figure 8: Qualitative results of visual reasoning. LaCoT can sample a more diverse and comprehen- sive reasoning chain than the GRPO model.
thought. While querying this knowledge in LVLM involves sampling from intractable posterior distributions. To address this challenge, we propose novel training algorithm based on amortized variational inference for latent visual chains-of-thought (CoT). Our approach incorporates token-level reward approximation and RGFN, enabling effective and efficient optimization of a policy model to generate diverse and plausible reasoning trajectories, outperforming both supervised fine-tuning and maximmiaaiiiiiionnn baselines. In addition, we introduce new inference-ting scaling strategy, BiN, which mitigates reward hacking and enhances interpretability with statistically robust selection criteria. Building upon these components, we present LaCoT that leverages a rationale sampler for general-purpose visual reasoning, and answer generator that is enhanced by high-quality reasoning chains. Given this system, future work should investigate the possibility of applying for knowledge distillation and synthetic data generation.
Limitations. Due resource constraints, we apply the proposed methods models up to 7B parameters, but we expect the conclusions to hold for larger models. In fact, our training and inference method can be applied any autoregressive model, including LLM and LVLM, with various model sizes. As with any on-policy method, exploration in tasks with complex latent remains an open challenge since multiple factors can affect the exploration time, such as sequence length and technical challenges like memory cost. Despite the improved inference performance, this work does not address issues such as hallucination, which are closely related to internal knowledge.
Acknowledgments and Disclosure of Funding
This research was supported in part by the DEVCOM Army Research Laboratory under Contract W911QX-21-D-00N the National Science Foundation under Grant 2502050, and the National Institutes of Health under Award R16GM159146. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
References
[1] Abhinav Agrawal and Justin Domke. Amortized variational inference for simple hierarchical models. In Neural Information Processing Systems, 2021.
[2] Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell, Variational best-of-n alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv, 2025.
10

---

[4] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-2737 1-27994, 2021.
[5] Yoshua Bengio, Tristan Deleu, J. Edward Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio, Gflownet foundations. In The Journal of Machine Learning Research (JMLR), 2023.
[6] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 2016.
[7] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[8] Hugging Face. Open rl: A fully open reproduction of deepscck-rl, January 2025.
[9] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306.13394 2023.
[10] Kye Gomez. Multimodal-tot. Accessed: 2025-04-20.
httpp///ggtttaoooooo/rgg/g////tt 2023.
[11] Google. Gemini 2.0 flash, 2025.
dewember- -et.:aeeeooeeeerrrre/eeeeeeaaaaa
[12] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[13] Zhang-Wor Wei Hong. Tzu-Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
[14] Edward Hu, Moksh Jain, Eric Elmoznino. Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[15] Hang Hua, Qing Liu, Lingzhi Zhang, Jing Shi, Zhifei Zhang. Yilin Wang, Jianming Zhang, and Jicbo Luo. Finecaption: Compositional image captioning focusing on wherever you want at any granularity. ArXiv, abs/2411. 1..14111111111, 2024.
[16] Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning. In AAAI Conference on Artificial Intelligence, 2024.
[17] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
[18] Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. ArXiv, 2025.
[19] Andrei V. Konstantinov, Lev V. Utkin, Alexey A. Lukashin, and Vladimir Mulukha. Neural attention forests: Transformer-base forest improvement. ArXiv, abs/2304.05980, 2023.
[20] Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández- Léna Néhale Ezzine. Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pages 18300. PMLR, 2023.
[21] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv, 2024.
11

---

[22] Huafeng Liu, Jiaqi Wang, and Liping Jing. Cluster-wise hierarchical generative model for deep amortized clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 15109-15118, 2021.
[23] Huafeng Liu, Tong Zhou, and Jiaqi Wang. Deep amortized relational model with group-wise hierarchical generative process. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7550-7557, 2022.
[24] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yue Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2023.
[25] Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In European Conference on Computer Vision, 2024.
[26] Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. A lappaccann framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, 2017.
[27] Kanika Madan, Jarrid Sector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning (ICML), 2023.
[28] Nikolay Malkin, Salem I Lahlou, Tristan Deleu, Xu Ji, Edward J Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. GFlownets and variational inference. In The Eleventh International Conference on Learning Representations, 2023.
[29] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:330.08777, 2023.
[30] Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. ArXiv, 2025.
[31] Jayy Pan, Junjie Zhang. Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https:/gituub..om/ijiiiiit TinyZero, 2025. Accessed: 2025-01-24.
[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 2023.
[33] John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference Machine Learning, 2015.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv: 2017.
[35] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, 2024.
[36] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 2022.
[37] Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
[38] Guohao Sun, Can Qin, Yihao Feng. Zeyuan Chen, Ran Xu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Structured policy optimization: Enhance large vision- language model via self-referenced dialogue. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2025.
12

---

[39] Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang. and Zhiqiang Tao. Self training large language and vision assistant for medical question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
[40] Guohao Sun, Can Qin. Jiamian Wang, Zeyuan Chen, Ran Xu, and Zhiqiang Tao. Sq-llava: Self-questioning for large vision-language assistant. In European Conference Computer Vision, 2024.
[41] Ily Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence sequence learning with neural net- works. In Proceedings of the 28th International Conference on Neural Information Processing Systems, 2014.
[42] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench- marks Track, 2024.
[43] Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering. In Proceedings of the AAAI Conference Artificial Intelligence, 2024.
[44] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[45] Yuhui Wang. Hao Hc, and Xiaoyang Tan. Truly proximal policy optimization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, 2020.
[46] Lai Wei, Wenkai Wang, Xiaoyu Shen, Yu Xie, Zhihao Fan, Xiaojin Zhang, Zhongyu Wei, and Wei Chen. Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with I1m and milm integration. arXiv preprint 2410.04410 2024.
[47] Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024.
[48] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin. Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R1-onevision: Advancing generalized multimodal reasoning through cross mros--modal modal formalization. ArXiv, 2025.
[49] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao Zhihui He, Qi-An Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone. ArXiv, 2024.
[50] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, 2024.
[51] Xiang Yue, Yuansheng Ni, Kai Zhang. Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. 2024 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2023.
[52] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: more robust multi-discipline multimodal understanding benchmark, In Annual Meeting of the Association for Computational Linguistics, 2024.
13

---

[53] Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:2008 -0266, 2017.
[54] Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning, pages 26412-26428 PMLR, 2022.
[55] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal 11m truly see the diagrams in visual math problems? European Conference on Computer Vision, 2024.
[56] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning language models. arXiv preprint arXiv:2302.00923, 2023.
[57] Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. ArXiv, 2024.
[58] Daniel M Ziegler, Nisan Stiennon, Jeffree Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv. org/abs, page 14, 1909.
A Proof of Proposition
Proof. (a) Assume that within the segment {t, + 1,...,
.. }} the true reward grows linearly, i.e.
R(21444 -(( osisA.
R(2L++iT) R(211 T) i
Substituting this expression into Eq. shows R(2144) = R(21 T) for every i, so the interpo- lation incurs zero error.
(b) Suppose R is viicediifferentiabbl along the trajectory and its discrete second derivative is bounded:
R(21:411 T) 2R(21: T)
R(21:-1 T) M,
Vs.
The classical linear-interpolation error bound then yields
M i(A -) 1) S
M
R( - R(21444 T)
0 OsiSA.
(10)
8
8
Thus the approximation error decays ((23); choosing À sufficiently small keeps it arbitrarily close
to O.
Let
F(2x) = R(21.1 T) ( T 21:0)
F(2.) = R(21= T) go( 11:0))
Es == R(21:s T) - R(21:0 T).
From Eq. 110 we have |E,l S Mdx² s0
F(zz) = F(2a)
Ee
R(Z1: T)
Denote Rmin = min R(ZI:T]) > 0 (positivity follows from likelihoods).
Then
|e./ R(2(1 T)
MA²
88mm
Consider any i E + X. Applying Eq. I to both F and F, we obtain
F(z:) II
Pr 2k-1) F(z)) II
(= 2x) (1+
kmi+1
Ami+1
14

---

Which cube is identical to the unfolded net?
Reasoning: Okay, I'm trying to figure out which cube is identical the unfolded net based on the given image. Let me break this down step by step. The top row has Looking at the cubes mentioned:
Cube A: Blue, green, yellow sides. Cube E: Purple, yellow, green sides.
2
3
4
The image shows that the top row (boxes 1-4) has purple and cyan boxes, ... Cube
represents the top-left box (1)... Cube would represent the middle-bottom box (4), and Wrong reasoning.
5
Cube would represent the bottom-right box (5).... X
6
Final output: E
Wrong answer.
SFT
Okay, I'm trying to figure out which cube is identical the unfolded net based on the image itself. Let me break down step by step. The hint says that the correct answer is C, but let's verify this directly from the image. Looking at Cube C. we can see that has the numbers !22"" and 1"5l" on its visible faces. This matches the arrangement of the numbers in the grid where 1"21" and 1"5!" Let's check the other cubes: Cube A has 1"2 and 1"31", Cube has 1'4"" and I""".... Cube must the correct choice.
Reasoning:
X
Wrong reasoning.
Final output: C X Wrong answer.
GRPO
Reasoning: Okay, so trying to figure out which cube is identical to the unfolded net based on image itself. Let start by understanding the problem.
First, the image shows unfolded net of die, which means it's
Looking at the image:
Cube A has faces showing 2. and 3. Cube B has faces showing 4. 1, and 3. Cube C has faces showing 2, 5, and 9. Cube D has faces showing 2, 3, and I. Cube E has faces showing 2, 5, and 9.
The net in the image has the following arrangement:
Face in the top center in- Face is in top comer.In- Face 3 is the top right comer.nn Face is the the bottom right corner.
bottom left comer.In- Face is the bottom center Face
Now, let's check each cube against the
Cube A: Face 2, 5, and 3. Cube D: Face 2, 3. and 5. If face 2 is in the top left. then face 3 would be in the top right.
and face 5 would be in the top center. This fits the net's arrangement where face 5 is at the top center...
Final output: D
LaCoT
Figure F9; Qualitative results of visual reasoning. We highlight the important reasoning steps.
where the O(22) term accumulates at most (j - i) relative perturbations bounded by MX² 108//88Riia R. Hence, the forward and backward product flows still match up multiplicative factor that vanishes quadratically as A ~ 0. Therefore, the interpolated rewards preserve flow consistency to arbitrary precision for sufficiently small segment lengths.
B Experiments
B.1 Qualitative results
In Fig. we provide qualitative results of a comparison between Qwen2.5-VL-7B (SFT), Qwen2.5- VL-7B (GRPO). and LaCoo--Qwen-777 As can be seen, LaCoT-Qwen-7BB reasoning chain, leading to the correct answer. Mcanwhile, due to limited generalizability. SFT and GRPO samples show the wrong visual COT. In Fig. FIO and Fig, FIT our LaCoT model can sample more straightforward and accurate reasoning chains, demonstrating the effectiveness and robustness of the proposed training and inference algorithm.
provide more accurate
B.2 Study of interpolation reward
In Table T5
we study the impact of the interpolation reward with different skipped steps (i.e., X) in the reward approximation process of the policy optimization. As mentioned in Proposition]] smaller À theoretically leads to more fine-grained reward supervision but longer training time.
B.3 Efficiency analysis
Give superior performance gain by sampling multiple rationales at inference time, but this process introduces additional inference cost, and we address this by using mini-batching (with batch size k=5) to generate N rationales in N/k forward passes. In Table T6 we report the average per-sample inference time (reasoning + answering) and corresponding performance of different reasoning-L.VV on Math Vista and Math Verse. As can be seen, LaCoT-Qwen-7B consistently achieves stronger
15

---

Okay, so need to figure out how many objects are left after subtracting all green rubber cubes and red matte blocks from this image. Let me start by understanding what each First, "green rubber cubes." The image shows three cubes: blue with reflective surface, one cyan with glossy finish, one yellow with shiny appearance. There no green cube in the image. Therefore, no green rubber cubes exist to remove. Next, "red matte blocks." The image shows two small red objects, both which appear to be cubes based on their shape. They do not look matte; they have glossy finish. Therefore, there are no red matte blocks to subtract either. X Wrong reasoning. Given that neither green rubber cubes nor red matte blocks are present in the image, we can conclude that no objects need to be removed from the total count.
Reasoning:
means.
Question: Subtract all green rubber cubes. Subtract all red matte blocks. How many objects are left?
Looking the initial setup described: therel's large blue sphere with reflective surface, smaller cylinder, medium-sizen sizzuud--szddddd red matte cube, blue-tinted metallic small yellow cylinder, another small yellow cylinder, and small purple sphere. Let''s count them:
sphere (reftective)In2. cylinder (cylinder)\n3. Red matte cube (cube)\n4. Blue-tinted metallic cube (cube)\n5. Yellow cylinder (cylinder)In6. Another yellow cylinder (oylinder)\n77 Purple sphere (sphere)
That makes seven objects in total. Since neither green rubber cubes nor red matte blocks were present in the image, all seven objects remain.
Therefore, the answer should be 7. Final output: 7. X
Wrong answer.
SFT
The image contains several objects: blue sphere, two cylinders, a red cube, a teal cube, yellow cylinder, purple sphere, another yellow cylinder. First, need to Identify and remove any green rubber cubes. As seen in the image, there are no green rubber cubes present.
Reasoning:
Next, will look for any red matte blocks. The image shows red cube, which appears to be matte. I will subtract this red cube from the total count. V
Now, let's count the remaining objects:An- Blue spherein- Two red cylindersin- Red cube (removedj\n- Teal cubeên- Yellow cylinderkn- Purple sphereln- Another yellow cylinder After removing the red cube, the remaining objects are:n1 Blue spherekn2. Two red cylindersin3, Teal cubeln4. Yellow cylinderIn5. Purple spherein6. Another yellow cylinder This totals objects. Therefore, after removing block, six objects remain Final output:
LaCoT
Figure F10:
Qualitative results of visual reasoning. We highlight the important reasoning steps.
Table T5:
Study the impact of the interpolation reward with different skipping steps (i.c., A) to the
policy model.
Math Vista mini
MathVision full
VarsV vision-only
MMMU
Overall Avg.
Method
val
Qwen2.5-VL-7B
63.7
25.4
38.2
50.0
44.3
32
64.9
23.0
42.5
51.9
45,6
LaCoT- LaCoo--Qwen-7B
68.4
24.9
39.7
54.9
47.0
performance, even with modest increases in inference time. Compared to other multi-rationale baselines, LaCoT strikes a favorable balance between computational cost and reasoning reliability, thereby improving both the trustworthiness of rationales and the accuracy of final answers.
B.4 Experiments compute resources
This work utilizes an 8*80GB GPU-node for training. We set the Deepspeed Zero-3 stage and gradient-checkpointing to reduce memory costs during optimization. It takes around 30 hours supervised fine-tuning on 250k reasoning data samples, and 120 hours for GRPO and RGFN fine-tuning on 3k data samples.
16

---

Question: What is the average of longest light blue bar and shortest gray bar?
Okay, so need to figure out the average of the longest light blue bar and the shortest gray bar from the given image. Let me start by understanding what each part means.
Reasoning:
three years: 2012, 2013, and 2014. The bars color-coded light blue for 2012, dark blue for 2013, and gray for 2014.
First, the image shows bar chart comparing solved and unsolved hamicides
Python code, therel's arrays: solved' and unsolved The 'olveed' array has values [415, 406, 385] corresponding the years 2012, 2013, and 2014 respectively. seen the image, the light blue bars represent 2012, the dark blue bars represent 2013, the gray bars represent 2014. So, the light blue bar is for 2012 with a value 415.
Looking the data provided
called homicide, data which includes
Next, the 'unsovved' array has values [128, 106, 131]. The first value (128) for 2012, the second (106) is for 2013, and the third (131) is for 2014. However, since the question mentions solving or unsolving homicides, it seems we''re focusing on the solved ones here because the user referred to "longest light blue bar. But wait, the light blue bar is actually the solved homicides for 2012, which is 415.
Now, the gray bar is for 2014, which is the shortest among all the gray bars. As seen in the image, the gray bars represent 2014, and the shortest one is 385, but the note says it'ss 385? Wait, no-the note mentions that 385 for 2014, but looking back, the 'unsolved' array has 385 as its value, which corresponds to 2014. So indeed the shortest unsolved homicides, which would 385. But let check again,
perhaps the gray Wait, maybe I'm mixing up something. values are [415, 406, 385). The unsolved homicides are shown gray, values [128, 106, 131). for 4, both solved and unsolved have numbers, but the gray bar specifically refers to unsolved homicides, which lower than the solved ones. So the shortest gray bar is 106 (for 2013).
clarify. The solved homicides shown light blue, and their
But hold on, the user mentioned the longest light blue bar and the shortest gray bar. So the light blue bar is 415, which is the longest. The gray bar varies between 131, 385, and 106. The smallest number here is 106, which is in 2013. So the shortest gray bar is 106. X Wrong reasoning.
So now, averaging these two: 415 and 106. To find the average, add them together and divide by 2. That gives (415 106)/2 = 521/2 = 260.5. Since the user asked for an integer, should round this. 260.5 rounds to 261. So the average 261.
Final output: 261.
Wrong answer:
SFT
Reasoning: Okay, so need to figure out the average of the longest light blue bar and the shortest gray bar in the given image.
First, let'ss look at the image itself. The image shows bar chart with two main categories: "Solved homicides" and "Unsolved homicides." Each category has three bars representing the years 2012, 2013, and 2014. The colors used are light blue for 2012, dark blue for 2013, and gray for 2014.
Now, focusing the light blue bars, which represent and for unsolved homicides, it's 128. Next, looking at the gray bars, which represent the year 2014. The values here are 385 for solved homicides 131 for unsolved. So, the gray bars show 385 and 131.
year 2012. The values for solved homicides 415, solved 128 for unsolved.
light blue bars show
The question asks for the average of the longest light blue bar and the shortest gray bar. The longest light blue bar would be the one with the highest value, which is 415 for solved homicides. The shortest gray bar would be the one with the lowest value, which is 131 for unsolved homicides. So, we have the numbers 415 (longest light blue) and 131 (shortest gray). To find the average, we add these two
together and divide by 2.
Calculating that: \4415 + 131 546W),, then W(546 2 2731))
Therefore, the average should be 273.
LaCoT
Final output: 273
Figure Fl1: Qualitative results of visual reasoning, We highlight the important reasoning steps.
B.5 Hyperparameter
We detail the hyperparameters used for training the reward model and LaCoT in our experiments in T7] During LaCoT training, we randomly sample (mini-batch size) Zs for every (X, Y) as exploration.
17

---

Table T6:
Inference time study of reasoning model with multiple rational sampling.
#Rationals (N)
5
10
MathVista
MathVerse
LLaVA-CoT-IIB
340s
830s
52.5
22.6
R1-One Vision-7B
32s
64.1
37.8
LaCoT-Qwen-7BB
30s
65s
68.4
39.7
Table T7: Hyperparameters for training.
LoRA dropout
0.05
Batch size (SFT)
Batch size (RGFN)
Gradient accumulation (SFT)
16
Learning rate
0.00001
Optimizer
A
Weight decay
0.05
Temperature max
1.0
Temperature min
0.5
Reward temperature start
1.0
Reward temperature end
0.7
Reward temperature horizon
exploration number
À
Tioa
1.5
Tiin
1.0
Maximum rationale length
700
Minimum rationale length
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The empirical results and theory analysis support our claims in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirationale goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitation discussion in section 4.2 (Results) and conclsion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
18

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex: system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This paper has one Proposition, and we provide the complete proof in the supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
We have provided details for reproducing the results in section Experimental
Results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
19

---

If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the instructions of training data in Section 4.1. We will release our training code later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ mnips.cc/publicc//
pwt.huooommisiiooooococcc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
20

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide the instructions of training data, hyperparameterr, model architec- ture in Section 4.1 and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We have reported the results of different random temperature. However, we did not report error bars since it will be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide detailed information in supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
21

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow Code of Ethics precisely.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the discussion of potential impact in Conclusion.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use open-soure data and model in this work.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (tte::ee//errrpsccccooooeerreerr for what should or should not be described.
24


Reviews:
[{"review_id": "CamutWStRP", "reviewer": "Reviewer_B7EG", "summary": "The paper introduces a new training algorithm for multimodal reasoning. Specifically, the paper reformulates reasoning in LMMs as a posterior inference problem, and proposes a new algorithm stemming from amortized variational inference. The paper then addresses some unique challenges in the multimodal domain, and proposes a sparse reward function for token-level learning signals. Additionally, the paper proposes a new inference time  scaling approach based on Bayesian inference over N latent rationales. Empirical results show effectiveness of the proposed algorithm against SFT and GRPO.", "questions": "1. Since LaCoT introduces an additional neural-based reward model, what would be the computational overhead compared to, e.g., GRPO?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The main strength of the paper lies in the methodology part, which introduces a different view on how to learn multimodal reasoning models. The idea offers an interesting alternative beyond maximum likelihood (SFT) and outcome-supervised policy gradient (e.g., GRPO).\n\n2. The description of the method is detailed and technically solid.\n\n3. The additional exploration on Bayesian inference of the newly proposed method offers a valuable justification that LaCoT can also do effective inference-time scaling, which is one of the key benefits of current token-based explicit CoT models. \n\nWeakness\n1. The main concern lies in the experimental section which I find is a bit thin.\n\n2. I wonder if the comparison with GRPO in Table 1 is fair, as LaCoT uses a different training set as R1-OneVision.\n\n3. The performance of GRPO in Table 2 is a bit strange, as is contradictory with most recent papers observation. My empirical experience also indicates that GRPO works better than SFT especially in terms of generalization. The details of this ablation is not provided and no analysis on why this abnormal behavior happens.", "was_revised": true, "final_justification": "After the authors add the new analysis results into next revision, and if there is room in the proceedings, the paper could be accepted.", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and constructive suggestions. Please find our detailed responses to each point below.\n> **`W1-1`**: The main concern lies in the experimental section which I find is a bit thin.\n\nWe appreciate the suggestion. We conduct additional experiments on widely used benchmarks, including **MMMU-pro**, **MMVet**, and **MME**, where MMMU-Pro is a more robust version of MMMU, designed to assess LVLMs' understanding and reasoning capabilities more rigorously. These benchmarks cover a broad range of tasks, including visual commonsense, fine-grained recognition, and multi-choice QA. \n\n\n\n| Method                | MMMU-pro | MMVet | MME  |\n|----------------------|----------|-------|------|\n| InternVL2-4B         | -        | 55.7  | 2046 |\n| Qwen2.5-VL-3B        | 22.4     | 61.4  | 2134 |\n| **LaCoT-Qwen-3B**    | **28.9** | **69.6** | **2208** |\n| InternVL2-8B         | 25.4     | 60.0  | 2210 |\n| Qwen2.5-VL-7B        | 34.6     | 70.5  | 2333 |\n| R1-Onevision         | 28.2     | 71.1  | 1111 |\n| **LaCoT-Qwen-7B**    | **35.3** | **74.2** | **2372** |\n\n\n\nThese results demonstrate that LaCoT consistently improves performance across different domains and model scales, suggesting strong generalization beyond mathematical reasoning tasks.\n\n\n> **`W1-2`**:   I wonder if the comparison with GRPO in Table 1 is fair, as LaCoT uses a different training set as R1-OneVision.\n\n\nOur SFT data is comparable with R1-OneVision. However, in the RL stage—which we consider the most critical for performance—R1-OneVision (GRPO) utilizes 10k newly collected samples. In contrast, LaCoT reuses only 3k samples from the SFT stage without additional data collection, which highlights its efficiency and generalization.\n    \n> **`W1-3`**: The performance of GRPO in Table 2 is a bit strange, as is contradictory with most recent papers observation. My empirical experience also indicates that GRPO works better than SFT especially in terms of generalization. The details of this ablation is not provided and no analysis on why this abnormal behavior happens.\n\nIn Table 2, we compare the effectiveness of different training algorithms on 3k samples, using consistent hyperparameters across all methods (e.g., same batch size, base model, and learning rate). For GRPO and RGFN (our approach), we set the exploration number to 6 and the sequence length to 700. While GRPO performs slightly worse than SFT, we think this is due to GRPO’s higher sensitivity to hyperparameters and its reliance on larger training data for stable generalization. This is also reflected in our previous experiments (R1-OneVision applies 10k training data), where its performance improves with more data.\n\n> **`Q1-1`**: Since LaCoT introduces an additional neural-based reward model, what would be the computational overhead compared to, e.g., GRPO?\n\nLaCoT introduces no additional GPU memory usage or data preprocessing overhead compared to GRPO during training. However, due to the token-level reward approximation, it does incur a higher computational cost in terms of runtime. Under identical experimental settings, LaCoT requires approximately **90 hours** of total training time, compared to **64 hours** for GRPO."}, {"review_id": "xitiGe8GSJ", "reviewer": "Reviewer_fLHH", "summary": "This paper proposes a method for improving chain-of-thought reasoning in vision-language models. The authors reformulate reasoning as a latent variable sampling problem and propose a method based on amortized variational inference using GFlowNets. In addition, the method includes a Bayesian inference method (BiN) for selecting optimal rationales at inference time. The proposed method show SoTA performances across several visual reasoning benchmarks.", "questions": "- How sensitive is the performance to the choice of lambda for approximating reward? \n- What are training and inference times compared to baselines (SFT, GRPO)? \n- How does the proposed reference-guided filtering compare to other exploration strategies like epsilon-greedy or entropy-based exploration?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- The authors propose a novel theoretical framework that reformulates reasoning as a latent variable sampling problem, and using amortized variational inference to improve upon existing methods.\n- The proposed method shows strong improvements over prior methods. \n\n**Weaknesses**:\n- The proposed method is evaluated only on mathematical reasoning tasks. The generalizability to other domains remains unclear.\n- The proposed method requires sampling multiple rationales at inference time, which is computationally expensive. The paper can benefit from a detailed analysis of computational costs compared to baselines.", "was_revised": true, "final_justification": "The rebuttal considered additional evaluation datasets from different domains and addressed the reviewer's concern about inference costs. This reviewer will keep the original positive rating of this paper.", "rebuttal": "We thank the reviewer for their thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n\n> **`W2-1`**: The proposed method is evaluated only on mathematical reasoning tasks. The generalizability to other domains remains unclear.\n\nWe appreciate the suggestion. We initially followed the R1-OneVision baseline to evaluate LVLM reasoning performance on mathematical benchmarks. To further assess the generalizability of **LaCoT** beyond mathematical reasoning, we conducted additional experiments on three diverse visual understanding benchmarks: **MMMU-pro**, **MMVet**, and **MME**. These benchmarks cover a broad range of tasks, including visual commonsense, fine-grained recognition, and multi-choice QA. \n\n| Method                | MMMU-pro | MMVet | MME  |\n|----------------------|----------|-------|------|\n| InternVL2-4B         | -        | 55.7  | 2046 |\n| Qwen2.5-VL-3B        | 22.4     | 61.4  | 2134 |\n| **LaCoT-Qwen-3B**    | **28.9** | **69.6** | **2208** |\n| InternVL2-8B         | 25.4     | 60.0  | 2210 |\n| Qwen2.5-VL-7B        | 34.6     | 70.5  | 2333 |\n| R1-Onevision         | 28.2     | 71.1  | 1111 |\n| **LaCoT-Qwen-7B**    | **35.3** | **74.2** | **2372** |\t\n\nThese results demonstrate that LaCoT consistently improves performance across different domains and model scales, suggesting strong generalization beyond mathematical reasoning tasks.\n\n\n> **`W2-2`**: The proposed method requires sampling multiple rationales at inference time, which is computationally expensive. The paper can benefit from a detailed analysis of computational costs compared to baselines.\n\nWe agree that rationale sampling introduces additional inference cost, and we address this by using mini-batching (with batch size k=5) to generate N rationales in N/k forward passes. Below, we report the average per-sample inference time (reasoning + answering) and corresponding performance of different reasoning-LVLM on MathVista and MathVerse:\n\n| #Rationales (N)        | 1    | 5     | 10    | MathVista | MathVerse |\n|------------------------|------|-------|-------|-----------|-----------|\n| LLaVA-CoT-11B          | -    | 340s  | 830s  | 52.5      | 22.6      |\n| R1-OneVision-7B        | 32s  | -     | -     | 64.1      | 37.8      |\n| **LaCoT-Qwen-7B**    | -    | 30s   | 65s   | **68.4**  | **39.7**  |\n\n\nWhere LLaVA-CoT-11B utilizes stage-wise beam search [1] at inference time.\n\nAs shown in the table, LaCoT-Qwen-7B consistently achieves stronger performance, even with modest increases in inference time. Compared to other multi-rationale baselines, LaCoT strikes a favorable balance between computational cost and reasoning reliability, thereby improving both the trustworthiness of rationales and the accuracy of final answers.\n\n\n\n\n> **`Q2-1`**:  How sensitive is the performance to the choice of lambda for approximating reward?\n\nWe appreciate the valuable comment. We examined the sensitivity of performance to the choice of $\\lambda$ in reward approximation by training with values in the range [8,32]. While the approximated rewards showed slight differences in early stages, they rapidly converged and became nearly identical after around 250 training steps, suggesting that LaCoT is relatively robust to the choice of $\\lambda$ within a reasonable range.\n \n> **`Q2-2`**: What are training and inference times compared to baselines (SFT, GRPO)?\n\nWe report the total training time of fine-tuning Qwen2.5-VL using different objective functions on 3k samples, all conducted on 8×A100 (80G) GPUs: \n| Method | Training Time | Inference Time/sample | MathVista  |\n|--------|---------------|-----------------------|------------|\n| SFT    | 1 hour        | 34s                   | 62.7       |\n| GRPO   | 64 hours      | 32s                   | 62.6       |\n| RGFN (ours)   | 90 hours      | 30s                   | 66.8 (N=5) |\n\n\nWhere N indicates the number of explored rationals of BiN.\n\nTo avoid out-of-memory (OOM) issues during GRPO and LaCoT fine-tuning, we employed DeepSpeed Stage 3 with gradient checkpointing enabled. While this setting substantially increased training time, it enabled on-policy exploration.\n\nThese results demonstrate that while LaCoT incurs additional computational cost due to exploration and token-level reasoning supervision, it remains feasible for practical training pipelines and yields notable performance gains.\n\n\n\n> **`Q2-3`**: How does the proposed reference-guided filtering compare to other exploration strategies like epsilon-greedy or entropy-based exploration?\n\n\nWe studied Epsilon-greedy in our early study, but we found that it does not scale well in training a reasoning LVLM. Specifically, the action space comprises the entire vocabulary (~50,000 tokens). In such a large space, most random actions are **semantically nonsensical** and low-reward, making exploration **inefficient** and often harmful. This results in **high-variance gradients** and **slow convergence**, leading to a catastrophic forgetting issue in our experiment.\n\nEntropy-based exploration methods such as PPO rely on optimizing policy gradients to maximize expected rewards. However, these methods often struggle to align the learned policy with a target distribution, especially in complex reasoning tasks. In contrast, our approach builds on the GFlowNet framework, which is designed to sample from a target distribution via trajectory-level credit assignment, making it better suited for tasks requiring diverse and calibrated generation [2].\n\nOur reference-guided filtering (RGFN) further enhances the stability of on-policy exploration by selectively retaining high-quality trajectories, ensuring both efficiency and robustness during training. This strategy offers a more principled and controllable alternative to stochastic exploration heuristics.\n\n[1] Xu, G., Jin, P., Li, H., Song, Y., Sun, L., & Yuan, L. LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. ICCV 2025.\n\n[2] Hu, E.J., Jain, M., Elmoznino, E., Kaddar, Y., Lajoie, G., Bengio, Y., & Malkin, N. Amortizing intractable inference in large language models. ICLR 2024"}, {"review_id": "7FrrEeENFt", "reviewer": "Reviewer_tynq", "summary": "Thia paper propose a novel  approach for COT reasoning by sampling in amortized latent space. Specifically, this approach, Amortized Variational Inference (AVI) yields token- level learning through optimizing the Evidence Lower Bound (ELBO), which encourages diverse trajectories and provides a principled way to draw samples from the target posterior distribution. \nThis bayesian approach eliminates the reliance on the critic model as in BoN-related approach during rationale-solution searching at inference time.", "questions": "See Above.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper proposes a novel training framework RGFN, achieved by the reward approximation and reference-guided policy exploration and a novel bayesian Inference over latent rationals approach. \n2. Experimental results over 4 multimodal math reasoning benchmark demonstrate its effectiveness. \n3. The approach is not only effectiveness but also efficient and get rid of extra critic model during inference searching. \n\nWeakness: \n1. Although the model is compared with SFT and GRPO results. Results on popular inference scaling approaches such as BofN etc. should also be compared with?\n2. Is this approach the first to adopt Amortizing Variational Inference method, more comparison with related work is needed. Also comparison with existing latent space reasoning approaches?", "was_revised": true, "final_justification": "After reviewing the author's rebuttal, I decide to maintain my score which still holds an up-word rating about this paper with accept.", "rebuttal": "We thank the reviewer for their thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n> **`W3-1`**:  Although the model is compared with SFT and GRPO results. Results on popular inference scaling approaches such as BofN etc. should also be compared with?\n\nWe appreciate the valuable suggestion. We provide comparison results of Best-of-N (BofN) approach below. Specifically, during inference, we sample N rational-answer pairs using LaCoT. For each pair, we compute the length-normalized log-likelihood of the generated answer as reward, and then select the final answer corresponding to the highest reward. To ensure a fair comparison, we do not use any external reward model.\n\n| Method            | MathVerse | MathVista | MMMU  | MMVet |\n|------------------|:---------:|:---------:|:-----:|:-----:|\n| 3B w/ BofN        |   21.2    |   57.1    | 44.7  | 67.1  |\n| 3B w/ BiN (ours)  | **40.0**  | **63.2**  | **48.8** | **69.6** |\n| 7B w/ BofN        |   26.5    |   62.2    | 47.3  | 71.2  |\n| 7B w/ BiN (ours)  | **39.7**  | **68.4**  | **54.9** | **74.2** |\n\n\nWe set number of candidates N={5, 10} for BoN and BiN, and we report the highest score of each method.\n\n> **`W3-2`**: Is this approach the first to adopt Amortizing Variational Inference method, more comparison with related work is needed. Also comparison with existing latent space reasoning approaches?\n\n\nTo the best of our knowledge, our work is the first to apply amortized variational inference to latent visual reasoning with a long CoT chain, where reasoning steps are learned and inferred in a latent space conditioned on both visual and textual input.\n\nPrevious works ([1] and [2]) focus solely on the textual domain, applying these methods for latent visual CoT is nontrivial due to long and structured multi-step reasoning.\n\n[1] Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J.E., & Tian, Y. (2024). Training Large Language Models to Reason in a Continuous Latent Space. _ArXiv, abs/2412.06769_.\n\n[2] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B.R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. _ArXiv, abs/2502.05171_."}, {"review_id": "Sg3ChPOoan", "reviewer": "Reviewer_AnFV", "summary": "The authors propose an RL algorithm to improve on existing CoT reasoning methods for LVLMs as current popular methods, SFT & RL approaches such as PPO and GRPO, fail to generalise well and may rely on noisy or biased reward models.  SFT models tend to struggle with tasks out-of-distribution while the RL approaches are constrained in exploration and often fail to discover novel and useful rationales.  They use theory from Generative Flow Networks (section 2 & 3) to do amortized variational inference (AVI), in effect casting visual reasoning as a variational inference problem, to generate sequences of latent rationales while also introducing a token-level reward approximation that they guide their training approach with, a method they call Reference-Guided GFlowNet fine-tuning (RGFN - section 3.2).\n\nThe overall aim is improved generalisation on out-of-distribution vision-languague tasks by way of RGFN to induce exploration of better rationales.  At inference time they combine this with a  bayesian inference method (BiN) for sampling latent rationales in contrast to Best-of-N and Beam search methods which incur a high computational burden, are difficult to scale, and rely on reward models (section 3.3). The resulting model is the LaCoT trained from open pre-trained models (Qwen2.5-VL-3B & 7B) on a mixture of visual reasoning datasets.\n\nKey to this approach (section 3.1) is token level reward modeled on the joint likelihood of the latent rationales with the question/answer pairs (X,Y) where linear interpolation provides reward estimates for latent tokens along the sequence. During training a reference rationale helps guide training toward better rationales vis-a-vis the rationale sampler and reward model (section 3.2).\n\nIn section 4 the approach is evaluated where LoRa is used to finetune the Qwen2.5-VL open models.  In 4.2 LaCot is compared to SFT & GRPO baselines on Math and reasoning benchmarks: MathVista, MathVerse, MMMU. LaCot archives SOTA (Table 1) among the open source models in these tasks, boosting diagram comprehension and OCR robustness while also yielding better diversity and generalization through its proposed rationales.  Further ablations demonstrate that SFT and GRPO fail to generalise well to more complex tasks and especially in cases where the reward model is bad or the KL limits effective exploration.", "questions": "I couldn't see where the reference rationales come from?  Could you explain that?  If it hasn't been included would it be worth briefly touching on this?\n\nDid you have any issues with hallucinations in your rationales during inference?  Presumably these would be more problematic as the sample size is lowered in BiN? \n\nWhen doing token level reward approximation I was a bit uncertain where rewards we actually be sampled vs. interpolated from Eq. 4, can you expand on this a bit?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nGenerating useful rationales from visual queries with LVLMs is an open area of research and currently is the focus of ongoing efforts to drive improvements to multimodal foundation models [1,2].  Rationalising over both vision and language modalities adds complexity to the problem of generating admissible CoT while avoiding hallucinations by the model  which will need to be overcome in the methods proposed in this paper.  Reasoning well from visual observation will shed light on how to approach problems in multimodal decision making, visual scene understanding, and inferring insights from abstract imagery, all of which require strong abstraction and generalisation priors to be successful.\n\nThis work leverages a set novel strategies with that aim to generate useful rationales on which a generative model can be conditioned such that scores on reasoning benchmarks are improved overall. The authors focus on good generalisation, constraining model exploration to avoid catastrophic forgetting though fine tuning, and an inference scaling method that draws on the variational inference model.  These yield the following improvements over prior approaches:\n\n * The use and adaptation of Generative Flow Networks to cast the problem of generating rationales in terms of amortized variational inference.\n * Token level reward approximation based on the joint likelihood of the proposed rationales.   Under the assumption that the reward is locally smooth the authors devise a method to save costly computations via linear interpolation of rewards corresponding to latent states.  * Overall this works to improve sampling diversity of the rationales and helping to explore during training.\n * Bayesian Inference to draw latent samples from the target posterior at inference time which in favour Best-of-N and Beam Search methods which are computationally costly, are difficult to scale, and rely on reward models.  This approach extends natural from the AVI framework used in LaCoT.\n * Policy exploration is constrained by reference latent to overcome catastrophic forgetting and encourage more diversity in rationale exploration.\n\nOverall flow of the paper is good, although I do have  few critiques on this also here which I will address below.  The big positives for me are the thorough explanation of the theory behind the overall approach and the creative way that the authors have brought together a range of methods to tackle a difficult problem.  The results presented in tables 1 & 2 and figures 5 & 6 demonstrate the efficacy of this approach considering the inference method, the number of rationales sampled, ablating RFGN, and overall reasoning scores over a wide range of models.\n\n**Weaknesses**\n\nAs mentioned above, while I found some of the aspects of this paper clearly presented and compelling, I think some parts could be made clearer.  \n\nThe biggest challenge that I had when reading this paper was that, although there are a number of good ideas put into the approach, it took me a few passes to really see how each component tied together into the holistic approach. I think this could possibly be addressed through some high level treatment of your approach, perhaps detailed at the beginning of section 3 with the potential inclusion of a new figure and that underlines the key fundamental strengths of the approach that you propose.\n \nThen in Section 3.3 I see the approach in general however I think it could benefit from a bit more detail in the figures and main text.  Also some expansion or more description of figure 3 could help, perhaps  complemented by an algorithmic element and some explanation (if I haven't missed it) as to whether we can expect higher quality reationales from BiN when comparing to other inference methods - maybe somecomparisons of concrete examples.\n\nOne nitpick, In the intro on the second to last paragraph you mention three points that overlap with your main contributions.  Could you simply include this all under one list? It may make things a little clearer. \n\nFinally, and this is probably out of scope for this work, but it may strengthen this work to consider LaCoT on benchmark domains targeting decision making and acting responses (e.g. E3VQA, MMIU [3,4]) that rely on some scene understanding and reasoning from world priors. \n\n[1] Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao. 2025. VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning. arXiv preprint arXiv:2504.07956 (2025).\n\n[2] Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang, Zheming Yang, Jian Xu, and Minghui Qiu. Reinforced mllm: A survey on rl-based reasoning in multimodal large language models. arXiv preprint arXiv:2504.21277, 2025.\n\n[3] Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718, 2024\n\n[4] I Lee, W Park, J Jang, M Noh, K Shim, B Shim. Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs. arXiv preprint arXiv:2505.21955, 2025•", "was_revised": true, "final_justification": "This work addresses generalisation over problems that require reasoning in LVLMs which is a relevant and challenging issue with a focus on highly referenced mathematics benchmarks. This paper presents the problem in terms of variational inference and combines a range of techniques into a novel methodology which the authors effectively apply across empirical benchmarks.  The work itself is compelling and could feasible lead to extensions that further explore generative rationales in multimodal foundation models.", "rebuttal": "We thank the reviewer for their thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n> **`W4-1`**: Perhaps detailed at the beginning of section 3 with the potential inclusion of a new figure and that underlines the key fundamental strengths of the approach that you propose...; Then in Section 3.3 I see the approach in general however I think it could benefit from a bit more detail in the figures and main text. ...; One nitpick, In the intro on the second to last paragraph you mention three points that overlap with your main contributions. Could you simply include this all under one list? It may make things a little clearer.\n\n- We will add a figure at the beginning of section 3 to clearly show our motivation.\n- Figure 4 visualizes Equation 8, illustrating how BiN performs Bayesian marginalization over latent chain-of-thought trajectories Z to approximate the likelihood of the answer P(Y|X). We agree that additional detail would improve clarity, and we will revise both the figure and the accompanying explanation in the final version to better highlight the role of latent reasoning and its connection to our overall inference pipeline.\n- We will revise the introduction and consolidate the overlapping points into a unified contribution list to improve clarity and readability in the final version.\n\n> **`W4-2`**: Consider LaCoT on benchmark domains targeting decision making and acting responses (e.g., E3VQA[1], MMIU[2]) that rely on some scene understanding and reasoning from world priors.\n\nWe appreciate the suggestion. We attempt to evaluate on **MMIU**, but find that LaCoT and our baseline models (e.g., Qwen2.5-VL) struggle to generalize due to their limited training on single-image reasoning tasks. Regarding **E3VQA**, the dataset was not publicly available at the time of writing, which prevented us from conducting a fair comparison.\n\nAlthough we are unable to follow the reviewer's suggestion, we conducted additional experiments on three other visual understanding benchmarks: **MMMU-pro**, **MMVet**, and **MME**. These benchmarks cover a broad range of tasks, including visual commonsense, fine-grained recognition, and multi-choice QA. \n\n| Method                | MMMU-pro | MMVet | MME  |\n|----------------------|----------|-------|------|\n| InternVL2-4B         | -        | 55.7  | 2046 |\n| Qwen2.5-VL-3B        | 22.4     | 61.4  | 2134 |\n| **LaCoT-Qwen-3B**    | **28.9** | **69.6** | **2208** |\n| InternVL2-8B         | 25.4     | 60.0  | 2210 |\n| Qwen2.5-VL-7B        | 34.6     | 70.5  | 2333 |\n| R1-Onevision         | 28.2     | 71.1  | 1111 |\n| **LaCoT-Qwen-7B**    | **35.3** | **74.2** | **2372** |\t\t\n\nThese results demonstrate that LaCoT consistently improves performance across different domains and model scales, suggesting strong generalization beyond mathematical reasoning tasks.\n\n\n> **`Q4-1`**: I couldn't see where the reference rationales come from? Could you explain that? If it hasn't been included would it be worth briefly touching on this?\n\nEach training sample in our dataset consists of a tuple {image, query, CoT, answer}, where the CoT (Chain-of-Thought) serves as the reference rationale for the answer. These rationales are generated by teacher models such as **GPT-4o** or **Deepseek-R1**, depending on the data source. We will clarify this detail in the revised version of the paper for better transparency.\n\n> **`Q4-2`**: Did you have any issues with hallucinations in your rationales during inference? Presumably these would be more problematic as the sample size is lowered in BiN?\n\nWe appreciate the reviewer’s valuable comment. We observed some hallucination issues in generated rationales when the sample size N=1, where BiN can occasionally produce incorrect or misleading reasoning steps on the **MMMU** dataset.\n\nTo assess this, we used **one-shot reasoning** as a baseline and varied the number of sampled rationales N during inference. As shown below, increasing N from 1 to 5 significantly mitigates hallucination and improves answer accuracy:\n\n| MathVerse     | 1    | 5    | 10   |\n|---------------|------|------|------|\n| one-shot reasoning     | 31.5 | -    | -    |\n| LaCoT-Qwen-3B | 33.7 | 37.2 | 40.0 |\n\n| MMMU          | 1    | 5    | 10   |\n|---------------|------|------|------|\n| one-shot reasoning     | 47.1 | -    | -    |\n| LaCoT-Qwen-3B | 44.7 | 48.7 | 48.8 |\n\nThese results suggest that **hallucination can indeed be more pronounced at smaller N**, but BiN’s sampling strategy is effective in addressing it as N increases. We will include additional qualitative examples in the final version to illustrate this effect.\n\n> **`Q4-3`**: When doing token level reward approximation I was a bit uncertain where rewards we actually be sampled vs. interpolated from Eq. 4, can you expand on this a bit?\n\nWe compute the actual reward for every $\\lambda$ steps, and apply linear interpolation to approximate the intermediate token-level rewards.\nThis design reflects a trade-off between accuracy and efficiency: while $\\lambda$=1 (i.e., no interpolation) would yield slightly better performance, it is computationally prohibitive in practice. Under our current setting, computing token-wise rewards with $\\lambda$=1 takes **~50 minutes per sample due to the long visual CoT sequences**. This requires **~400 hours of training for one epoch** on a 3k-sample dataset using 8*80G GPUs. Thus, our interpolated approximation provides a practical and effective alternative that maintains strong performance while significantly reducing training time.\n\n\n\n\n[1] Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718, 2024\n\n[2] I Lee, W Park, J Jang, M Noh, K Shim, B Shim. Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs. arXiv preprint arXiv:2505.21955, 2025"}]

Author rebuttal:

Meta-review: This paper proposes a variational learning algorithm for reasoning in vision-language models in which chains-of-thought are treated as a latent variable, following prior work in latent CoT for text tasks. Performance is compared against SFT and GRPO on visual reasoning tasks including for math/abstract reasoning and visual commonsense. This appears to be an elegant work and I suggest updating the paper with the feedback from all reviewers, but especially AnFV, in mind.
Decision: Accept (poster)