Title: SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision language model; reinforcement finetuning; vlm reasoning; data selection
Abstract: We introduce ThinkLite-VL, a family of visual reasoning models that achieve state-of-the-art (SoTA) performance using an order of magnitude fewer training samples, relying purely on reinforcement fine-tuning (RFT) self-improvement without any knowledge distillation. Our central insight is that sample difficulty critically influences RFT effectiveness: appropriately challenging examples can drive substantial reasoning improvements, even in low-data regimes.   However, quantifying sample difficulty in a reliable and scalable manner remains non-trivial. To address this, we repurpose Monte Carlo Tree Search (MCTS) to measure sample difficulty via the number of reasoning iterations a vision-language model (VLM) requires to solve each instance. This MCTS-based selection procedure identifies samples that induce deeper reasoning while remaining solvable, allowing us to filter a high-quality subset from 70k open-source examples spanning math, natural image understanding, and chart comprehension.  Using this approach, we select just 11k challenging samples for RFT on Qwen2.5-VL-7B-Instruct and 7.5k samples for Qwen2.5-VL-72B-Instruct. The resulting models, ThinkLite-VL-7B and ThinkLite-VL-72B, significantly outperform their respective base models across eight visual reasoning benchmarks.  In particular, ThinkLite-VL-7B improves the average performance of Qwen2.5-VL-7B-Instruct by 7\% and surpasses all existing 7B-level models, as well as much larger models such as GPT-4o, O1 and Qwen2.5-VL-72B, achieving a new SoTA score of 75.1 on MathVista.  ThinkLite-VL-72B further advances the SoTA frontier, achieving an accuracy of 79.7 on MathVista and an average benchmark improvement of 4.42 over the open-source SOTA.  These results demonstrate that MCTS-guided difficulty filtering provides a scalable and effective path toward data-efficient self-improvement in multimodal reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Summary and Contribution\n\n- Sample difficulty is critical for effective VLM self-improvement—appropriately challenging examples that require deeper reasoning but remain solvable can drive substantial improvements even with limited data.\n- The authors repurpose Monte Carlo Tree Search (MCTS) to measure sample difficulty by counting how many reasoning iterations a base VLM needs to solve each training example. This allows them to rank and filter the most optimally challenging samples.\n- Starting with 70k open-source samples across math, natural images, and charts, they select only 11k samples (7B model) and 7.5k samples (72B model) based on MCTS difficulty scores, then apply pure RFT without knowledge distillation.\n- ThinkLite-VL-7B achieves 75.1% on MathVista (new state-of-the-art), outperforming much larger models like GPT-4o and O1. ThinkLite-VL-72B reaches 79.7% on MathVista.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- While model performance typically scales with data size, not all researchers have access to massive datasets and compute resources. This paper demonstrates that strategic sample selection can achieve state-of-the-art results with orders of magnitude fewer training examples, making high-performance VLM training more accessible.\n- The use of MCTS to quantify sample difficulty through reasoning iteration counts is creative and theoretically motivated, providing a principled alternative to random sampling or simple heuristics.\n- The authors include valuable ablations comparing their approach against self-consistency methods, online data selection, random sampling, and analysis of unsolved problems, demonstrating the benefits of their MCTS-guided approach.\n- Achieving new state-of-the-art on MathVista while using dramatically fewer training samples is impressive and practically significant.\n\n### Weaknesses\n\nWhile I do acknowledge multiple ablations that authors presented above, they are still lacking some key ablations. I discuss them in detail below. \n\n- Please correct me if I am wrong, but the self-consistency baseline appears to use direct answers rather than chain-of-thought reasoning. A fair comparison should include self-consistency with CoT prompting to isolate the benefits of MCTS-guided selection.\n- The self-consistency baseline in Table 3 uses 23K samples while the proposed method uses 11K. This makes direct comparison difficult—the authors should compare methods using identical sample counts (11K) to isolate the effect of selection strategy. They could either do random sampling from that 23K, or sort with increasing order of solve-rate and select first 11K. This is important, as they themselves report, training on FullSet performs worse than on carefully selected samples. \n- The authors claim RFT is sufficient, but don't compare against simple supervised fine-tuning on the same MCTS-selected samples. The performance gains might stem from good data curation rather than the RL training approach.\n- The improvement over self-consistency (64.18% vs 63.15%) appears modest. The authors should provide confidence intervals or significance tests to establish whether improvements are meaningful.\n- While claiming \"no SFT,\" the authors use Instruct models that have already undergone supervised fine-tuning on similar data. Testing on truly base models would better isolate the RFT contribution.\n- The online data selection baseline selects samples with accuracy 0 to 0.9, but doesn't justify why harder problems (lower accuracy) weren't prioritized, which seems counterintuitive given their difficulty-focused approach.\n\nMinor Comments:\n- The paper claims both 7B and 72B models achieve \"new SOTA\" on MathVista, but if 72B outperforms 7B, only one can be truly state-of-the-art. This should be clarified as \"SOTA within model class\" or similar.\n- The related work states they focus on \"large-scale RL,\" but the core contribution is data efficiency with small-scale, carefully selected samples. This framing mismatch should be resolved.\n- Lack of discussion of their limitations in conclusion section.\n\nQUESTIONS:\n- Why do you think the performance when trained on all 70K worse than in the subset of dataset? \n- Can you provide a fair comparison by re-running the self-consistency baseline using exactly 11K samples (same as your method) instead of 23K? Additionally, can you clarify whether your self-consistency baseline uses chain-of-thought reasoning or direct answers, and provide results for both variants?\n- How much of your performance gain comes from MCTS-guided sample selection versus the RFT training approach? Can you compare RFT vs. supervised fine-tuning on the same MCTS-selected samples to isolate these effects?\n- The improvement over self-consistency (64.18% vs 63.15%) appears modest. Can you provide confidence intervals or statistical significance tests to establish whether these improvements are meaningful?\n- While you claim \"no SFT,\" you use Instruct models that have already undergone supervised fine-tuning. Can you test your approach on truly base models (non-Instruct) to better isolate the RFT contribution and validate your claims about self-improvement?\n- Why does your online data selection baseline select samples with accuracy 0 to 0.9 rather than prioritizing the hardest problems (lowest accuracy)? This seems counterintuitive given your difficulty-focused approach—can you explain this choice and show results with harder problem selection?\n\nLIMITATIONS:\nNo, their conclusion section currently does not contain any discussion of limitations. I encourage the authors to appropriately discuss the limitations, based on some of the weaknesses I highlighted above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ThinkLite-VL, which utilizes MCTS to sample data and then fine-tune through RFT. These results show that MCTS based difficult filtering can provide a scalable and effctive path thwards data-efficient self-improvement.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1.\tThis paper focuses the problem of sample difficulty, which might be the core problem of RFT. \n2.\tThe proposed model seems to be generally effective across lots of benchmarks.\n## Weaknesses:\n1.\tAlthough sampling data is an important topic in general LLM training, this whole paradigm looks heuristic and empirical. A concrete limitation is that this framework cannot work if MCTS cannot find the correct answer after searching, where the MCTS search may not be meaningful there. This pipeline seems to be naïve and not novel. Although the proposed data sampling might be beneficial, it is only one choice, and there could be many possibilities uncovered.\n2.\tAbout the contributions. In the paper, (3) Data-efficient and (4) Strong empirical gains seem to be a duplicate. The fifth contribution open-source release isn’t clear. The authors do not mention whether they would release the training code or only the trained model. Also, authors would better submit some code to ensure or online demos for the reviewer to assess this contribution.\n3.\tWriting issues: \n(1)  Figure 1 and Figure 2 seem to be repetitive. \n(2)  Sampled data size through MCTS should be listed in Figure 3.\n\nQUESTIONS:\n1.\tHow does Figure 3 show open-ended format is critical. The caption seems not to be relevant.\n\n2.\tIt seems that the proposed technique can be generally used in cross modal training. Have the authors tried this?\n\n3.\tWould the model forget old simple questions after only learning the difficult ones?\n\nLIMITATIONS:\nThis part is relatively sound and the authors mention a meaningful limitation in the overall training efficiency. The reviewer does not penalize the corresponding limitation", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ThinkLite-VL, a methodology for enhancing the visual reasoning capabilities of Vision-Language Models (VLMs) through Reinforcement Fine-Tuning (RFT) using a significantly reduced number of training samples. The core contribution is a novel sample selection strategy that repurposes Monte Carlo Tree Search (MCTS) to quantify sample difficulty. This MCTS-based procedure measures difficulty by the number of reasoning iterations a base VLM (Qwen2.5-VL series) requires to correctly solve an instance. From an initial pool of 70k open-source examples, the authors filter high-quality subsets of 11k (for a 7B model) and 7.5k (for a 72B model) samples. These are then used for RFT, reportedly enabling ThinkLite-VL models to achieve state-of-the-art (SoTA) performance on several visual reasoning benchmarks, particularly MathVista, without explicit knowledge distillation during the RFT phase. The authors frame this as a data-efficient *self-improvement* process, though this improvement is critically guided by the external MCTS filtering stage.\n\nSTRENGTHS AND WEAKNESSES:\n$$\\textbf{Strengths}$$\n\n- The use of MCTS iterations, derived from a base model's reasoning process, to estimate sample difficulty for curating RFT training data is an intriguing approach. \n\n- The paper demonstrates strong performance improvements using remarkably few samples (11k for 7B, 7.5k for 72B) for the RFT stage, a significant reduction compared to many existing methods that require larger datasets for similar fine-tuning efforts. \n\n- ThinkLite-VL models achieve compelling results on eight VLM benchmarks, notably setting new SoTA scores on MathVista (75.1% for 7B, 79.7% for 72B), even outperforming some much larger models and proprietary systems. \n\n- The success of the RFT phase without requiring knowledge distillation from larger teacher models is a notable advantage, potentially simplifying the training pipeline for reasoning enhancement. \n\n- The paper not only finds that MCTS-derived difficulty is model-specific but also provides strong empirical justification in its appendix (Table 9) for its data selection strategy. It demonstrates that a specific mix of medium-difficulty (*Iter5+*) and very hard (*Unsolved*) samples yields optimal performance, outperforming both narrower and broader data subsets. \n\n---\n\n$$\\textbf{Weaknesses}$$\n\n- The paper's central claim of efficiency (*SOTA with Less*) pertains only to the number of RFT samples. The appendix reiterates that the MCTS filtering stage requires *substantial time* but provides no quantification of this cost. Knowing the GPU hours will help assess the method's true end-to-end computational efficiency. \n\n- The appendix reveals specific prompts and fixed parameters without exploring their sensitivity. The paper does not investigate how performance is affected by key MCTS hyperparameters.\n\n- The released datasets have limited *off-the-shelf* utility for different VLM architectures. Widespread adoption would require users to re-run the unquantified, potentially expensive MCTS filtering process for their specific models, hindering scalability.\n\nQUESTIONS:\n- Q1. The appendix mentions that MCTS filtering requires *substantial time*. To assess the method's overall efficiency, could you please provide a concrete breakdown of the computational cost (e.g., total GPU hours) for the MCTS filtering of the initial 70k samples, and compare this to the cost of the subsequent RFT phase? \n\n- Q2. How sensitive is the data selection process and final model performance to this constant and other choices like the expansion temperature and the specific critic model/prompt? \n\n- Q3. The training and test sets are largely distinct, but to rigorously test generalization, have the authors considered a stricter out-of-distribution experiment? For example, holding out all datasets from one category (e.g., *Chart Understanding*) during MCTS filtering and RFT, and then evaluating performance on benchmarks from that unseen category (e.g., TabMWP, IconQA if they have separate test splits not used elsewhere). \n\n- Q4. Do you have recommendations on when the provided curated datasets can be effectively used versus when a full, new MCTS filtering run is necessary?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a data selection method for data-efficient reinforcement learning training of visual large language models. Specifically, the authors filter training data based on offline difficulty measured by the number of MCTS iterations the model needs to solve questions correctly. Experimental results demonstrate that training models on their selected datasets achieves leading performance across multiple benchmarks on both 7B and 72B models while using significantly less training data. Ablation studies attempt to show that MCTS-based sample selection is crucial, leading to superior performance over other offline and online data selection methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is clearly written with good logical flow, clear figures, and well-designed experimental result tables that effectively communicate the methodology and findings.\n- Extensive experimental validation across 8 benchmarks demonstrates that models trained on significantly less data (11k for 7B, 7.5k for 72B) achieve leading performance, showing impressive training efficiency and effectiveness of the proposed approach.\n\nWeaknesses:\n\n- The experimental setup for important baselines—particularly the self-consistency offline and online data selection methods—lacks clarity, raising concerns about the validity of experimental results and analysis in sections 4.2 and 4.3. Since MCTS sampling includes step-by-step reasoning before final answers, self-consistency baselines should also incorporate chain-of-thought reasoning rather than direct answers. This critical setting is not clearly specified, potentially leading to unfair comparisons.\n- Several technical details about the self-consistency baselines are unclear or potentially incorrect: For offline self-consistency (line 282), samples with accuracy lower than 0.2 across 50 rollouts are selected—does \"accuracy\" refer to \"mean accuracy@50\"? Following standard practices in methods like DAPO, samples with 0 < mean accuracy@k < 1 are typically selected, suggesting the threshold should be \"higher than or equal to 0.02\" rather than \"lower than 0.2.\" The rationale for this threshold selection is not provided. Similarly, for online self-consistency (line 297), the number of rollouts per sample is missing, and the upper bound of 0.9 rather than 1.0 lacks justification.\n- The paper lacks discussion of inference costs for MCTS-based data sampling compared to self-consistency methods, which is crucial for evaluating practical applicability.\n- Additionally, to fairly compare offline versus online data selection, the authors should evaluate MCTS-based sampling in an online setting. Without this comparison, the conclusion that \"online filtering offers negligible improvement except faster convergence\" cannot be generalized across all data selection methods.\n- The motivation for applying this method specifically to visual large language models is unclear. The MCTS-based data sampling approach appears equally applicable to text-only model training, yet the paper lacks discussion of VLM-specific design considerations or challenges. It would be valuable to understand whether this method would achieve similar performance improvements when applied to text LLM training compared to other data selection methods, which would help establish the method's broader applicability and the specific value proposition for multimodal models.\n\nQUESTIONS:\nPlease refer to the concerns raised in the weakness section.\n\nLIMITATIONS:\nYes, the authors discuss their limitations in Appendix D.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}