Title: NaDRO: Leveraging Dual-Reward Strategies for LLMs Training on Noisy Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Combinatorial Optimization

Preference Learning

Process Reward
Abstract: Group Relative Policy Optimization (GRPO) fine-tuning has demonstrated significant enhancements in reasoning tasks. However, it often relies on high quality labeled dataset, which is typically difficult to obtain. To address this challenge, we introduce \textbf{N}oise-\textbf{A}ware \textbf{D}ual-\textbf{R}eward \textbf{O}ptimization (\textbf{NaDRO}) to  effectively enhances the training of Large Language Models (LLMs) under noisy or ambiguous supervision. NaDRO operates through two key components: \textbf{(1) Preference-based Outcome Reward (POR)},which makes a principled bias-variance tradeoff, reducing training variance by learning from robust preference rankings instead of overfitting to single-best estimates; and \textbf{(2) Context Perception Reward (CPR) mechanism}, which ensures that LLMs conduct necessary qualitative assessment of the current problem state to foster deeper situational understanding prior to decision-making. To validate our approach in a realistic decision-making testbed, we model classic combinatorial optimization problems like the Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP) as Markov Decision Processes, generating training data via cost-limited exploration. Our results demonstrate that the fine-tuned Qwen 7B and Llama 3.1-8B models achieve statistically robust performance, significantly outperforming leading LLM baselines and standard fine-tuning methods on these complex benchmarks. Code is released at \url{https://github.com/microsoft/HeurAgenix/tree/NaDRO}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to help LLM learn effectively from noisy or incomplete data. The proposed algorithm, NaDRO, which is built on dual rewards: preference-based outcome reward and context perception reward, synergistically optimizes learning signals from both outcome preference and intermediate reasoning process. The authors validate the empirical performance on two specific problems: travelling salesman problem and capacitated vehicle routing problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe paper is well-written, with a clear and logical flow that makes it easy to follow. The methodology is explained in detail, providing a solid understanding of the approach. The ablation study shows the importance of each component clearly.\n\nWeaknesses: \n1. The ablation studies clearly demonstrate that the dual-reward strategy improves performance. However, it is better for the authors to more explicitly articulate the intuition or theoretical basis for combining these specific rewards.\n2. Given that the paper does not provide a theoretical analysis for NaDRO, the empirical results become only evidence supporting the effectiveness of the method. Therefore, it is better to add even a minimal variance analysis across different random seeds to demonstrate that the observed performance improvements are robust.\n3. In the experiments, the comparison methods are appropriately cited. However, it would be helpful to add a brief explanation of the algorithms, for instance, clarifying that LKH stands for ‘Lin-Kernighan-Helsgaun’ and is a widely used heuristic for solving the TSP.\n\nQUESTIONS:\nQuestions: \n1. While the NaDRO outperforms other LLM-based methods on MCTS-generated data, the conclusion in claiming offering a promising pathway towards real-world noisy data handling is not fully justified by current experiments. I would suggest adding a brief discussion on what types of real-world noise might be relevant and how NaDRO could be adapted or expected to handle them.\n2. The paper adopts fixed thresholds (1.15 and 0.8) for labeling high and low cost in CPR annotation, but it’s unclear how these specific values were determined.\n3. The empirical evaluation does not demonstrate a clear advantage over classical baselines such as LKH in TSP, which raises concerns about the practical relevance of using LLMs for these tasks. What is the advantage of LLM-based methods in such tasks and when would such method be preferable to strong classical solvers?\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes NaDRO (Noise-Aware Dual-Reward Optimization), a framework for fine-tuning LLMs in noisy training environments. It introduces two reward mechanisms:\n\n1. Preference-based Outcome Reward (POR): Guides learning using relative rankings of actions instead of relying on potentially noisy absolute scores from sources like Monte Carlo Tree Search (MCTS).\n\n2. Context Perception Reward (CPR): Provides intermediate supervision by encouraging the model to qualitatively assess the problem state before making decisions.\n\nThe framework is tested on combinatorial optimization tasks (TSP and CVRP), and shows improved performance over strong baselines, including GPT-4o, GRPO-finetuned models, and traditional metaheuristics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses the practical problem that finetuning LLM for combinatorial optimization problem. The problem is important and challenge.\n2. The use of relative preferences (POR) is well justified as a noise-robust alternative to direct score regression. CPR encourages process-level reasoning, which adds interpretability and structure.\n3. NaDRO-fine-tuned models show substantial gains on TSP and CVRP benchmarks, even outperforming larger LLMs in some cases.\n\nWeaknesses:\n\n1. The primary weakness of the paper is that the proposed approach is highly heuristic. The main contribution lies in the design of the reward signals (POR and CPR), but these are introduced without accompanying theoretical analysis or systematic empirical justification beyond the benchmark results. There is no ablation on the design of the rewards, nor any attempt to analyze their properties or limitations.\n\n2. Most baseline comparisons are made against publicly available LLMs (e.g., GPT-4o, DeepSeek) without task-specific fine-tuning on TSP or CVRP. Throughout the experimental section, only one model—GRPO-Qwen—is fine-tuned on the dataset. Given that the authors provide results for NaDRO-Llama-8B, it is unclear why corresponding results for Llama-8B (zero-shot) and GRPO-Llama-8B (fine-tuned using existing methods) are omitted. These comparisons are important: the former would clarify the benefit of any fine-tuning, while the latter would isolate the effect of the new reward design relative to established methods.\n\n3. Most critically, the paper lacks a proper ablation study isolating the effects of the two proposed reward components. Without such analysis, it is difficult to assess whether the POR and CPR signals meaningfully contribute to the observed improvements. Since the paper is largely empirical in nature, the absence of such foundational validation weakens the contribution. As it stands, the work does not meet the scientific rigor expected of a NeurIPS paper.\n\n4. Throughout the paper, the citation format is inaccurate. The authors should use \\citep{} when the citation is not a part of the sentence.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nI didn't find the discussion of limitation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework called NaDRO, which combines two mechanisms—Preference-based Outcome Reward (POR) and Context Perception Reward (CPR)—to train large language models (LLMs) using noisy Monte Carlo Tree Search (MCTS) data, addressing decision-making noise in combinatorial optimization problems. The experimental section validates NaDRO's effectiveness by comparing multiple baseline models (e.g., Qwen 7B, Llama 3-8B) and demonstrates performance advantages in tasks such as TSP and CVRP. The paper is well-structured, the framework design is practical, and the code and data are open-sourced.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n--Practical Problem Setting: The paper focuses on the challenge of training LLMs on noisy data, particularly for long-sequence decision-making tasks in combinatorial optimization (e.g., TSP/CVRP), which is highly relevant to real-world applications.\n--Reproducible Methodology: The method is thoroughly designed and reproducible. Appendices provide complete implementation details (state/action space definitions, CPR annotation rules, hyperparameter configurations), and the code is open-sourced (Appendices A-F cover data generation, training pipeline, and parameter settings).\n--Comprehensive Experimental Validation:\nEvaluated on two classical problems (TSP and CVRP).\nExtensive baseline comparisons with significant results.\n\nWeaknesses:\n--Lack of Theoretical Analysis:\nNo convergence proof for POR/CPR in policy optimization or theoretical bounds on noise robustness (only heuristic experiments in Figure 3).\n--Potential Bias in CPR Labeling:\nCPR labels in Appendix C are based on greedy solution thresholds (±15%–20%), but the universality of this rule for decision quality is not validated, possibly introducing bias.\n--Limitations in Experimental Validation:\nNo Statistical Significance: Results (e.g., Tables 1–3) lack standard deviations/confidence intervals, making it unclear whether performance differences are significant (authors acknowledge limited computational resources for multiple runs, see Page 27 Q7).\nNarrow Noise Scope: Only MCTS-simulated noise is tested (Section 3.2), without real-world noise (e.g., crowdsourced or expert disagreement noise), weakening generalization claims.\nInsufficient Baseline Comparisons: Missing comparisons with state-of-the-art noise-robust methods (e.g., Robust RLHF by Shen et al. 2024), limiting the evaluation to basic GRPO.\n--Ablation Study Ambiguity:\nIn ablation experiments (Table 3), the performance gap between POR and GRPO narrows as TTS level increases. Would this gap further diminish with TTS level > 10?\n\nQUESTIONS:\n--Theoretical Clarifications:\nProvide error bounds for POR under noisy distributions or prove CPR’s impact on policy gradient optimization.\n--Enhanced Experimental Rigor:\nReport standard deviations for multi-run experiments.\nTest robustness in additional noise scenarios (e.g., artificially injected noise).\nInclude comparisons with SOTA noise-robust methods.\n--Extended TTS Analysis:\nSupplement experimental results for TTS level > 10 to clarify performance trends.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes NaDRO, a framework for fine-tuning LLMs to perform well on long-horizon decision-making tasks using noisy, imperfect supervision—particularly from sources like MCTS (Monte Carlo Tree Search). This is fundamentally a challenge, since it is hard to estimate the long-term value of actions: scores are inherently noisy, being estimated from a tiny fraction of the action space. The commonly used methods (e.g., GRPO) require high-quality labeled data. NaDRO addresses this by introducing a dual-reward formulation: it constructs more robust training signals by splitting rewards into a Preference-based Outcome Reward (POR) and a Context Perception Reward (CPR). Rather than relying on absolute action scores, NaDRO leverages pairwise preference rankings to construct POR and combines it with CPR in an alternating optimization scheme, enabling the model to generalize better in the face of sparse, noisy, or inconsistent reward signals.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* Reward Decomposition Strategy: The paper proposes a technically sound reward decomposition—splitting noisy feedback into a Preference-based Outcome Reward (POR) and a Context Perception Reward (CPR)—which improves gradient stability and enables more robust training from noisy, partial supervision.\n* Rank-Based POR Mechanism: Instead of relying on unstable top-1 MCTS scores, POR converts noisy Q-values into ordinal rankings and applies position-sensitive scaling, preserving learning signal while mitigating variance—this feels like a principled alternative to argmax-based distillation.\n* State-Level Supervision via CPR: CPR provides intermediate, context-sensitive reward signals during trajectory generation, helping to stabilize training and improve credit assignment in long-horizon reasoning tasks where final rewards are sparse or noisy.\n* Comprehensive Evaluation: The experiments benchmark NaDRO across TSP and CVRP using multiple LLM backbones, including DeepSeek variants, and show improvements over strong baselines like GRPO and RLAIF. Ablation studies further isolate the roles of POR and CPR, confirming that both components contribute to final performance\n\n# Weaknesses\n\n* Heuristic CPR Labels May Limit Generality: The CPR supervision relies on handcrafted heuristics (e.g., binning cost/tour length into categories), which introduces labeling bias and may not generalize to domains where such features are unavailable or noisy.\n* No Explicit Credit Assignment Analysis: While CPR may help credit assignment, the paper lacks analysis showing whether it improves learning dynamics or sample efficiency, particularly in longer-horizon settings.\n* No Sensitivity Analysis of POR: The paper does not evaluate how sensitive POR is to the choice of rank cutoffs (e.g., top-k) or to the quality of the MCTS rankings. It is unclear how robust POR remains if ranking noise causes mislabeling near the decision threshold.\n\nQUESTIONS:\n# Questions\n\n1. How does CPR contribute specifically to solving long-horizon credit assignment, beyond acting as an auxiliary classification loss?\n    While CPR improves performance, it’s unclear whether its benefits stem from better credit assignment over multi-step trajectories, or simply from regularizing the model with intermediate supervision. Could one isolate its impact by measuring reward variance or learning dynamics with/without CPR over varying trajectory lengths?\n2. How sensitive is POR to the rank cutoff parameters and MCTS quality?\n    Since POR transforms noisy Q-values into ordinal rankings and selects top-k actions for positive supervision, could a poorly chosen cutoff degrade performance by discarding near-optimal actions that fall just below the threshold?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}