Title: Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Curriculum Design

Large Language Models

Reinforcement Learning

Chain-of-Thought Reasoning
Abstract: Training agents to operate under strict constraints during deployment, such as limited resource budgets or stringent safety requirements, presents significant challenges, especially when these constraints render the task complex. In this work, we propose a curriculum learning strategy that gradually tightens constraints during training, enabling the agent to incrementally master the deployment requirements. Inspired by self-paced learning techniques in unconstrained reinforcement learning (RL), our approach facilitates a smoother transition to challenging environments by initially training on simplified versions of the constraints and progressively introducing the full deployment conditions. We provide a theoretical analysis using an RL agent in a binary-tree Markov Decision Process (MDP) to demonstrate that our curriculum strategy can accelerate training relative to a baseline approach that imposes the trajectory constraints from the outset. Moreover, we empirically validate the effectiveness and generality of our method across both RL and large language model (LLM) agents in diverse settings, including a binary-tree MDP, a multi-task navigation domain, and a math reasoning task with two benchmarks. These results highlight the potential of curriculum design in enhancing the efficiency and performance of agents operating under complex trajectory constraints during deployment. Moreover, when applied to LLMs, our strategy enables compression of output chain-of-thought tokens, achieving a substantial inference speedup on consumer hardware, demonstrating its effectiveness for resource-constrained deployment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests a curriculum learning strategy to learn the textual reasoning chain-of-thoughts limited to a certain target length (in tokens). The proposed CURL-TRAJC strategy is based on the properties of a binary tree and is given theoretical justification. The supporting experiments are performed on 3 synthetic/toy benchmarks and 2 math benchmarks. The paper aims to improve LLM’s end-user experience with LLM inference in exchange for extra complexity in the training stage.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n[1] The study is well-rounded, with a theoretically clear problem statement, rigorous theoretical analysis of the solution and the support by experiments.\n\nWeaknesses:\n\n[1] The background for the problem is too brief. While the example on Figure 1 is very insightful, it is not clear, what could other practical uses be apart from solving math puzzles.\n\n[2] The experiments are performed on 3 synthetic/toy benchmarks and the only two practically significant benchmarks (SVAMP and GSM8K) are both math, which is not diverse enough to prove the generalization of the method. A non-math benchmark should be considered (as the authors admit in the discussions), for example HumanEval or even MLEBench.\n\n[3] Phrasing “Compressing LLM Output Tokens” is highly misleading. The individual tokens are not compressed, but rather the reasoning as a sequence of tokens is learned to be shorter.\n\n[4] “Response after 400K Episodes” suggests that training is very slow. 400K episodes looks like a lot to teach the model to reason briefly. Curriculum learning makes a promise to make training fast, which does not seem to be the case here.\n\n[5] Figure 4, CuRL-TrajC clearly is able to get to the target cost the fastest, however its final rewardlags behind the Unconstrained by a large margin, and is not better than other baselines. It is expected for the tradeoff to be like that, but I do not see much of the advantage of CuRL-TrajC compared to ExpSchedule. Yes, it arrives to the same brief reasoning and poor reward, but faster.\n\nQUESTIONS:\n[1] What is “deployment”? This term from the industry/business is not used in academia often.\n\n[2] Is there any way to preserve the reward even with squeezing the size of reasoning into the constraint?\n\n[3] How could tool invocations be incorporated into reasoning?\n\nLIMITATIONS:\nThe method is shown to work only for math problems. The method is limited to text-only reasoning, without support for within-reasoning tool invocation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a curriculum learning strategy for training agents to operate deployment time constraints like limited resources or safety requirements. The method adaptively adjusts cost budget during training, starting off with more relaxed constraints and then slowly makes the constraints more restrictive. This helps smoothen the transition into the more strict deployment conditions. The paper provides theoretical analysis that demonstrates the curriculum strategy accelerates training compared to baseline approaches that enforce the strict constraints right from the start. Empirically, the method is tested on reinforcement learning and LLM agents and achieves significant inference time speedup for LLM output token compression hardware.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- the problem is well-motivated and the writing is generally clear\n- while having an adaptive budget for curriculum isn’t exactly a novel concept [1], I think examining the empirical performance of adapting costs in application to RL for LLM is quite interesting\n- the approach is promising in that it demonstrates significant performance improvements compared to baseline in highly complex tasks like solving gsm8k reasoning questions\n- overall the approach is theoretically grounded in computational/sample complexity\n\nWeakness\n- for th rl experiments it would have been nice to see experiments with a bit more complex algorithms than reinforce, like sota ones including ppo\n- also safe rl environments that are more complex than puddle-grid would be valuable\n- (minor) it would be nice to have a table of contents of all the variables defined (perhaps in appendix) for improved readability\n- (minor) in the figures, you should indicate which algorithm is the papers with something like (ours) to distinguish from baselines\n\n[1]Narvekar et al. \"Curriculum learning for reinforcement learning domains: A framework and survey.\" JMLR 2020\n\nQUESTIONS:\nQuestions\n- does the teacher component incur an additional computational overhead?\n- is token length =40 a real world constraint? I suppose constraints like memory footprint or harmfulness scores would be considered a more meaningful constraints in the semantic experiments of the LLM space\n- how would you choose for a new task what the best maximal/relaxed cost budget to start off with is? Is it always 1/(1-gamma)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new curriculum learning paradigm for addressing exploration tasks with hard constraints. The gist of the approach is the use of a teacher that can shape the reward of the learner, specifically constructed to reflect a sparse reward structure, so that they can learn with increasingly lower budgets. A theoretical analysis is provided for the case of a binary-tree MDP. Empirical evaluation covers 3 domains (4 if counting both PuddlePoint), with the last one being a real-world LLM solving a math problem using a chain-of-thought step count as a budget. The results show improved performance under strict test-time constraints, including higher success rates and shorter outputs for LLMs, without violating the budget.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The problem formulation of exploration under hard constraints or a budget is an important step towards practical agents.\n+ Using curriculum learning with RL exploration for LLMs is a novel and promising approach with practical potential.\n+ The paper is mostly well-written and easy to follow.\n\nWeaknesses:\n- While the idea of the curriculum is novel and has merit, the outcome of the proposed approach is limited: Theorem 1 covers only the binary-tree MDP case. Most exploration problems are more complex than that, so it is unclear how much merit this theorem provides. Moreover, the curriculum relies on accurate performance estimation at various constraint levels using only the learner's rollouts. In complex domains like MathLLM, where each rollout is expensive and noisy, this may limit the teacher's ability to adaptively select effective budgets.\n- The paper cites a recent survey of curriculum learning for RL agents [19], but ignores the content of the survey beyond the reference. It can provide useful vocabulary for this work (e.g., threshold performance).\n- The paper similarly mentions a recent work that is similar [14], but refers to it beyond the reference number only in the empirical methods. This work seems to be the closest to this one, so it should be better explained (in the introduction, not as an afterthought in the methods) what the gaps are that [14] still has and are addressed here.\n\nQUESTIONS:\n1. What are your thoughts about the theoretical analysis of this approach beyond the binary-tree MDP? Obviously it might be much more challenging to provide sound claims, but I wonder about your intuitions. When will it work well and when will it fail?\n2. Have you explored generalization to unseen tasks beyond the ones used during training?\n3. How sensitive is the approach to curriculum parameters like the performance threshold beta?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- In the paper the authors try to address the problem of deployment constraints in terms of budget or safety requirements while training RL agents.\n- The authors propose a curriculum learning strategy to dynamically/adaptively adjust the constraints (from relaxed to strict) during the training of RL agents based on their performance. \n- The authors try to find answers to the question: “How to guide the training and get the best performance when deployment-time constraints advance”\n- The paper introduces a curriculum learning framework to have a sample and compute efficient training strategy under strict trajectory-level constraints so that the final policy can adapt to strict resource and safety requirements.\n- The authors test this framework on a toy binary-tree MDP example and as well as multi-task navigation and LLM based math reasoning tasks.\n- When applying this curriculum learning based strategy for LLM tasks, they claim to have compressed output tokens (a deployment constraint) with a 4.5x inference speedup on consumer hardware. \n- The paper formulates this strategy as a constrained MDP problem where the RL agent (student) tries to roll out the policy to accomplish a task (from the task space) given a required constrained/budget, get the training-time (constrained) rewards and updates the policy. A teacher component is responsible for adjusting/replacing the deployment-time budget with the relaxed training-time cost. \n  - This is done to ensure that the  final performance of the policy is eps-near-optimal. \n  - The framework now tries to optimize the teacher component to select permissible training time budget so that the required policy performance is achieved efficiently in terms of cost and sample complexity.\n\nSTRENGTHS AND WEAKNESSES:\n** Strengths:**\n- The authors try to solve the sparse reward problem of training with the strict deployment constraints from the beginning and instead try to introduce a curriculum strategy.\n- The paper did a good job in providing a thorough theoretical analysis of how the proposed strategy would work for the toy example of binary-tree MDP.\n  - The paper is able to prove that imposing a target/deployment-time budget at the start of the training would require an exponential number of rollouts as compared to polynomial $O(H^3)$ sample complexity in case of curriculum strategy. \n  - The paper shows theoretically that the proposed strategy reduces the number of rollouts required to learn an eps-suboptimal policy. \n- Results from experiments on other environments (navigation and LLM reasoning) supported the theoretical results. \nThe results and the plots are nicely presented. \n\n**Weakness:**\n- It would be nice to have some tabular results as well to have some numerical analysis.\n\n- While there have been good efforts in terms of having theoretical and empirical analysis, this work seems to be deriving few things from the ProCurl-Target work (it does not mean that this fact downweighs this paper’s efforts), this can still be experimented and extended to some other high-dimensional tasks and other LLM or VLM domains to have more real-world constraints.\n\n- This work can present more LLM based use-cases as well given the title of the paper. \n- It would be can be worth trying to incorporate some constraints related to: \n  - different (reduced) input token numbers \n  - token generation: tokens/sec\n  - other test-time scaling constraints.\n  - some subjective constraints like safety, preference, etc.\n\nQUESTIONS:\n- How can we impose other real-time constraints like memory constraints in terms of model size, quantization, GPU memory, etc?\n- Do we have any numbers/plots in terms of latency or inference-speed?\n\nSuggestions:\n- Models that can be deployed under different/dynamic test-time constraints or budgets can be trained (similar to [Matryoshka style learning](https://arxiv.org/abs/2205.13147)) and can be adaptively used as per budget requirements as they change with time.\n\nLIMITATIONS:\n- The authors mentioned extending this work to the programming/code generation domains as well rather than just for math.\n- How the model optimizes for the given constraints during the training is still unclear.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}