Title: Trajectory Graph Learning: Aligning with Long Trajectories in Reinforcement Learning Without Reward Design
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Trajectory Alignment

Trajectory Graph Learning
Abstract: Reinforcement learning (RL) often relies on manually designed reward functions, which are difficult to specify and can lead to issues such as reward hacking and suboptimal behavior. Alternatives like inverse RL and preference-based RL attempt to infer surrogate rewards from demonstrations or preferences but suffer from ambiguity and distribution mismatch. A more direct approach, inspired by imitation learning, avoids reward modeling by leveraging expert demonstrations. However, most existing methods align actions only at individual states, failing to capture the coherence of long-horizon trajectories.

In this work, we study the problem of directly aligning policies with expert-labeled trajectories to preserve long-horizon behavior without relying on reward signals. Specifically, we aim to learn a policy that maximizes the probability of generating the expert trajectories. Nevertheless, we prove that, in its general form, this trajectory alignment problem is NP-complete. 
To address this, we propose Trajectory Graph Learning (TGL), a framework that leverages structural assumptions commonly satisfied in practice—such as bounded realizability of expert trajectories or a tree-structured MDP. These enable a graph-based policy planning algorithm that computes optimal policies in polynomial time under known dynamics. For settings with unknown dynamics, we develop a sample-efficient algorithm based on UCB-style exploration and establish sub-linear regret. Experiments on grid-world tasks demonstrate that TGL substantially outperforms standard imitation learning methods for long-trajectory planning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a very interesting and well mathematically analyzed problem formulation to find a optimal subset of expert trajectories that are free of transition conflicts in specific states. The conflict-free trajectories are optimal with respect to the deducable policy that lead to trajctories well aligned to the expert trajectories. They present two algorithms to find policies in environments with unknown or known model dynamics (first is an approximation).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The mathematical description of the problem is sound and reasonable. \n- The proof in showing the np-completeness seems correct and complete.\n- The idea of optimizing the MWIS to remove the covariance shift makes sense.\n\nWeaknesses:\n- Motivation:\nWhy should the agent be closer aligned to the expert trajectory with the cost of removing training data? Some removed trajectories could hold valuable information. For example: Was the tweak with \"make the holes passable\" in the evaluation necessary to make the algorithm able to solve the task? \n- Evaluation:\nNot reproducable in my opinion. Please explain the experimental setup and evaluation more in detail. For example: what is 'imitation.algorithms.bc'? Is BC and TGL trained on the PPO trajectories?  \nAlso the metric is questionable: Does it make sense that the PPO expert is worse in imitating itself? \nFurthermore: Whats the runtime dependency to the problem size? Is it applicable to real-world problems? Maybe the authors could extend the gridsize and measure the runtime.\nAbstact says \"substantially outperforms standard imitation learning methods for long-trajectory planning.\" but the experiments only include horizons of 10 steps. \n\n\nAll in all I appreciate the mathematically sound problem formulation and method. I would raise my score to accept, if the authors could give a motivation that explains why the algorithm is useful in practice (maybe with examples) and a more elaborated evaluation that leads to more insights and also presents more metrics like the accuracy on the main tasks.  \n\n\nMinor weaknesses:\n- Paper checklist: The instruction block is present, albeit its written to be deleted.\n- Some typos.  \n- L 144 (formula) \"Phi in BigPhi\" -> BigPhi not introduced.\n- Its hard to understand why the problem can be formulated as MWIS and I needed to read it multiple times. Please add a few lines to L 188 that qualitatively describe why a valid policy is equivalent to a independent set in the graph. That would make it easier to follow.\n\nQUESTIONS:\nIs there a difference between (l143) d^phi and (l182) p_i for phi_i? Why is it defined two times?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the direct trajectory alignment problem, which aims to find a deterministic policy that maximizes the visitation probability of a given set of expert trajectories. The authors make a key theoretical contribution by proving that this problem is NP-complete, by showing its equivalence to the maximum-weight independent set problem. To overcome this computational hardness, the paper proposes polynomial-time algorithms for two special cases under known dynamics. For the unknown dynamics, a UCB-based online learning algorithm is developed and analyzed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed framework of direct trajectory alignment is a novel and interesting contribution, as it bypasses the need for reward model or any model assumptions.\n- The paper is well-structured and clearly presents its problem formulation, algorithms, and theoretical analyses.\n\nWeaknesses:\n- The proposed approaches in Section 4 and 5 appear to be well established techniques and their error bounds are also naïve.\n- While the paper claims that the method is effective for long-horizon problems, but this central claim lacks sufficient analytical or empirical support. It is not made clear why optimizing the proposed objective function inherently leads to better performance on long-horizon tasks compared to other methods.\n- The experimental validation is limited. The evaluation is conducted on a single, small-scale grid-world environment and the metric is the trajectory match probability, which is directly related to the method's objective function. This makes it difficult to assess the framework's generalization capabilities or its effectiveness on other standard performance metrics.\n- The analysis is limited to deterministic policies and the tabular setting.\n\nQUESTIONS:\n- In general, the use of direct trajectory alignment is not well motivated. In the example in Appendix A.2, it is unclear to me why the DTA policy is superior than the BC policy. Could you elaborate on when and why DTA is better than BC?\n\n- In numerical experiments, could you present other performance metrics (e.g. distance to a ground-truth expert policy or task performances of the resulting policy). Additionally, how does the policy's performance vary as the horizon length (H) changes?\n\n- In line 240, I believe the complexity bound should be M^{1/\\epsilon_0}.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the problem of directly aligning policies with expert-labeled trajectories in reinforcement learning without relying on reward modeling. It proposes Trajectory Graph Learning (TGL), a framework that leverages structural assumptions to enable efficient policy planning.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths:\n1. The motivation of the paper is strong. It highlights the limitation of traditional behavior cloning in imitation learning, which only focuses on state-action pair alignment and neglects the broader trajectory-level structure, and proposes the idea of direct trajectory alignment, which is meaningful.\n2. The proposal of the TGL framework is innovative to some extent, aiming to solve the problem of trajectory alignment under structural assumptions.\n* Weaknesses:\n1. The paper is relatively theoretical, and some details are not well explained. For instance, the supplementary material includes the full paper with appendices in a single ZIP file, which is not in line with the NeurIPS submission guidelines. Reviewers primarily rely on the officially submitted PDF for their assessment, and any additional technical appendices should ideally be submitted together with the main paper before the full submission deadline to facilitate a more straightforward review process.\n2. The theoretical part of the paper is relatively strong, but to enhance the practical significance of the research, it would be valuable to include comparisons with related methods such as Inverse RL and PbRL in the experiments. This would help to demonstrate the practical utility and advantages of the TGL framework beyond its theoretical merits.\n3. Given the current experiments are only conducted on a 4×4 Frozen Lake environment, it would be beneficial to validate the TGL framework on more complex environments such as larger grid-worlds or continuous state/action spaces. For example, environments like the Mujoco robotics simulations or more complex navigation tasks could provide a better understanding of the framework’s generalizability and robustness.\n\nQUESTIONS:\n1. Given the current experiments are only conducted on a 4×4 Frozen Lake environment, it would be beneficial to validate the TGL framework on more complex environments such as larger grid-worlds or continuous state/action spaces. For example, environments like the Mujoco robotics simulations or more complex navigation tasks could provide a better understanding of the framework’s generalizability and robustness.\n2. Could you include comparisons with related methods such as Inverse RL and PbRL in the experiments? This would help to demonstrate the practical utility and advantages of the TGL framework beyond its theoretical merits.\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations of their work. It is suggested to supplement the discussion on the limitations of the TGL framework in the paper, such as its applicability in different tasks and environments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Trajectory Graph Learning (TGL), a novel framework that directly aligns policies with expert-labeled trajectories to preserve long-horizon behavior without relying on reward signals. By modeling the trajectory alignment problem as a Maximum-Weight Independent Set (MWIS) problem, the authors address the inherent complexity of the problem and develop efficient algorithms for both known and unknown environment dynamics. The theoretical analyses demonstrate the NP-completeness of the general trajectory alignment problem and provide polynomial-time solutions under certain structural assumptions. Experimental results on grid-world tasks show that TGL outperforms standard imitation learning methods, especially for long-trajectory planning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1．\tThe idea of modeling the trajectory alignment problem as an MWIS problem is interesting and provides a novel perspective on solving the trajectory imitation challenge.\n\n2．\tThe paper provides a thorough theoretical analysis, proving the NP-completeness of the general problem and presenting efficient algorithms under practical assumptions.\n\n3．\tExperiments on grid-world tasks demonstrate the effectiveness of the proposed framework compared to standard imitation learning methods.\n\n\nWeaknesses:\n\n1．\tThe experiments are conducted on relatively simple grid-world tasks. More experiments on larger or continuous MDPs are needed to fully evaluate the scalability and robustness of the proposed method.\n\n2．\tThe experimental results lack error bars or other statistical significance measures, which makes it difficult to assess the consistency and reliability of the reported performance.\n\n3．\tThe paper does not appear to have been compiled using the official NeurIPS 2025 LaTeX template, which is a requirement for submission.\n\nQUESTIONS:\n1.\tCan the authors conduct experiments on more complex and real-world environments to further validate the scalability and robustness of the proposed TGL framework? \n\n2.\tCan the authors report error bars or perform statistical significance tests on their experimental results to provide a more rigorous evaluation of the performance of TGL compared to baselines?\n\n3.\tCan the authors provide more details on the practical relevance and validity of the structural assumptions (bounded realizability of expert trajectories, tree-structured MDP) made in their theoretical analyses?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}