Title: Imitation Learning with Temporal Logic Constraints
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

linear temporal logic

imitation learning
Abstract: Designing reinforcement learning agents to satisfy complex temporal objectives expressed in Linear Temporal Logic (LTL), presents significant challenges, particularly in ensuring sample efficiency and task alignment over infinite horizons. Recent works have shown that by leveraging the corresponding Limit Deterministic Büchi Automaton (LDBA) representation, LTL formulas can be translated into variable discounting schemes over LDBA-accepting states to maximize a lower bound on the probability of formula satisfaction. However, the resulting reward signals are inherently sparse, making exploration of LDBA-accepting states increasingly difficult as task horizons lengthen to infinity. In this work, we address these challenges by leveraging finite-length demonstrations to overcome the exploration bottleneck for LTL objectives over infinite horizons. We segment agent exploratory trajectories at LDBA-accepting states and iteratively guide the agent within each segment to learn to efficiently reach these accepting states. By incentivizing the agent to visit LDBA-accepting states from arbitrary states, our approach increases the probability of LTL formula satisfaction without the need for extensive or lengthy demonstrations. We demonstrate the applicability of our method in a variety of high-dimensional continuous control domains. It achieves faster convergence and consistently outperforms baseline approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the problem of learning to accomplish infinite-horizon tasks posed as LTL formulas. In these tasks, the goal is to repeatedly reach an accepting state in the Limit Deterministic Büchi Automaton (LDBA) that results from the LTL. These tasks are challenging since they have naturally sparse rewards. To address this, the authors propose to leverage segmented finite expert demonstrations and trajectories produced by the learned policy itself. The method enables providing guidance for each segment and results in faster convergence with respect to several baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The motivation and the background are clear and described with the right amount of detail. The running example is of great help throughout the paper.\n- The core contributions (segmentation and discriminating trajectories across multiple stages) are clearly described. The ablation studies assess their impact in different tasks, and clear performance gains are shown.\n- Excellent scholarship -- key works in the area are acknowledged throughout the paper.\n- The assumptions and limitations of the method are clearly highlighted.\n\n## Weaknesses\n- The term *occupancy measures* in line 169 is not defined then. Following a (presumably) new mention in line 349, I understand it is related to task satisfaction. It would be convenient to clarify it in future versions.\n- Figure 5 is key for the paper, but it is in the appendix.\n- The figure for `FlatWorldStabilization` would be clearer if the agent stayed within the yellow zone (for clarity, seems stabilizing seems a bit ambiguous).\n- The emphasis of the paper is on infinite-horizon tasks, but most of the testing tasks (`Doggo` and `Fetch`, as highlighted in Fig. 4) are (or seem) finite-horizon. The motivation upon reaching this prt of the paper becomes unclear [see Question 1].\n- It would be interesting to see how performance changes with demonstrations where an accepting state is reached a variable number of times and highlight potential assumptions on the demonstrations as well as their applicability in contexts where the environment can change [see Questions 2 and 3].\n\n## Minor Comments\n- Line 124: \"incentivize\" -> \"incentivizes\".\n- Line 210: \"data sampled from a sampled replay buffer\" -> Perhaps the authors meant \"data sampled from a replay buffer\".\n- MaxStage in Eq. 7 -> \\textsc{MaxStage} (like in the lines below).\n- Line 328: footnote should go after the dot, i.e. \"...stays still.\\footnote{...}\".\n- The Conclusion should be in its main section (probably left within the Related Work because of lack of space).\n\nQUESTIONS:\nI would really appreciate if the authors can especially focus on Q1 and Q2. I am keen on increasing my score if my doubts are addressed.\n\n**Q1.** The approach seems general enough to be applicable in the context of a finite-horizon task since finite-horizon tasks can also be naturally segmented. Indeed, most of the tasks used in the ablations (`Doggo` and `Fetch`) are finite-horizon in my understanding, since they have absorbing accepting states. However, upon testing the segmentation in the ablation studies (Fig. 4) it is said that there is negligible difference between segmenting and not doing so. My questions on this matter are:\n* Why would segmentation only benefit tasks without absorbing accepting states?\n* Following lines 187-188, does this have something to do with satisfying LTL constraints over infinite horizons being more challenging? Why?\n* Why motivating the use of infinite-horizon tasks but then most of the tasks are finite-horizon?\n* Why are the ablations not performed (or shown) for `GridCircular Hard` or `FlatWorld` tasks?\nPlease let me know I am wrong in saying that tasks with absorbing accepting states do not have an infinite horizon.\n\n**Q2.** In what contexts would finite-length demonstrations (or a finite number of them) be potentially problematic or insufficient? In the considered tasks, the environment never changes after reaching the accepting state (e.g. the yellow area in `FlatWorld Stabilization` does not change location); thus, a single demonstration where the accepting state is reached once seems enough. However, there are infinite-horizon tasks where the environment can change, e.g. see the tasks Cookie (the location of a cookie changes after pressing a button) and Symbol in [1]. Under which circumstances would demonstrations in these settings be useful? Have any inherent assumptions on the demonstrations been made in the submitted work? Is the assumption visiting an accepting state only once or twice stronger than it seems?\n\n**Q3.** Has the effect of having longer or shorter demonstrations (hence, reaching the accepting states more or fewer times) been studied? If not, how do you think it can influence the performance of the algorithm?\n\n**Q4.** Has the approach been compared against methods that leverage the structure of an automaton for reward shaping? For example, potential-based reward shaping is employed in the cited works by Camacho et al. using the minimum distance to the accepting state and performing value iteration over the automaton graph. How about method using hierarchical reinforcement learning, which similarly leverage the decomposition of a task into substages? If not, what are the key differential factors you would expect to see (or if they are not applicable in your context, please explain why).\n\n**References**\n[1] Toro Icarte et al. (2019). Learning Reward Machines for Partially Observable Reinforcement Learning.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of satisfying LTL objectives in reinforcement learning over infinite horizons by using finite-length demonstrations to guide exploration. The proposed method segments trajectories at accepting states and trains the agent to reach these states from arbitrary points, improving satisfaction probability without requiring long demonstrations. Experiments in high-dimensional continuous control domains show faster convergence and better performance than baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\nThe motivation to accelerate policy learning under LTL constraints is clear and well-grounded. The authors present a theoretical analysis of the proposed method and support it with empirical results across diverse tasks and environments. However, key aspects of the design choices and evaluation remain insufficiently justified or explored, as outlined in the questions and limitations below.\n\nQUESTIONS:\n1. In Eq. (4), the reward is defined as a function of the product state $R((s,b))$, independent of the action $a$. This differs from the standard RL setting, where the reward typically depends on both the state and the action, i.e., $R(s, a)$. Could the authors clarify the rationale behind this formulation and its implications, such as how it may affect the optimality of the learned policy in cases where different actions lead to different outcomes despite receiving the same reward?\n2. For the multi-stage reward formulation, how does the choice of the hyperparameter $\\beta$ affect performance, given that the reward directly influences policy learning? Additionally, since the reward depends on the length of the longest acyclic path, could the authors provide more details on how this is computed and how it impacts learning, especially in cases where the path is long, which may lead to very small or vanishing reward signals?\n3. The proposed method assumes access to expert demonstrations. Could the authors clarify the underlying assumptions about these demonstrations? Specifically, do the demonstrations contain trajectories that visit each accepting state, and do they provide sufficient coverage of the state space? For instance, in the FlatWorld Cycle environment, if the demonstrations only reach the yellow region, how would the method support learning policies that reach the green region? Similarly, if all demonstrations originate from the left side of the environment, would this limit the agent’s ability to explore or learn effectively when initialized from the right side?\n4. In the experiments, how many episodes are averaged per evaluation (in addition to the 10 random seeds) for each environment? Also, aside from Cheetah Flip, are the initial positions of the agents and the placement of regions or objects randomly generated during evaluation?\n\nLIMITATIONS:\nThe proposed method faces scalability challenges due to its multi-stage discriminator learning framework. Specifically, the number of accepting states ($b_N$ in Section 3.2) grows with the complexity of the LTL specification and the number of atomic propositions, leading to a corresponding increase in the number of discriminators. This scaling makes the training process more computationally demanding. Moreover, jointly training multiple discriminators alongside the Q-function and policy introduces additional optimization difficulties and may hinder stable convergence.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents TiLoIL, an imitation learning method for achieving objectives specified using Linear Temporal Logic (LTL). The key innovations are two forms of decomposition: first, segmenting expert trajectories to learn how to reach accepting states in a corresponding LDBA, and second, breaking down tasks into stages to facilitate learning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n- Enabling agents to follow temporally extended symbolic objectives is an important problem.\n- Clearly identifies the challenge of using finite trajectories for general LTL rather than resorting to LTLf (LTL over finite traces).  \n- The proposed methodology appears sound, and experimental results demonstrate effectiveness on selected tasks.  \n\n### Weaknesses:\n\n- **Clarity of formulation and demonstrations**: The formulation could be clearer about the nature of the expert demonstrations and how they relate to the LTL objectives. Why don't *all* expert demonstration satisfy the given LTL? The setup is presented as multi-objective, but it seems the primary goal is to find a policy that satisfies LTL constraints, unless there are additional implicit objectives encoded in the demonstrations. Clarification would help.\n\n- **Positioning within the literature**: The related work could be better situated. The second paragraph of Appendix B should be included in the main paper and expanded to clarify the similarities and differences with existing approaches. For example, from my understanding, TiLoIL requires environment interaction, whereas [20] operates in a purely offline setting. [20] also uses trajectory segmentation and does not require expert demonstrations, something noted as future work for TiLoIL. Similar comparisons should be made against the other prior work in a more systematic fashion, e.g., by categorizing the methods.\n\n- **Use of LDBAs in continuous tasks**: It is unclear how LDBAs and product MDPs are constructed and used for tasks with continuous state spaces. Was the state space discretized? Given that LDBAs are worst-case exponential in the size of the LTL formula, this may limit scalability to more complex objectives.\n\n- **Experimental setup and insights**: While TiLoIL performs well on many tasks, the task selection could be more clearly motivated. For example, Minecraft and Pacman appear in [63], while [6] includes two Cheetah variants. Clarifying why these tasks were chosen would help us understand the results. Additionally, it would be valuable to discuss the types of tasks where TiLoIL is likely to succeed or fail, to guide future research. On the Cheetah Flip task, the multi-stage approach appears to have hindered learning. The explanation offered is a good start, but a deeper analysis of failure cases would improve understanding and potential resolutions.\n\nQUESTIONS:\nPlease see the points under weaknesses for more detailed discussion. The main questions are:\n\n- Are there any implicit objectives in the demonstrations beyond the LTL specifications?\n- How should TiLoIL be categorized in relation to existing work (e.g., online vs. offline, imitation learning vs. reinforcement learning, or a hybrid)?\n- How are the product MDPs constructed, particularly in continuous domains?\n- What kinds of tasks is TiLoIL likely to succeed or struggle with, and why?\n\nI am willing to adjust my score depending on the responses to the above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with the problem of imitation learning where the learning agent has access to both a set of positive expert demonstrations, and in addition a task specification in LTL which the expert demonstrations satisfy. Since LTL is evaluated over infinite trajectories, RL approaches design proxy rewards based on accepting states (an LTL formula can be translated to a LDBA state machine, where an infinite trajectory satisfies the LTL if it visits an accepting state infinitely often). The inherent difficulty in this approach is that the rewards are sparse, making exploration difficult. Expert demonstrations can theoretically help, but are by nature finite.\nTo address these difficulties the authors splits trajectories into \"segments\", sub-trajectories between accepting states (these are effectively treated as independent demonstrations), and within those segments they leverage the structure of the LTL formula (as evident in the LDBA) to further recognize \"stages\" of satisfaction of the formula, allowing them to densify the rewards. Their approach falls within the framework of GAIL, alternately training a discriminator to distinguish between states from (suitably defined) positive and negative sub-trajectories, and using the learned discriminator(s) to define dense rewards for policy improvement. They optimize for both satisfaction of the formula (reaching an accepting state) and IL goal of making the policy trajectories indistinguishable from the expert demonstrations. They evaluate their learned policies on a number of standard tasks in high-dimensional spaces (some of them circular, to specifically evaluate the merits of their segmentation method), and test two aspects of their work - comparing it to SOTA policy optimization methods that make use of LTL objectives (but not expert demonstrations) to see if it improves exploration, and to IL methods that do not explicitly leverage LTL specifications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is novel in that it combines the LDBA-MDP product formulation with expert demonstrations, combining global task specification with more local guidance within the segments from the demonstrations.\n\n- The evaluation is presented clearly, the ablation results elucidate the importance of their segmentation technique in cyclical tasks. The results seem good compared to existing SOTA methods.\n\n- The paper is well-written, and related work and context is thoroughly explored. \n\nWeaknesses:\n\n- The contributions of the paper are not clearly outlined. As a non-expert in the field, it took me a while to (hopefully) disentangle the issues of infinite horizon and reward sparsity.\n\n- While epsilon-transitions in the LDBA are formally defined, I struggled to understand from the paper alone what their meaning is in the product MDP. The running example makes no use of them, their relation to deterministic vs non-deterministic states is not mentioned and overall I'm not sure why they were introduced (other than for completeness). Its not a major issue, but for a non-expert it was a sticking point.\n\nQUESTIONS:\n- I understand that non-optimal demonstrations are out of scope, but given that in the real world they exist, what is the effect of adding such demonstrations in practice? Will the results degrade gracefully, or can one sub-optimal demonstration, no matter how close to satisfaction, collapse the entire optimization?\n\n- If the epsilon-transitions have any meaning in your framework, I'd love to see them incorporated into an example (The flatworld perhaps)\n\n- Can this method be adapted to a setting where the observations of the environment are noisy/partially observed? What if I'm only 90% certain I hit 'blue'? I realize it is out of scope, but addressing this in the discussion/limitations would add broader context.\n\nLIMITATIONS:\nThe authors address the main limitation - that the method assumes optimal demonstrations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}