Title: One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Linear Temporal Logic

Safe Reinforcement Learning

Zero-shot Generalization
Abstract: Generalizing to complex and temporally extended task objectives and safety constraints remains a critical challenge in reinforcement learning (RL). Linear temporal logic (LTL) offers a unified formalism to specify such requirements, yet existing methods are limited in their abilities to handle nested long-horizon tasks and safety constraints, and cannot identify situations when a subgoal is not satisfiable and an alternative should be sought. In this paper, we introduce GenZ-LTL, a method that enables zero-shot generalization to arbitrary LTL specifications. GenZ-LTL leverages the structure of Büchi automata to decompose an LTL task specification into sequences of reach-avoid subgoals. Contrary to the current state-of-the-art method that conditions on subgoal sequences, we show that it is more effective to achieve zero-shot generalization by solving these reach-avoid problems $\textit{one subgoal at a time}$ through proper safe RL formulations. In addition, we introduce a novel subgoal-induced observation reduction technique that can mitigate the exponential complexity of subgoal-state combinations under realistic assumptions. Empirical results show that GenZ-LTL substantially outperforms existing methods in zero-shot generalization to unseen LTL specifications.

One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning

Abstract

Generalizing to complex and temporally extended task objectives and safety constraints remains a critical challenge in reinforcement learning (RL). Linear temporal logic (LTL) offers a unified formalism to specify such requirements, yet existing methods are limited in their abilities to handle nested long-horizon tasks and safety constraints, and cannot identify situations when a subgoal is not satisfiable and an alternative should be sought. In this paper, we introduce GenZ-LTL, a method that enables zero-shot generalization to arbitrary LTL specifications. GenZ-LTL leverages the structure of Büchi automata to decompose an LTL task specification into sequences of reach-avoid subgoals. Contrary to the current state-of-the-art method that conditions on subgoal sequences, we show that it is more effective to achieve zero-shot generalization by solving these reach-avoid problems one subgoal at a time through proper safe RL formulations. In addition, we introduce a novel subgoal-induced observation reduction technique that can mitigate the exponential complexity of subgoal-state combinations under realistic assumptions. Empirical results show that GenZ-LTL substantially outperforms existing methods in zero-shot generalization to unseen LTL specifications. The code is available at https://github.com/BU-DEPEND-Lab/GenZ-LTL.

1 Introduction

Generalization is a critical aspect of reinforcement learning (RL), which aims to develop policies that are capable of adapting to novel and unseen tasks [38, 48, 21] and varying safety constraints [7, 17]. Various methods have been proposed and show promising results in applications such as autonomous driving [49, 36], robotics [27, 33, 28], and healthcare [8, 41]. However, most existing approaches fall short in handling complex and temporally extended task objectives and safety constraints. For example, an autonomous vehicle may be instructed to reach a sequence of destinations for passenger pick-up and drop-off while adhering to traffic rules such as speed limits and yielding at intersections.

Temporal logic provides a principled framework for specifying system behaviors over time. Linear temporal logic (LTL) [39], in particular, has been considered extensively in single-task and multitask RL settings [20, 29, 51]. LTL has well-defined semantics and a compositional syntax, which facilitates the unambiguous specification of task objectives and safety constraints. However, existing methods often fail to account for the potential conflicting objectives within a specification [50, 40, 23]. For example, an autonomous vehicle navigating between multiple destinations aims to minimize travel time, but adhering to speed limits and other safety constraints may slow the vehicle down. In such cases, the vehicle must prioritize satisfying safety requirements over optimizing for travel time.

In this paper, we propose a novel framework, called GenZ-LTL, for learning RL policies that generalize in a zero-shot manner to satisfy arbitrary LTL specifications, including both finite-horizon and infinite-horizon tasks. We adopt the subgoal decomposition approach, which exploits the structure of the equivalent Büchi automaton to decompose an LTL specification into individual subgoals, each comprising a reach component that defines task progression and an avoid component that encodes safety constraints. We show that training a goal-conditioned policy to achieve the subgoals one at a time , which is often perceived as myopic by existing literature, is more effective in general than conditioning the policy on subgoal sequences [23] or the entire automaton [54]. This somewhat surprising result stems from two key innovations in this paper: a proper safe RL formulation with state-wise constraints for solving the individual reach-avoid subgoals, and a novel subgoal-induced observation reduction technique to address the exponential complexity of observation-subgoal combinations. The former builds on recent advances of Hamilton-Jacobi reachability in RL [12] whereas the latter exploits symmetry in the state observations to extract information only relevant to the immediate subgoal. To zero-shot generalize to arbitrary LTL specifications at test time, we use the equivalent Büchi automata as well as the learned value functions to guide the subgoal selection. Our main contributions are summarized as follows.

We propose a novel approach for learning RL policies that can satisfy arbitrary LTL specifications in a zero-shot manner by sequentially completing individual subgoals. To our knowledge, this is the first safe RL formulation with state-wise constraints for LTL task satisfaction.

We introduce a subgoal-induced observation reduction technique to mitigate the combinatorial explosion of observation-subgoal pairs and enable efficient generalization, along with a timeout mechanism that handles subgoal switching and unsatisfiable subgoals.

We conduct comprehensive experiments across a set of navigation-style environments and diverse task specifications, demonstrating that our method consistently outperforms state-ofthe-art baselines in zero-shot generalization to arbitrary LTL specifications.

2 Related Work

Specification-guided and goal-conditioned RL. Various methods have been proposed to guide RL agents by extending traditional goal-conditioned RL [35] to temporal-logic goals. Early works focused on learning policies for a single and fixed specification, including methods based on LTL specifications and automaton-like models [20, 18, 51, 46, 45, 3], and methods leveraging quantitative semantics of specifications [32, 26, 44, 16]. More recently, the importance of adapting to changing conditions in real-world applications has brought increased attention to zero-shot generalization to new specifications. Existing approaches can be broadly categorized into subgoal decomposition [30, 31, 34, 40, 23] and direct specification/automata encoding [50, 54, 55]. Our work follows the direction of subgoal decomposition, but shows that it is more effective to solve the subgoals one at a time through proper safe RL formulations and subgoal-induced observation reduction.

RL with safety constraints. Various methods have been proposed to solve RL problems with safety constraints, such as policy optimization-based methods [59, 56, 24, 53], Gaussian processesbased approaches [52, 60], and control theory-based methods, including Hamilton-Jacobian (HJ) reachability [10, 57, 47, 13, 61] and control barrier functions [14, 6]. We point the readers to [15] for a comprehensive survey of safe RL techniques. In the context of formal specification-guided RL, the interplay between task progression and safety constraints remains underexplored. Most existing methods simply assign a positive reward when a specification or a subgoal is satisfied, and a negative reward at episode termination if unsatisfied. This approach is ineffective in achieving safety [1, 9, 4], especially when the task progression objective and the safety requirements exert conflicting influences on policy learning. In this paper, we show how to prioritize safety satisfaction properly by incorporating Hamilton-Jacobi reachability constraints.

3 Preliminaries

Reinforcement learning. RL agents learn policies by interacting with environments, which is usually modeled as a Markov Decision Process (MDP), defined by the tuple M := ( S , A , P, r, γ, d 0 ) , where S is the state space and A is the action space, P : S ×A×S ↦→ [0 , 1] defines the transition dynamics, r : S × A ↦→ R is the reward function, γ ∈ [0 , 1) is the discount factor, and d 0 ∈ ∆( S ) is the initial state distribution. Let π : S × A ↦→ [0 , 1] denote a policy and τ = { s t , a t , r t } ∞ t =0 denote a trajectory

where r t = r ( s t , a t ) . The value function under policy π is V π r ( s ) = E τ ∼ π,s 0 = s [ ∑ ∞ t =0 γ t r t ] and the corresponding state-action value function is Q π r ( s, a ) = E τ ∼ π,s 0 = s,a 0 = a [ ∑ ∞ t =1 γ t r t ] . An additional constraint violation function h : S ↦→ R can be used to model state-wise safety constraints, such that a state s is considered safe if h ( s ) ≤ 0 and unsafe if h ( s ) > 0 . The reachability value function V π h ( s ) is defined as V π h ( s ) := max t ∈ N h ( s ) , s 0 = s, a ∼ π that captures the worst-case constraint violation under policy π from state s . The largest feasible set contains the largest set of states from which the agent can reach a goal safely: S f := { s | ∃ π, V π h ( s ) ≤ 0 } . To handle such constraints, Hamilton-Jacobi (HJ) reachability has been incorporated into RL [12] by simultaneously learning the reachability value function to estimate the feasible set and learning the policy:

max π E s ∼ d 0 [ V π r ( s ) · ✶ [ s ∈ S f ] -V π h ( s ) · ✶ [ s / ∈ S f ]] s.t. V π h ( s ) ≤ 0 , ∀ s ∈ S f ∩ S 0 where S 0 := { s | s ∼ d 0 } is the set of initial states. Intuitively, the goal is to maximize expected return and ensure safety for initial states within the largest feasible set, while minimizing the reachability value function for states outside it, where constraint violations are inevitable.

Linear Temporal Logic. LTL [39] is a formal language for specifying temporal properties of a system. LTL formulas are built from a set of Boolean operators - negation ( ¬ ), conjunction ( ∧ ), disjunction ( ∨ ), and temporal operators - "until" ( U ), "eventually" ( F ), and "always" ( G ). Given a finite set of atomic propositions AP and a ∈ AP , the syntax of LTL is defined recursively as:

Intuitively, the formula φ 1 U φ 2 holds if φ 1 is satisfied at all time steps prior to the first occurrence of φ 2 . The operator F φ holds if φ is satisfied at some future time, while G φ holds if φ is satisfied from the current step onward. To relate LTL to MDPs, we assume a known labeling function L : S ↦→ 2 AP that maps each state to the set of true atomic propositions. The satisfaction probability of φ under a policy π is Pr( π | = φ ) = E τ ∼ π [ ✶ [ τ | = φ ]] , where τ | = φ denotes that the trace of the trajectory, Tr ( τ ) = L ( s 0 ) L ( s 1 ) . . . , satisfies the formula φ , and ✶ is the indicator function. We are interested in training a policy that maximizes this probability for an arbitrary φ .

̸

Büchi automata and subgoals. To enable algorithmic reasoning over LTL specifications, we use Büchi automata [5] as a formal representation of temporal-logic formulas. Given an LTL formula φ , the corresponding (non-deterministic) Büchi automaton is defined by a tuple B φ := ( Q , Σ , δ, F , q 0 ) where Q is the finite set of automaton states, Σ = 2 AP the finite alphabet, δ ⊆ Q × Σ × Q the transition relation, F ⊆ Q the set of accepting automaton states, and q 0 the initial state. An infinite word ω = α 0 α 1 . . . ∈ Σ ω induces a run r = q 0 q 1 . . . over ω if for every i ≥ 0 , ( q i , α i , q i +1 ) ∈ δ . A run r is accepting if it visits an accepting state infinitely often. A product MDP M φ [11] synchronizes a given MDP and Büchi automaton with a new state space S φ = S × Q and a transition function P φ (( s ′ , q ′ ) | ( s, q ) , a ) that equals P ( s ′ | s, a ) if a ∈ A and ( q, L ( s ) , q ′ ) ∈ δ , and 0 otherwise. In practice, rather than explicitly building the product MDP, one can simply update the current automaton state q with the propositions L ( s ) observed at each time step. Given a state q , a reach subgoal is defined as an α + ∈ Σ such that ( q, α + , q ′ ) ∈ δ and q and q ′ are consecutive automaton states along some accepting run, and q ′ = q if q / ∈ F . Conversely, an avoid subgoal is defined as an α -∈ Σ such that ( q, α -, q ′ ) ∈ δ and q and q ′ are not consecutive automaton states along any accepting run. Intuitively, a reach subgoal indicates progression towards satisfying the LTL specification, and an avoid subgoal indicates a way that leads to violation of the specification.

4 Method

In this section, we present our proposed method, as illustrated in Figure 1. During training, we begin by sampling subgoals σ = ( α + , A -) , where α + is a single reach subgoal and A -is a (possibly empty) set of avoid subgoals (and α + / ∈ A -). Given the current state s and subgoal σ , we apply subgoal-induced observation reduction (details in Section 4.2), when applicable, to obtain a processed state s σ . We then train a general reach-avoid subgoal-conditioned policy along with its corresponding value functions. During testing, given a target LTL task specification φ , we first construct an equivalent Büchi automaton, and then extract a set of reach-avoid subgoals { σ i } n i =1 based on the current MDP and automaton state. For each candidate subgoal σ i , we compute s σ i and evaluate the value functions to select the optimal subgoal σ ∗ . The agent then generates actions a according to π ( a | s σ ∗ ) . In Section 4.1, we describe how to construct reach-avoid subgoals from a Büchi automaton. In Section 4.2, we introduce a subgoal-induced observation reduction technique to

Training

Testing

r,h

D

Agent

LTLformula

Environment

a

Environment

Subgoal-induced

S

S

observation

V

(So

BuchiAutomata

Agent

reduction

Subgoal

sampler

=（a+，A

Extractsubgoal

{0i}=

Subgoal-induced

observation

V(s0i)

Vπ(sgi)

π（as

candidates

reduction

mitigate the exponential complexity arising from the combination of subgoals and MDP states. In Section 4.3, we detail the learning of value functions and policies under reachability constraints to comply with the safety constraints ( A -) while optimizing for the task progression objective ( α + ).

4.1 Reach-Avoid Subgoal Construction from Büchi Automaton

Büchi automata can be represented as directed state-transition graphs [40], and an accepted run can be viewed as having a lasso structure with a cycle containing an accepting state and a prefix path that leads to the cycle. Similar to existing approaches [40, 23], we use depth-first search (DFS) over the state-transition graph to enumerate all possible lasso structures starting from a given state q . From these lasso paths, we extract the set of reach subgoals, denoted by A + , following the definition in Section 3. Similarly, we extract the set of avoid subgoals A -by considering transitions to a state that is different from q and not along any of the lasso paths. Each pair ( α + ∈ A + , A -) then becomes a candidate reach-avoid subgoal. To satisfy a subgoal ( α + , A -) , the agent must visit a state s such that L ( s ) = α + , while avoiding any states labeled with propositions in A -. For example, for ( A + = {{ a, b } , { c }} , A -= {{ d } , { e }} ) , it forms subgoals σ 1 = ( { a ∧ b } , A -) and σ 2 = ( { c } , A -) , and the agent can choose to reach a state labeled with a ∧ b or a state labeled with c , while avoiding states labeled with d and e in both cases. While our method follows a similar subgoal decomposition approach as prior works, the way that we solve the reach-avoid subgoals is significantly different. In fact, a straightforward way is to encode a reach-avoid subgoal as a bitvector of dimension | AP | +2 | AP | , where the first | AP | bits encodes the reach subgoal, and the rest of the bits encode the avoid subgoal.

4.2 Subgoal-Induced Observation Reduction

Given a reach-avoid subgoal, our next step is to train a subgoal-conditioned policy based on the MDP state or observation of the MDP state, and the subgoal. However, notice that the number of possible subgoals grows exponentially with the number of atomic propositions. Additionally, in order for the agent to complete a task in such a complex environment, it needs to be able to make observation of every state-subgoal combination, i.e. being able to determine the label L ( s ) of a state s that it observes (possibly indirectly through an observation function o ). Otherwise, task completion cannot be determined in general. For instance, if the task requires the agent to reach a 'green' region, then an agent equipped with only 'blue' sensors cannot learn to accomplish this task if the green and blue regions are placed randomly in the environment. Thus, the number of observations that the policy needs to condition on also grows exponentially with | AP | , and directly learning a goal-conditioned policy this way quickly becomes intractable as | AP | grows. While various dimensionality reduction techniques have been proposed in the RL literature [2, 19, 58], they are based on abstraction or bisimulation of the MDP states, and do not consider state-subgoal combinations.

In this paper, we introduce a novel subgoal-induced observation reduction technique that can circumvent this exponential complexity. The key observation is that, ultimately, each subgoal reduces to a reach-avoid problem, where there is some state that we want to reach while avoiding some other states. In other words, what really matters is whether a state is 'reach' or 'avoid', and not its label L ( s ) once a subgoal is fixed. Thus, we can perform a symbolic simplification of the state/observation space

along with the subgoal while preserving information essential for solving the reach-avoid problem. Formally, for an MDP state s , we assume that it can be partition into a proposition-independent component s ¬ AP and a proposition-dependent component s AP (e.g, the blue and green sensors). For the proposition-dependent component, we further assume that it can be broken down into individual observations each corresponding to an element in 2 AP , and the observations are produced by identical observation functions under output transformation . We say that two observation functions (with a slight abuse of notation of s ), o 1 ( s ) ∈ R k and o 2 ( s ) ∈ R k are identical under an output transformation g if ∀ s, o 1 ( s ) = g ( o 2 ( s )) . For instance, suppose o 1 ( s ) ∈ R 360 is a LiDAR sensor for sensing the shortest distance to a blue state along each of its 360 beams, and o 2 ( s ) ∈ R 360 is the same type of LiDAR sensor for sensing the shortest distance to a green state but rotated 180 ◦ , then g is the 180 ◦ rotation (a permutation of the beam indices). Then, we can reduce the proposition-dependent component (of size k × 2 | AP | ) and the encoding of a reach-avoid subgoal (of size ( | AP | +2 | AP | ) into just two observations each of size k , where the first observation is an observation corresponding to the reach subgoal and the second observation is a fused observation of the observations corresponding to the avoid subgoals. For simplicity of deriving a fusion operator that preserves task-relevant semantics such as distances to avoid zones, we assume that the o ( s ) are coordinate-wise monotonic. The actual fusion operator f is environment/sensor-specific - in the LiDAR example, as shown in Figure 2, it will be an elementwise min over R k . Implementation details are provided in Appendix B. Empirically, we demonstrate the benefits of this reduction, when it is applicable, in Sections 5.3 and Apendix C.

S-AP

Environment

Subgoal-induced observationreduction

SAP

SAP = (dgrn  d d a,--),d  R

(proposition-wise LiDAR distances)

Sreach

Savoid

d=min(dblue，dyellow）

dgreen

Agent

Subgoal

=(a+,A-)

α+={green}

A={{blue},{yellow}}

=

（SAP,Sreach，Savoid）

sampler

(reach green)

(avoidblue oryellow)

4.3 Goal-Conditioned Policy Learning with Reachability Constraints

Akey difference between our approach and existing subgoal-based approaches lies in the treatment of the avoid subgoal. We treat an avoid subgoal as a hard constraint, since whenever a violation occurs, satisfying the LTL formula becomes impossible. For example, consider the specification ¬ a U b , where a and b are atomic propositions; this requires the agent to avoid reaching any state labeled with a until a state labeled with b is reached. If a is encountered before b , the whole trajectory can no longer satisfy the specification. To enforce these hard constraints, we leverage the notion of HJ reachability as introduced in Section 3 and formulate our RL objective as follows:

where ξ is the set of all possible subgoals (details on its construction are provided in Appendix B), s σ := f ( s, σ ) is the processed state if the observation reduction is applied, and s σ := ( s, σ ) otherwise, S σ f := { s σ | σ, ∃ π, V π h ( s σ ) ≤ 0 } denotes the largest subgoal-dependent feasible set, S σ 0 := { s σ | σ, s ∼ d 0 } denotes the set of processed initial states given a subgoal σ , and V π h ( s σ ) is the reachability value function of a policy π starting from a processed state s σ with the current subgoal σ . We use π as a shorthand for the policy π ( a | s σ ) . However, since S σ f is unknown when learning policies, directly solving Eq. (1) is not possible. Inspired by previous methods [1, 57], we convert the objective into the following constrained policy optimization problem:

where A π r ( s σ , a ) := Q π r ( s σ , a ) -V π r ( s σ ) and A π h ( s σ , a ) := Q π h ( s σ , a ) -V π h ( s σ ) are the advantages functions with Q π h ( s σ , a ) := max t ∈ N h ( s σ t ) , s 0 = s σ , a 0 = a, a t ∼ π and J h ( π ) := max t ∈ N h ( · ) , a ∼ π is the maximum safety violation for a trajectory under policy π . By maximizing the discounted cumulative reward, the policy is optimized to effectively satisfy the reach subgoal, while the safety constraints ensures that the avoid subgoals are satisfied, Then we incorporate the clipped surrogate objective [43] to handle the trust-region constraint and a state-dependent Lagrangian multiplier [57] to enforce the state-wise safety constraint:

The overall method is summarized in Algorithm 1. The training process of our method is simple compared to previous methods that sample full LTL specifications [50, 40] or employ curriculum learning that gradually exposes the policy to more challenging reach-avoid sequences [23]. We sample one subgoal at a time, apply observation reduction to obtain a reduced state when applicable, and execute the policy to collect data. When the subgoal is satisfied, a new one is sampled, and this cycle continues until the episode terminates due to reaching the maximum length or a subgoal violation. During evaluation, given an LTL formula φ , we convert it to a Büchi automaton and track the current automaton state q . We identify the set of candidate subgoals { σ i } n i =1 that can advance toward satisfying φ . The subgoal that achieves the best trade-off between task progression and satisfying safety constraint is selected as σ ∗ = arg max σ V r ( s σ ) -λ ( s σ ) V h ( s σ ) . The agent takes actions based on the selected subgoal and updates it upon each transition to a new automaton state.

Subgoal switching. In general, we cannot determine a priori whether a selected subgoal is satisfiable in the environment. To handle this issue, we introduce a mechanism to identify unsatisfiable subgoals and allow the agent to switch to alternative ones. The mechanism is triggered if the current subgoal is not satisfied within a timeout threshold, which is set to (1 + ϵ scale ) · µ subgoal, where µ subgoal denotes the maximum number of steps required to complete a subgoal after policy convergence during training, and ϵ scale is a scaling factor. When a reach subgoal times out, the best option from the remaining reach subgoals in A + is selected. This procedure is sound because the avoid subgoals are respected during the attempt for the prior reach subgoal(s). Appendix B.2 provides further details.

Remark. If there is no more subgoal remaining, our method returns failure. However, this does not necessarily mean that the specification is unsatisfiable. It is possible that the specification is satisfiable but the greedy nature of solving the subgoals sequentially prevents us from finding a solution. In general, unless the environment MDP is known, we cannot determine a priori if the LTL specification is satisfiable or not. If the agent is allowed to reset in the same environment, then one can enumerate the possible subgoal sequences from the initial state of the automaton.

5 Experiments

In this section, we evaluate our method GenZ-LTL in multiple environments and across a wide range of LTL specifications of varying complexity to answer the following questions: Q1: How well can GenZ-LTL zero-shot generalize to arbitrary LTL specifications? Q2: How does GenZ-LTL perform under increasing specification complexity and environment complexity? Q3: What are the individual impacts of the proposed observation reduction and safe RL approach? Additional evaluations and results in are provided in Appendix C.

Environments. Our environments include the LetterWorld environment [50], a 7 × 7 discrete grid world with discrete actions where letters are placed at randomly sampled positions, and the ZoneEnv environment [50], a high-dimensional environment with lidar observations, where a robotic agent with a continuous action space must navigate between randomly positioned colored regions. We exclude the FlatWorld environment [23, 51] from our experiments, as its configuration is fixed and the state space includes only the agent's ( x, y ) -position, without any information about the environment state. We also consider a variant of ZoneEnv that contains overlapping regions which is by design more challenging than FlatWorld . Additional details are provided in Appendix A.1.

Baselines. We compare GenZ-LTL against the following baselines, all of which train a goalconditioned policy of some sort: 1) LTL2Action [50], which applies a technique called LTL progression to dynamically track the parts of a specification that remain to be satisfied, and use a graph neural network to encode the specification; 2) GCRL-LTL [40], which uses a heuristic to identify the optimal subgoal sequence; 3) DeepLTL [23], which learns policies conditioned on embeddings of subgoal sequences; 4) RAD-embeddings [54], which pre-trains embeddings of compositions of deterministic finite automata (DFAs) and learns policies conditioned on these embeddings. While these baselines consider safety, as the LTL specification itself can involve the notion of safety, they do not model safety constraints explicitly in their RL formulation. Further details of the implementation are provided in Appendix B.

LTL specifications. We consider a wide variety of LTL specifications with varying complexity, including (1) sequential reach-only specifications of the form F ( α 1 ∧ ( F α 2 ∧ ( . . . ∧ F α n )) ; (2) sequential reach-avoid specifications of the form ¬ α 1 U ( α 2 ∧ ( ¬ α 3 U ( α 4 ∧ . . . ∧ ( ¬ α n -1 U α n )) , and (3) complex specifications that combine reach-only and reach-avoid properties. We also conduct a separate evaluation on infinite-horizon tasks that span a diverse set of LTL objectives, including safety, liveness, and their combinations. The specifications used for evaluation are listed in Table 1 and a full set of specifications is provided in Appendix A.2, with trajectory visualizations in Appendix D. In this section, all specifications are satisfiable.

LetterWorld

ZoneEnv

Finite-horizon

φ

1

F

(

a

∧

(

¬

b U c

))

∧

Fd

φ

9

(

Fb

)

∧

(

¬

b U

(

g

∧

Fy

))

φ

2

(

Fd

)

∧

(

¬

f U

(

d

∧

Fb

))

φ

10

¬

(

m

∨

y

)

U

(

b

∧

Fg

)

φ

3

¬

a U

(

b

∧

(

¬

c U

(

d

∧

(

¬

e U f

))))

φ

11

¬

g U

((

b

∨

m

)

∧

(

¬

g U y

))

φ

4

(

a

∨

b

∨

c

∨

d

F

(

e

∧

F

(

f

∧

Fg

)))

U

(

h

∧

Fi

)

φ

12

(

g

∨

b

(

¬

y U m

))

U y

φ

5

F

(

d

∧

(

¬

(

a

∨

b

)

U

(

b

∧

(

¬

e U c

))))

∧

F

(

¬

(

f

∨

g

∨

h

)

U a

)

φ

13

F

(

g

∧

(

¬

(

b

∨

y

)

U

(

y

∧

(

¬

m U b

))))

∧

F

(

¬

g U y

)

φ

6

F

((

k

∧

((

¬

b

∨

c

)

U f

))

∧

(

¬

(

a

∨

e

∨

h

)

U g

))

∧

Fd

φ

14

F

((

b

∨

g

)

∧

(

¬

y U

(

b

∧

(

¬

(

g

∨

m

)

U m

))))

∧

F

(

y

∧

(

¬

))

φ

7

¬

(

j

∨

b

∨

d

)

U

(

a

∧

(

¬

c U

(

f

∧

F

(

g

∧

(

¬

d U e

)))))

φ

15

¬

(

m

∨

y

)

U

(

b

∧

(

¬

g U

(

y

∧

F

(

g

∧

(

¬

b U m

)))))

φ

8

¬

(

f

∨

g

)

U

(

a

∧

(

¬

b U c

)

∧

F

(

d

∧

(

¬

e U f

)))

φ

16

F

(

b

∧

(

¬

y U

(

g

∧

F

(

y

∧

(

¬

(

m

∨

g

)

U b

)))))

Infinite-horizon

ψ

1

GF

(

e

∧

(

¬

a

U f

))

∧

G

¬

(

c

∨

d

)

ψ

4

GFb

∧

GFg

∧

G

¬

(

y

∨

m

)

ψ

2

GFa

∧

GFb

∧

GFc

∧

G

¬

(

e

∨

f

∨

i

)

ψ

5

GFb

∧

GFy

∧

GFg

∧

G

¬

m

ψ

3

GFc

∧

GFa

∧

GF

(

e

∧

(

¬

f

U g

))

∧

GFk

∧

G

¬

(

i

∨

j

)

ψ

6

FGy

∧

G

¬

(

g

∨

b

∨

m

)

Evaluation metrics. For fair comparisons, we follow the standard practice of executing a deterministic policy by taking the mean of the action distribution [22]. For each LTL specification, we report the following rates: success rate η s , defined as the ratio of trajectories that satisfy the specification; violation rate η v , defined as the ratio of trajectories that violate it; and other rate η o = 1 -( η s + η v ) . We also report the average number of steps µ to satisfy the specification among successful trajectories as a secondary metric, as step count is less important when the success rate is low.

5.1 Can GenZ-LTL zero-shot generalize to arbitrary LTL specifications (Q1)?

Finite-horizon tasks. We first evaluate finite-horizon tasks, i.e. tasks that can be satisfied with finitelength trajectories. All the specifications considered are complex LTL formulas with nested temporal and Boolean operators (details in Table 1). The evaluation results are shown in Table 2. Our method consistently outperforms the baselines in terms of both success and violation rates. LTL2Action encodes the full structure of the current LTL specification, making it less effective at adapting to out-of-distribution (OOD) specifications at test time. GCRL-LTL learns a goal-conditioned policy but does not model safety constraints explicitly, limiting its ability to enforce constraint satisfaction. DeepLTL relies on curriculum training which samples random reach-avoid subgoal sequences only up to a certain length. When evaluating specifications φ 9 -φ 12 , although the specifications themselves are unseen, the underlying subgoal sequences are likely to have been encountered during training, leading to better efficiency, i.e., smaller µ . However, for specifications with longer subgoal sequences, e.g, φ 13 -φ 16 , which are OOD with respect to the training sequences, DeepLTL exhibits reduced performance. RAD-embeddings pre-trains the automaton embeddings by sampling from a fixed distribution. As a result, it also struggles with the issue of OOD generalization when faced with unseen

LTL2Action

GCRL-LTL

DeepLTL

RAD-embeddings

GenZ-LTL(ours)

ηs

↑

ηv

↓

µ

↓

ηs

↑

ηv

↓

µ

↓

ηs

↑

ηv

↓

µ

↓

ηs

↑

ηv

↓

µ

↓

ηs

↑

ηv

↓

µ

↓

LetterWorld

φ

1

0

.

51

±

0

.

10

0

.

00

±

0

.

00

29

.

03

±

3

.

08

0

.

87

±

0

.

04

0

.

02

±

0

.

02

15

.

37

±

0

.

39

0

.

87

±

0

.

02

0

.

00

±

0

.

00

8

.

12

±

0

.

21

0

.

96

±

0

.

04

0

.

00

±

0

.

00

18

.

71

±

2

.

06

0

.

98

±

0

.

02 0

.

00

±

0

.

00

7

.

38

±

0

.

13

φ

2

0

.

66

±

0

.

16

0

.

14

±

0

.

11

23

.

25

±

4

.

93

0

.

90

±

0

.

03

0

.

00

±

0

.

00

9

.

26

±

0

.

75

0

.

91

±

0

.

02

0

.

01

±

0

.

01

5

.

88

±

0

.

12

0

.

90

±

0

.

07

0

.

07

±

0

.

04

13

.

64

±

2

.

71

1

.

00

±

0

.

00 0

.

00

±

0

.

00

5

.

48

±

0

.

09

φ

3

0

.

75

±

0

.

07

0

.

14

±

0

.

04

24

.

60

±

4

.

38

0

.

72

±

0

.

06

0

.

09

±

0

.

07

14

.

02

±

1

.

09

0

.

81

±

0

.

02

0

.

01

±

0

.

00

9

.

00

±

0

.

14

0

.

86

±

0

.

05

0

.

08

±

0

.

03

20

.

02

±

2

.

05

0

.

96

±

0

.

01 0

.

00

±

0

.

00

8

.

61

±

0

.

46

φ

4

0

.

56

±

0

.

19

0

.

00

±

0

.

00

29

.

69

±

8

.

50

0

.

99

±

0

.

00 0

.

00

±

0

.

00

25

.

54

±

0

.

84

0

.

89

±

0

.

02

0

.

00

±

0

.

00

7

.

05

±

0

.

28

0

.

89

±

0

.

04

0

.

00

±

0

.

00

18

.

78

±

2

.

82

0

.

98

±

0

.

01

0

.

00

±

0

.

00

7

.

41

±

0

.

37

Ave.

0

.

62

±

0

.

16

0

.

07

±

0

.

09

26

.

64

±

5

.

87

0

.

87

±

0

.

11

0

.

03

±

0

.

05

16

.

05

±

6

.

13

0

.

87

±

0

.

04

0

.

00

±

0

.

01

7

.

51

±

1

.

21

0

.

90

±

0

.

06

0

.

04

±

0

.

04

17

.

79

±

3

.

37

0

.

98

±

0

.

02 0

.

00

±

0

.

00

7

.

22

±

1

.

18

φ

5

0

.

30

±

0

.

13

0

.

00

±

0

.

00

39

.

12

±

2

.

08

0

.

67

±

0

.

07

0

.

07

±

0

.

05

20

.

53

±

0

.

49

0

.

78

±

0

.

05

0

.

00

±

0

.

00

11

.

24

±

0

.

39

0

.

88

±

0

.

05

0

.

00

±

0

.

00

26

.

02

±

1

.

62

0

.

99

±

0

.

00 0

.

00

±

0

.

00

10

.

60

±

0

.

12

φ

6

0

.

18

±

0

.

10

0

.

00

±

0

.

00

28

.

40

±

6

.

15

0

.

70

±

0

.

08

0

.

15

±

0

.

05

14

.

69

±

0

.

75

0

.

81

±

0

.

02

0

.

00

±

0

.

00

8

.

23

±

0

.

18

0

.

90

±

0

.

04

0

.

00

±

0

.

00

19

.

48

±

1

.

96

0

.

95

±

0

.

01 0

.

00

±

0

.

00

7

.

48

±

0

.

06

φ

7

0

.

29

±

0

.

16

0

.

33

±

0

.

20

40

.

71

±

1

.

43

0

.

59

±

0

.

05

0

.

14

±

0

.

07

19

.

52

±

1

.

25

0

.

71

±

0

.

02

0

.

03

±

0

.

01

11

.

82

±

0

.

39

0

.

73

±

0

.

05

0

.

17

±

0

.

05

26

.

07

±

3

.

73

0

.

94

±

0

.

01 0

.

00

±

0

.

00

10

.

97

±

0

.

08

φ

8

0

.

20

±

0

.

08

0

.

47

±

0

.

22

37

.

50

±

4

.

16

0

.

63

±

0

.

09

0

.

09

±

0

.

07

20

.

10

±

0

.

82

0

.

75

±

0

.

02

0

.

02

±

0

.

01

11

.

20

±

0

.

25

0

.

76

±

0

.

11

0

.

13

±

0

.

05

25

.

60

±

2

.

89

0

.

93

±

0

.

01 0

.

00

±

0

.

00

10

.

24

±

0

.

08

Ave.

0

.

24

±

0

.

12

0

.

20

±

0

.

25

36

.

43

±

6

.

08

0

.

65

±

0

.

08

0

.

11

±

0

.

06

18

.

71

±

2

.

54

0

.

76

±

0

.

05

0

.

01

±

0

.

02

10

.

62

±

1

.

47

0

.

82

±

0

.

10

0

.

07

±

0

.

08

24

.

29

±

3

.

77

0

.

95

±

0

.

03 0

.

00

±

0

.

00

9

.

82

±

1

.

42

ZoneEnv

φ

9

0

.

13

±

0

.

08

0

.

18

±

0

.

25

496

.

09

±

128

.

52

0

.

88

±

0

.

05

0

.

03

±

0

.

00

474

.

41

±

23

.

42

0

.

89

±

0

.

03

0

.

04

±

0

.

00

327

.

80

±

23

.

78

0

.

90

±

0

.

07

0

.

08

±

0

.

07

423

.

11

±

51

.

08

0

.

99

±

0

.

01 0

.

01

±

0

.

01

380

.

15

±

5

.

17

φ

10

0

.

39

±

0

.

32

0

.

38

±

0

.

24

421

.

46

±

58

.

76

0

.

88

±

0

.

03

0

.

06

±

0

.

01

303

.

43

±

4

.

46

0

.

88

±

0

.

03

0

.

09

±

0

.

01

221

.

94

±

23

.

26

0

.

92

±

0

.

02

0

.

06

±

0

.

02

301

.

21

±

101

.

99

0

.

97

±

0

.

01 0

.

02

±

0

.

02

253

.

44

±

10

.

22

φ

11

0

.

82

±

0

.

15

0

.

08

±

0

.

03

284

.

41

±

95

.

62

0

.

86

±

0

.

04

0

.

06

±

0

.

03

306

.

94

±

11

.

26

0

.

91

±

0

.

06

0

.

04

±

0

.

02

215

.

42

±

15

.

01

0

.

93

±

0

.

06

0

.

04

±

0

.

02

247

.

83

±

42

.

98

0

.

99

±

0

.

01 0

.

01

±

0

.

01

249

.

57

±

11

.

85

φ

12

0

.

94

±

0

.

03

0

.

02

±

0

.

03

122

.

89

±

20

.

97

0

.

90

±

0

.

01

0

.

05

±

0

.

02

136

.

32

±

10

.

27

0

.

97

±

0

.

01

0

.

01

±

0

.

00

116

.

40

±

13

.

07

1

.

00

±

0

.

01 0

.

00

±

0

.

00 105

.

97

±

17

.

31 1

.

00

±

0

.

00 0

.

00

±

0

.

00

135

.

60

±

8

.

06

Ave.

0

.

57

±

0

.

37

0

.

17

±

0

.

21

331

.

21

±

165

.

88

0

.

88

±

0

.

04

0

.

05

±

0

.

02

305

.

28

±

123

.

33

0

.

91

±

0

.

05

0

.

04

±

0

.

03

220

.

39

±

78

.

77

0

.

94

±

0

.

05

0

.

04

±

0

.

05

269

.

53

±

129

.

95

0

.

99

±

0

.

01 0

.

01

±

0

.

01

254

.

69

±

89

.

18

φ

13

0

.

17

±

0

.

23

0

.

01

±

0

.

01

886

.

66

±

104

.

03

0

.

73

±

0

.

06

0

.

09

±

0

.

02

618

.

42

±

22

.

80

0

.

91

±

0

.

03

0

.

00

±

0

.

00

508

.

26

±

54

.

81

0

.

69

±

0

.

17

0

.

00

±

0

.

00

664

.

59

±

29

.

91

0

.

98

±

0

.

02 0

.

00

±

0

.

00 436

.

25

±

18

.

35

φ

14

0

.

36

±

0

.

29

0

.

01

±

0

.

01

885

.

13

±

148

.

93

0

.

68

±

0

.

06

0

.

11

±

0

.

06

611

.

24

±

38

.

37

0

.

91

±

0

.

03

0

.

00

±

0

.

00

473

.

45

±

52

.

67

0

.

77

±

0

.

21

0

.

00

±

0

.

00

628

.

57

±

22

.

89

0

.

98

±

0

.

01 0

.

00

±

0

.

00 434

.

45

±

14

.

61

φ

15

0

.

08

±

0

.

05

0

.

49

±

0

.

30

855

.

56

±

311

.

81

0

.

70

±

0

.

06

0

.

10

±

0

.

03

599

.

71

±

18

.

62

0

.

78

±

0

.

05

0

.

12

±

0

.

05

523

.

50

±

64

.

73

0

.

62

±

0

.

09

0

.

02

±

0

.

01

680

.

11

±

9

.

51

0

.

97

±

0

.

01 0

.

01

±

0

.

00 388

.

01

±

15

.

47

φ

16

0

.

03

±

0

.

04

0

.

02

±

0

.

04

931

.

19

±

420

.

81

0

.

68

±

0

.

09

0

.

08

±

0

.

04

596

.

29

±

25

.

87

0

.

89

±

0

.

10

0

.

00

±

0

.

00

515

.

48

±

45

.

53

0

.

70

±

0

.

18

0

.

00

±

0

.

00

616

.

28

±

59

.

88

0

.

99

±

0

.

01 0

.

00

±

0

.

00 396

.

83

±

20

.

39

Ave.

0

.

12

±

0

.

18

0

.

17

±

0

.

28

886

.

39

±

288

.

60

0

.

70

±

0

.

07

0

.

09

±

0

.

04

606

.

42

±

26

.

76

0

.

87

±

0

.

08

0

.

03

±

0

.

06

505

.

17

±

54

.

03

0

.

70

±

0

.

15

0

.

00

±

0

.

01

647

.

39

±

40

.

74

0

.

98

±

0

.

02

0

.

01

±

0

.

01

408

.

71

±

27

.

47

automata, which is reflected in its performance degradation under more complex LTL specifications: the success rate η s drops from 0.9 on φ 1 -4 to 0.82 on φ 5 -8 and 0.94 on φ 9 -12 to 0.7 on φ 13 -16 . In contrast, our method conditions only on the current subgoal, sidestepping the issue of OOD subgoal sequences. By incorporating an HJ reachability constraint, our method improves adherence to safety constraints. Additionally, since we optimize for the discounted cumulative reward obtained upon satisfying a subgoal, efficient policies that take fewer steps are encouraged. We further evaluate our method in ZoneEnv with agents featuring more complex dynamics, which induce additional hard constraints such as preventing head-down falls. Our method consistently outperforms the baselines in both success and violation rates. Details of the setup and results are provided in Appendix C.

Infinite-horizon tasks. Next, we evaluate performance on infinite-horizon tasks, i.e. tasks that can only be satisfied with infinite-length trajectories. RAD-embeddings is excluded from this evaluation, as DFAs cannot represent such specifications. Since it is not possible to simulate infinite trajectories in these environments, we set the maximum episode length to be 10 times of the length used during training. The evaluated specifications are listed in Table 1, and results are reported in Table 3, including the average number of visits to accepting states for non-violating trajectories µ acc and the violation rate that is calculated based on the proportion of trajectories that violate the G ¬ α constraint in the evaluated specifications. For example, in the specification GF α 1 ∧ GF α 2 ∧···∧ GF α n -1 ∧ G ¬ α n , each successful cycle of α 1 -α n -1 (in any order) increases µ acc, while any visit to α n counts as a violation. In the case of FG α , µ acc is computed as the number of consecutive steps, counted backward from the final timestep, in which α holds. As shown

in Table 3, our method outperforms the baselines on both metrics. By conditioning only on the current subgoal, it naturally extends to infinite-horizon tasks. The agent can follow the infinite subgoal sequence extracted from the automaton one subgoal at a time. The integration of reachability constraints further enables safe execution, whereas the baselines often struggle to satisfy safety constraints in long-horizon settings.

5.2 How does GenZ-LTL perform under increasing specification complexity and environment complexity (Q2)?

To further evaluate the zero-shot adaptation ability of our method, we design experiments that test performance under increasing complexity in specification and environment: (1) Subgoal chaining: we evaluate performance on reach-only and reach-avoid specifications with an increasing of sequence length of subgoals up to n = 12 ; (2) Environment complexity: we vary the region size and the number of regions associated with each atomic proposition; and (3) Unsatisfiable subgoals.

DeepLTL

GCRL-LTL

LTL2Action

GenZ-LTL(ours)

Reach-only specifications

Reach-avoidspecifications

1.0

1.0

0.8

0.5

ns

0.8

No 0.4

0.6

ns

0.8

0.6

nv

0.6

No0.3-

0.4

0.6

0.4

0.4

0.2

0.4

0.2

0.2

0.2

0.1

0.0

0.0

0.0

0.0

4

6

8

10

12

4

6

8

10

12

2

4

6

8

10

2

4

6

8

10

2

4

6

8

10

Sequencelengthn

Subgoal chaining. In this experiment, we increase the number of subgoals that the agent must reach sequentially. Since satisfying longer specifications naturally requires more steps, we double the maximum episode length used in Section 5.1. The results are shown in Figure 4. For the reach-only specifications, the performance of DeepLTL and GCRL-LTL degrades more rapidly as n grows. In the case of DeepLTL, it suffers from OOD specifications that involve longer subgoal sequences than those used during training. GCRL-LTL performs the worst as it struggles to complete the subgoals in an efficient manner, preventing it from reaching all the subgoals within the maximum episode length. LTL2Action performs better than DeepLTL and GCRL-LTL on reach-only specifications, as the corresponding graph structures remain relatively simple compared to the complex and nested specifications in Section 5.1. Our method maintains a high success rate and a low violation rate despite the increase in sequence length, as it solves one subgoal at a time, thus avoiding the distribution shift that affects other methods and resulting in better generalization. For reach-avoid specifications, where safety satisfaction is critical, we can observe that our method achieves near-zero constraint violations compared to the baselines, which validates our safe RL approach with reachability constraints.

DeepLTL

GCRL-LTL

LTL2Action

GenZ-LTL(ours)

ns

06°0

0.95

1.00

70.15

0.25

0.20

No

0.06

0.04

Ns0.7

1.0

0.9

0.8

nv

0.6

0.5

0.4

N00.2

0.3

0.4

0.85

0.10

0.02

9°0

0.3

0.80

0.05

+00°0

0.5

0.4

0.2

0.1

0.1

0.75

00°0

0.3

0.0 -

0.0

0.2

0.3

0.4

0.5

0.6

0.2

0.3

0.4

0.5

0.6

0.2

0.3

0.4

0.5

0.6

1

2

3

4

5

1

Z

3

4

5

1

2

3

4

5

Zonesize

Zonecount

Environment complexity. We next investigate the impact of varying environment complexity performance. The results are shown in Figure 5. As the zone size increases or the number of zones associated with each atomic proposition increases, satisfying the safety constraints becomes more challenging. While all methods experience performance degradation, our method achieves the highest success rates and the lowest violation rate, which highlights its better ability to comply with safety constraints and generalize under environment variations.

Unsatisfiable subgoals. To evaluate the subgoal-switching mechanism, we use a satisfiable specification that contains an unsatisfiable subgoal: ϕ 1 : ¬ yellow U (( blue ∧ green ) ∨ magenta ) in ZoneEnv . We randomly generate region layouts such that the green and blue regions (without overlaps) are positioned closer to the agent than the magenta region. In this setup, blue ∧ green is usually selected as the first subgoal based on the learned value functions. As shown in Figure 6, the baselines struggle to satisfy the specification: the subgoal is unsatisfiable due to non-overlapping regions, and the agent gets stuck due to the lack of a mechanism to switch to an alternative. In contrast, our method can escape this deadlock by selecting an alternative subgoal. The trajectories in Figure 11 in Appendix C further illustrate this behavior.

ns

nv

No

Unsatisfiablesubgoals

1.0

0.8

Rate

0.6

0.4

0.2

0.0

LTL2Action

GCRL-LTL

DeepLTL

Ours

5.3 Ablation study: what are the individual impacts of the proposed observation reduction and safe RL approach (Q3)?

We consider a ZoneEnv environment with one region per color and reach-avoid specifications with a sequence length of n = 2 . The environment layout is randomized in a controlled manner such that the region to be avoided is placed between the agent and the region to be reached according to an LTL specification. Under this setting, the agent must first circumvent the avoid region before reaching the goal region. For example, consider ¬ magenta U blue , the magenta area is placed between the agent and the blue area. We construct a variant of our method by removing the observation reduction module and instead encoding the subgoal using the bitvector encoding discussed in Section 4.1. We also include DeepLTL as a reference, as it achieves the best performance among the baselines. The results are shown in Figure 7. We can observe that integrating the state-wise constraints leads to improved safety performance

ns

nv

no

LettweWorld

ZoneEnv

1.000

1.0

0.975

0.9

0.950

Rate

0.925 -

0.8

0.900

0.7

0.875

0.6

0.850

0.825

0.5

0.800

DeepLTL Ours

Ours

0.4

DeepLTL Ours

Ours

(w/o s)

(w/o s%)

compared to the baseline method, and further incorporating the subgoal-induced observation reduction technique improves performance by significantly reducing the input dimension of the policy.

6 Conclusion

This paper presents a novel method for learning RL policies that generalize, in a zero-shot manner, to arbitrary LTL specifications by performing subgoal decomposition through the corresponding Büchi automata and completing one subgoal at a time. Empirical results demonstrate that our method, despite being myopic, achieves substantially better generalization to complex and unseen specifications compared to existing baselines. We posit that the real complexity of LTL task generalization does not lie in the structure of the LTL formula or the corresponding automata, but rather in the combinatorial nature of the atomic propositions associated with the MDP states. The observation reduction technique that we introduce is a step towards addressing this complexity. Future work will explore alternative techniques that can exploit the Boolean structure within a subgoal, such as Boolean task algebra [37], with the goal of further improving the sample efficiency and generalization ability of our method.

References

J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

C. Allen, N. Parikh, O. Gottesman, and G. Konidaris. Learning markov state abstractions for deep reinforcement learning. Advances in Neural Information Processing Systems , 34:82298241, 2021.

M. Bagatella, A. Krause, and G. Martius. Directed Exploration in Reinforcement Learning from Linear Temporal Logic. In Seventeenth European Workshop on Reinforcement Learning , Oct. 2024.

H. Bharadhwaj, A. Kumar, N. Rhinehart, S. Levine, F. Shkurti, and A. Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.14497 , 2020.

J. R. Büchi. Symposium on decision problems: On a decision method in restricted second order arithmetic. In Studies in Logic and the Foundations of Mathematics , volume 44, pages 1-11. Elsevier, 1966.

Y. Cheng, P. Zhao, and N. Hovakimyan. Safe and efficient reinforcement learning using disturbance-observer-based control barrier functions. In Learning for Dynamics and Control Conference , pages 104-115. PMLR, 2023.

M. Cho and C. Sun. Constrained meta-reinforcement learning for adaptable safety guarantee with differentiable convex programming. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 20975-20983, 2024.

S. Cui and P. Mitra. Automated multi-task learning for joint disease prediction on electronic health records. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757 , 2018.

J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin. Bridging hamiltonjacobi safety analysis and reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8550-8556. IEEE, 2019.

J. Fu and U. Topcu. Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints. In Robotics: Science and Systems X , July 2014.

M. Ganai, S. Gao, and S. Herbert. Hamilton-jacobi reachability in reinforcement learning: A survey. IEEE Open Journal of Control Systems , 2024.

M. Ganai, Z. Gong, C. Yu, S. Herbert, and S. Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems , 36:69764-69797, 2023.

B. Gangopadhyay, P. Dasgupta, and S. Dey. Safe and stable rl (s 2 rl) driving policies using control barrier and control lyapunov functions. IEEE Transactions on Intelligent Vehicles , 8(2):1889-1899, 2022.

S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Z. Guo, W. Zhou, and W. Li. Temporal logic specification-conditioned decision transformer for offline safe reinforcement learning. In Forty-first International Conference on Machine Learning , 2024.

Z. Guo, W. Zhou, S. Wang, and W. Li. Constraint-conditioned actor-critic for offline safe reinforcement learning. In The Thirteenth International Conference on Learning Representations , 2025.

E. M. Hahn, M. Perez, S. Schewe, F. Somenzi, A. Trivedi, and D. Wojtczak. Mungojerrie: Linear-Time Objectives in Model-Free Reinforcement Learning. In Tools and Algorithms for the Construction and Analysis of Systems , Lecture Notes in Computer Science, pages 527-545, 2023.

P. Hansen-Estruch, A. Zhang, A. Nair, P. Yin, and S. Levine. Bisimulation makes analogies in goal-conditioned reinforcement learning. In International Conference on Machine Learning , pages 8407-8426. PMLR, 2022.

M. Hasanbeig, A. Abate, and D. Kroening. Logically-constrained reinforcement learning. arXiv preprint arXiv:1801.08099 , 2018.

A. Hendawy, J. Peters, and C. D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Representations , 2024.

S. Huang, Q. Gallouédec, F. Felten, A. Raffin, R. F. J. Dossa, Y. Zhao, R. Sullivan, V . Makoviychuk, D. Makoviichuk, M. H. Danesh, C. Roumégous, J. Weng, C. Chen, M. M. Rahman, J. G. M. Araújo, G. Quan, D. Tan, T. Klein, R. Charakorn, M. Towers, Y. Berthelot, K. Mehta, D. Chakraborty, A. KG, V. Charraut, C. Ye, Z. Liu, L. N. Alegre, A. Nikulin, X. Hu, T. Liu, J. Choi, and B. Yi. Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning. arXiv preprint arXiv:2402.03046 , 2024.

M. Jackermeier and A. Abate. Deepltl: Learning to efficiently satisfy complex ltl specifications for multi-task rl. In The Thirteenth International Conference on Learning Representations , 2025.

A. K. Jayant and S. Bhatnagar. Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems , 35:24432-24445, 2022.

J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

K. Jothimurugan, R. Alur, and O. Bastani. A composable specification language for reinforcement learning tasks. Advances in Neural Information Processing Systems , 32, 2019.

D. Kim, K. Lee, and S. Oh. Trust region-based safe distributional reinforcement learning for multiple constraints. Advances in neural information processing systems , 36:19908-19939, 2023.

K. Kim, G. Swamy, Z. Liu, D. Zhao, S. Choudhury, and S. Z. Wu. Learning shared safety constraints from multi-task demonstrations. Advances in Neural Information Processing Systems , 36:5808-5826, 2023.

B. G. León, M. Shanahan, and F. Belardinelli. Systematic generalisation through task temporal logic and deep reinforcement learning. arXiv preprint arXiv:2006.08767 , 2020.

B. G. León, M. Shanahan, and F. Belardinelli. Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning. In arXiv . arXiv, 2021.

B. G. León, M. Shanahan, and F. Belardinelli. In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications. In International Conference on Learning Representations , 2022.

X. Li, C.-I. Vasile, and C. Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3834-3839. IEEE, 2017.

Q. Lin, B. Tang, Z. Wu, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference on Machine Learning , pages 21127-21152. PMLR, 2023.

J. X. Liu, A. Shah, E. Rosen, M. Jia, G. Konidaris, and S. Tellex. Skill Transfer for Temporal Task Specification. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 2535-2541, 2024.

M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299 , 2022.

Z. Ma, X. Liu, and Y. Huang. Unsupervised reinforcement learning for multi-task autonomous driving: Expanding skills and cultivating curiosity. IEEE Transactions on Intelligent Transportation Systems , 2024.

G. Nangue Tasse, S. James, and B. Rosman. A boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems , 33:9497-9507, 2020.

J. Oh, S. Singh, H. Lee, and P. Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning , pages 2661-2670. PMLR, 2017.

A. Pnueli. The temporal logic of programs. In 18th annual symposium on foundations of computer science (sfcs 1977) , pages 46-57. ieee, 1977.

W. Qiu, W. Mao, and H. Zhu. Instructing goal-conditioned reinforcement learning agents with temporal logic objectives. Advances in Neural Information Processing Systems , 36:3914739175, 2023.

A. Rafiei, R. Moore, S. Jahromi, F. Hajati, and R. Kamaleswaran. Meta-learning in healthcare: A survey. SN Computer Science , 5(6):791, 2024.

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

A. Shah, C. Voloshin, C. Yang, A. Verma, S. Chaudhuri, and S. A. Seshia. Deep policy optimization with temporal logic constraints. arXiv e-prints , pages arXiv-2404, 2024.

A. Shah, C. Voloshin, C. Yang, A. Verma, S. Chaudhuri, and S. A. Seshia. LTL-Constrained Policy Optimization with Cycle Experience Replay. In arXiv , 2024.

D. Shao and M. Kwiatkowska. Sample efficient model-free reinforcement learning from LTL specifications with optimality guarantees. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 4180-4189, 2023.

O. So and C. Fan. Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learning. In Proceedings of Robotics: Science and Systems , July 2023.

A. A. Taiga, R. Agarwal, J. Farebrother, A. Courville, and M. G. Bellemare. Investigating multitask pretraining and generalization in reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Z. Tang, X. Chen, Y. Li, and J. Chen. Efficient and generalized end-to-end autonomous driving system with latent deep reinforcement learning and demonstrations. arXiv preprint arXiv:2401.11792 , 2024.

P. Vaezipoor, A. C. Li, R. A. T. Icarte, and S. A. Mcilraith. Ltl2action: Generalizing ltl instructions for multi-task rl. In International Conference on Machine Learning , pages 1049710508. PMLR, 2021.

C. Voloshin, A. Verma, and Y. Yue. Eventual discounting temporal logic counterfactual experience replay. In International Conference on Machine Learning , pages 35137-35150. PMLR, 2023.

A. Wachi, Y. Sui, Y. Yue, and M. Ono. Safe exploration and optimization of constrained mdps using gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

N. Xiong, Y. Du, and L. Huang. Provably safe reinforcement learning with step-wise violation constraints. Advances in Neural Information Processing Systems , 36:54341-54353, 2023.

B. Yalcinkaya, N. Lauffer, M. Vazquez-Chanlatte, and S. Seshia. Compositional automata embeddings for goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems , 37:72933-72963, 2024.

B. Yalcinkaya, N. Lauffer, M. Vazquez-Chanlatte, and S. A. Seshia. Provably correct automata embeddings for optimal automata-conditioned reinforcement learning. arXiv preprint arXiv:2503.05042 , 2025.

T.-Y. Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge. Projection-based constrained policy optimization. In International Conference on Learning Representations , 2020.

D. Yu, H. Ma, S. Li, and J. Chen. Reachability constrained reinforcement learning. In International conference on machine learning , pages 25636-25655. PMLR, 2022.

H. Zang, X. Li, L. Zhang, Y. Liu, B. Sun, R. Islam, R. Tachet des Combes, and R. Laroche. Understanding and addressing the pitfalls of bisimulation-based representations in offline reinforcement learning. Advances in Neural Information Processing Systems , 36:28311-28340, 2023.

Y. Zhang, Q. Vuong, and K. Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems , 33:15338-15349, 2020.

W. Zhao, T. He, and C. Liu. Probabilistic safeguard for reinforcement learning using safety index guided gaussian process models. In Learning for Dynamics and Control Conference , pages 783-796. PMLR, 2023.

Y. Zheng, J. Li, D. Yu, Y . Yang, S. E. Li, X. Zhan, and J. Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations , 2024.

Broader Impacts

Our work aims to improve the generalization of reinforcement learning to satisfy arbitrary LTL specifications. This framework enables flexible and expressive task definitions, but it also places responsibility on the user to ensure that the specifications are well designed. The behavior of the agent is tightly coupled with the given specification, so poorly constructed or harmful specifications may lead to unsafe or unintended behaviors. Thus, careful design and verification of specifications are essential when applying such methods in real-world settings.

A Experimental Settings

A.1 Environments

We use the LetterWorld [50] and ZoneEnv [50, 23] for evaluation. The LetterWorld is a 7 × 7 grid world that contains 12 letters corresponding to atomic propositions AP = { a , b , . . . , l } . Each letter appears twice and is randomly placed on the grid. The observation consists of the entire grid in an egocentric view. The agent can move in four directions: up, down, left, and right. The grid wraps around, i.e., if the agent moves out of bounds, it reappears on the opposite side. The maximum episode length is T = 75 . The ZoneEnv contains different colored regions corresponding to atomic propositions AP = { blue , green , magenta , yellow } and walls acting as boundaries. Our implementation is adapted from [23]. We use the point robot that has a continuous action space for acceleration and steering. Observations consist of Lidar measurements of the colored regions and egocentric state information from other sensors. The maximum episode length is T = 1000 . We further construct several variants of the ZoneEnv to evaluate the generalization performance of our method and baselines by modifying: (1) the region size, with r = { 0 . 2 , 0 . 3 , 0 . 4( default ) , 0 . 5 , 0 . 6 } ; (2) the number of regions per atomic proposition, with count = { 1 , 2( default ) , 3 , 4 , 5 } ; (3) and the number of atomic propositions, expanded to include new symbols { red , cyan , orange , purple , teal , lime } . Illustrations of the environments are provided in Figure 8 and trajectories under different LTL specifications are provided in Figure 12, 15, and 16.

We exclude the FlatWorld [23, 45] environment from our evaluation. It contains colored regions similar to ZoneEnv . However, the region layout is fixed, meaning that certain Boolean formulas over the atomic propositions cannot be evaluated since they are never true in this environment. Additionally, the observation space is limited to the agent's ( x, y ) -position. Due to this lack of variability and expressiveness, we exclude FlatWorld from our evaluation.

A.2 LTL Specifications

We evaluate zero-shot generalization across a range of specifications, including complex and nested finite-horizon tasks (Table 1), complex infinite-horizon tasks (Table 1), varying specification complexity with different sequence lengths of reach-only and reach-avoid specifications (Table 4), varying environment complexity in reach-avoid tasks (Table 4), and different numbers of atopic propositions (Table 5). The specifications φ 1 -φ 4 and φ 9 -φ 12 are from [23]. To better evaluate each method's generalization ability and its handling of the trade-off between task progression and safety constraints, we construct more complex specifications, φ 5 -φ 8 and φ 13 -φ 16 , which feature longer temporal sequences and stricter safety requirements, i.e., more atomic propositions appear under negation.

B Implementation Details

We use the official codebases of the baselines: LTL2Action 1 , GCRL-LTL 2 , RAD-embeddings 3 , and DeepLTL 4 . Our implementation is adapted from DeepLTL by incorporating a safety value function and a state-dependent Lagrangian multiplier and removing DeepLTL-specific modules. For DeepLTL, LTL2Action, and our method, we use a fully connected actor network with three hidden layers of sizes [64, 64, 64]. The value function network has two hidden layers of sizes [64, 64]. For our method, both the safety value function and the Lagrangian multiplier network share the same architecture as the value function. GCRL-LTL employs a larger actor-critic architecture with [512, 1024, 256] units due to its use of a graph neural network. For fair comparison, we use the Adam optimizer with a learning rate of 3 e -4 and train all methods for 15M environment interactions. The discount factor is set to γ = 0 . 94 for LetterWorld and γ = 0 . 998 for ZoneEnv . Additional hyperparameter details are provided in the code in the supplementary materials. We train all the methods using a 16-core AMDCPUand an NVIDIA GeForce RTX 4090 GPU, and it takes roughly 3 hours to train our model.

Our method is summarized in Algorithm 1. We begin by constructing a subgoal set that includes all feasible subgoals: we first enumerate each assignment α ∈ 2 AP as a candidate α + . For each such α + , we filter out the remaining assignments that conflict with it. We then enumerate all possible

combinations of the filtered assignments to form A -, resulting in the full set of feasible reach-avoid subgoals ξ = { ( α + , A -) i } i M =1 , from which subgoals are sampled uniformly. The agent receives a reward r = 1 if the subgoal is satisfied and r = 0 otherwise. At each timestep, it receives a safety signal h = -1 if the subgoal is not violated, and h = 1 otherwise. The reward and safety functions are defined as r = ✶ [ L ( s σ ) ∈ α + ] and h = 2 · ✶ [ L ( s σ ) ∈ A -] -1 , respectively. If the current subgoal is satisfied, we sample a new subgoal whose avoid set does not contain the current state, i.e., L ( s t ) / ∈ A -, and whose reach set also excludes the current state, i.e., L ( s t ) / ∈ α + , to prevent the new subgoal from being immediately violated or trivially satisfied. If a subgoal is violated, the episode terminates. We use Generalized Advantage Estimation (GAE) [42] to estimate value functions:

where λ GAE = 0 . 95 is the discount factor of GAE. At test time, given a target LTL specification, we convert it into a Büchi automaton and apply the DFS to enumerate all possible paths to accepting states. For each path, we evaluate the current subgoal using the learned value functions and select the optimal subgoal to execute. This process is applied iteratively until an accepting state is reached.

Algorithm 1 GenZ-LTL

Require: Initial policy π , value and safety value function V r , V h , state-dependent Lagrangian multiplier λ , subgoal set ξ , maximum training iterations K , interactions per iteration N

1: for k = 0 , 1 , 2 , . . . , K do

2: for n = 0 , 1 , 2 , . . . , N do

3: if the current subgoal is not specified or the current subgoal is satisfied/violated then 4: Sample a subgoal σ ∼ Unif( ξ )

5: end if

6: Collect trajectory τ = { s σ t , a t , r t , h t }

7: end for

9: Compute advantage functions A π r , A π h , based on V r and V h using (4) and (5).

8: Compute reward-to-go ˆ R t . = ∑ T i = t γ i r i and cost-to-go ˆ H t . = max t h t for each τ

10: Compute the Lagrangian multiplier λ .

11: Fit value function, safety value function by regression on mean-square error.

12: Update the policy parameters by maximizing (3).

13: Update the multiplier parameters by minimizing (3).

14: end for

B.1 Subgoal-induced observation reduction.

For Lidar measurements in the ZoneEnv , f ( · ) is implemented as the element-wise min to obtain the closest distance along each Lidar beam, such that the resulting s σ For grid maps in LetterWorld , we assign distinct values to each cell associated with σ at location ( i, j ) : f ( s i,j ) = v reach if L ( s i,j ) = α + ; v avoid if L ( s i,j ) ∈ A -; v neutral otherwise. This reduction allows the agent to focus on "reach" and "avoid" rather than specific set of atomic propositions and eliminates the need of encoding subgoals, enabling efficient policy learning and generalization to new atomic propositions that satisfies the conditioned discussed in Section 4.2 as shown later in Appendix C. If multiple observation functions are available, each capable of observing an assignment α ∈ 2 AP , we can first fuse the observations within each observation type based on the current subgoal σ and then concatenate the results to form the final state s σ . In cases where each atomic proposition is tied to a distinct observation type, our method can be applied; however, learning policies in such settings is inherently intractable due to the exponential number of subgoal-state combinations.

The actual implementation of the observation reduction may appear straightforward, as the fusion operator is environment/sensor-specific. However, the idea behind this technique is broadly applicable. When solving the reach-avoid problem, what truly matters is whether a state satisfies the reach or avoid condition under the current subgoal, rather than its specific label L ( s ) as we mentioned in Section 4.2. Moreover, our method can be extended to more realistic settings with some additional effort, but without requiring fundamental changes to the core approach. For example, in autonomous driving scenarios with multiple sensors such as cameras and LiDAR, observation reduction can be enabled through perception modules. Semantic segmentation can be applied to both image and point

cloud data to extract relevant features based on the current reach-avoid subgoals. For instance, if the avoid subgoal involves avoiding other cars and pedestrians, and the reach subgoal involves staying within the current lane, then perception can be used to segment the scene accordingly. This allows the agent to focus on task-relevant information and use processed observations, rather than raw sensory data, to learn value functions and policies effectively.

̸

̸

B.2 Timeout & Subgoal-switching Mechanism

At the test time, given a target LTL specification, we convert it into a Büchi automaton and apply a depth-first search (DFS) to enumerate all possible paths to accepting states. However, not all of these paths, or in some cases, any, are feasible in the physical environment, as some may include subgoals that are unsatisfiable. Although the labeling function is known, meaning we can determine which propositions are true for any given state, it does not necessarily imply that we can exhaustively iterate over the entire state space to identify unsatisfiable assignments.

For example, consider the specification ¬ yellow U (( blue ∧ green ) ∨ magenta ) . To satisfy this formula, the agent must eventually reach either a magenta region or a region where blue and green overlap. However, the latter becomes unsatisfiable if the blue and green regions are disjoint. In such cases, the agent must be able to adapt by pursuing an alternative subgoal, a capability that is essential for goal-conditioned policies, including our approach as well as existing baselines such as DeepLTL and GCRL-LTL. However, these baselines do not address this issue.

To address this, we implement a timeout mechanism that allows the agent to revisit the automaton and search for alternative subgoals when the current one cannot be satisfied. We determine subgoal satisfiability by enforcing a timeout threshold (1 + ϵ scale ) · µ subgoal as discussed in Section 4.1: if the agent fails to achieve this subgoal within that timestep, it is considered unsatisfiable, and the agent switches its subgoal. If no feasible subgoals remain, the episode terminates.

This timeout & subgoal-switching mechanism is explained in Algorithm 2, which outlines the general mode of operation during the test time. We use a set U : Q× Σ to store the transitions that are marked as unsatisfiable after a timeout. These unsatisfiable transitions are ignored by the FindSubgoals function. Apart from this, the only difference between FindSubgoals and the DFS algorithm used in DeepLTL is that we separate the subgoals after identifying the paths with accepting cycles.

C Additional Experiments

Training curves. We first evaluate simple reach-only and reach-avoid specifications with sequence length of n = 3 every specific training interval, and the success and violation rates are shown in Figure 9. We can observe that our method outperforms the baselines in terms of achieving a lower

violation rate (close to zero constraint violation) while maintaining a high success rate. The violation rates of the baselines also decrease over training because task progression and safety constraints are jointly encoded in the LTL specification, so optimizing for satisfaction of the specification implicitly improves safety performance. However, due to the inherent trade-off between these requirements, the baselines struggle to further reduce violation rates. In contrast, by formulating the problem using safe RL framework with reachability constraints, our method achieves stricter constraint satisfaction, thereby improving the overall probability of satisfying the full LTL specifications.

DeepLTL

GCRL-LTL

LTL2Action

GenZ-LTL(ours)

LetterWorld

ZoneEnv

1.0

1.0

0.30

0.8

0.3

0.8

0.25

ns

0.6

Nv0.2

ns

0.6

nv

0.20

0.15

0.4

0.2

0.1

0.4

0.2

0.10

0.05

0.0

0.0

0'0

0.00

0.0

0.2

0.4

0.6

80

1.0

1.2

1.4

1e7

0.0

0.20.4

0.60.81.0

1.2

1.4

1e7

0'0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1e7

0'0

0.2

0.4

0.6

80

1.0

1.2

1.4

1e7

Training steps

Partial observability. To satisfy arbitrary LTL specifications, we leverage the structure of the Büchi automaton to decompose each formula into a sequence of subgoals and handle one subgoal at a time, progressing through the sequence incrementally. While our method could be extended to condition on the full subgoal sequence, we argue that this introduces significant drawbacks that outweigh the potential benefits. Among the baselines, DeepLTL conditions its policy on the full subgoal sequence to achieve non-myopic behavior. Intuitively, this "look-ahead" capability may help the agent satisfy the specification in fewer steps. However, subgoal sequences come with several disadvantages. First, policies trained on full sequences are more likely to encounter OOD issues at test time, as the sequence length can vary across specifications. Although one might limit the policy to observe at most k subgoals, where k is the maximum length seen during training, choosing k is nontrivial. A large k can lead to exponential sample complexity due to the combinatorial explosion of subgoal sequences, e.g., each subgoal is drawn from 2 AP , while a small k limits the look-ahead benefit. Moreover, in practice, the agent may not perceive all relevant subgoals at once, rendering the subgoal sequence ineffective. To illustrate this, we conduct experiments where the agent's observability is restricted to half of the environment. We train both DeepLTL and our method in LetterWorld and ZoneEnv , and then evaluate them using specifications φ 1 -φ 3 and reach-avoid tasks with n = 4 . We report the average success rate η s and violation rate η v . As shown in the left and middle plots of Figure 10, our single-subgoal-at-a-time method outperforms DeepLTL in terms of both the success and violation rates, despite the latter using full subgoal sequences.

ns

nv

No

Partialobservability(LetterWorld)

Partialobservability(ZoneEnv)

Overlappingareas

1.00 -

1.000

1.00 -

0.95

0.975

0.95

七

0.85

0.90

0.950

0.925

0.900

0.90 -

0.85

R

080

0.75

0.875

0.850

0.80 -

0.70

0.825

0.75

0.65

DeepLTL DeepLTL

Ours

Ours

0.800

DeepLTL DeepLTL

Ours

Ours

0.70

DeepLTL

Ours

(partial)

(partial)

(partial)

(partial)

Overlapping areas. We construct a variant of ZoneEnv to introduce a more challenging setting where multiple atomic propositions can be true simultaneously, i.e., different colored regions may overlap in arbitrary configurations. Note that FlatWorld is an overly simplified case where the layout is fixed. In ZoneEnv , overlapping areas can be handled in two ways: (1) by adding separate observations for each assignment involving multiple true propositions, or (2) by using observations only for individual atomic propositions (default setting for ZoneEnv ) and learning policies through interaction with the environment, during which the agent implicitly discovers the relationships among propositions and observations. The first approach does not offer meaningful advantages over the second, as our observation reduction technique can extract the relevant parts of the observation

for any target assignment and fuse them as needed. Thus, we adopt the second approach, which presents a more challenging generalization problem. To accommodate this scenario, we modify DeepLTL's curriculum training to allow multiple atomic propositions to be true at the same time. The positions of both the agent and the regions are randomly sampled. We evaluate both methods using reach-avoid specifications with n = 2 : ϕ 2 : ¬ ( magenta ∨ yellow ) U ( blue ∧ green ) , ϕ 3 : ¬ ( blue ∨ magenta ) U ( yellow ∧ green ) , ϕ 4 : ¬ ( yellow ∨ blue ) U ( green ∧ magenta ) , and ϕ 5 : ¬ ( green ∨ magenta ) U ( yellow ∧ blue ) . Trajectory illustrations are shown in Figure 11. We report the average rates and the results in the rightmost plot in Figure 10 show that our method achieves a higher success rate and a lower violation rate. Our method can better satisfy safety constraints due to the integration of reachability constraints.

4

Number of atomic propositions. We also evaluate performance as the number of atomic propositions increases. In this experiment, we emphasize that the observations corresponding to the additional atomic propositions satisfy the conditions outlined in Section 4.2. It is important to note that when new atomic propositions are observed through modalities not encountered during training - such as those representing the agent's speed or heading - zero-shot adaptation remains an open challenge. The results are presented in Table 6, and example trajectories are shown in Figure 12. Among the evaluated methods, only our method can generalize in a zero-shot manner to new atomic propositions whose observations are produced by identical observation functions under output transformation, as the subgoal-induced observation reduction technique fuses these propositions into "reach" and "avoid", allowing the agent to make decisions without relying on their specific labels L ( s ) .

[AP| = 4

[AP| = 6

[AP| = 8

[AP| = 10

Deterministic v.s. stochastic policies. As discussed in Section 5, we follow the standard practice of using a deterministic policy during evaluation. We also note that DeepLTL is evaluated using a stochastic policy in LetterWorld in its original paper. To provide a comprehensive comparison, we report results for both stochastic and deterministic policies in LetterWorld , as shown in Figure 13. We observe that DeepLTL exhibits inconsistent behavior: while the success rate improves under a stochastic policy, the violation rate also increases. This trade-off is unacceptable in safety-critical applications and indicates that DeepLTL struggles to maintain safety. In contrast, our method delivers consistent performance without compromising safety for task progression, since we explicitly model the violation of subgoals as a reachability constant in a safe RL formulation. In our view, stochastic policies are not suitable for evaluation. For reach-only specifications, they contain no safety constraints; a stochastic policy may satisfy them simply by randomly exploring the environment, thereby overstating performance. For reach-avoid specifications, stochasticity further introduces inconsistency: the mean action may be safe, while a sampled action may lead to a violation, or vice versa. This randomness complicates evaluation and prevents a fair comparison between methods.

ns

nv

no

P1-@4stochastic

@1-p4deterministic

Φs-Ps,stochastic

@s-Φg deterministic

1.00

1.00

1.0

1.0

Rate

0.95

0.95

0.9

0.9

0.90

0.90

0.8

0.8

0.85

0.85

0.80

0.80

0.7

0.7

Ours

Ours

Ours

Ours

DeepLTL

DeepLTL

DeepLTL

DeepLTL

Complex Dynamics. To evaluate the performance of our method in environments with more complex dynamics, we conduct additional experiments using the Ant agent [25], consisting of a torso and four legs connected by hinge joints, in the Zone environment. The state space includes the agent's ego-state (40 dimensions, compared to 12 for the Point agent used in our main experiments) and LiDAR observations of the regions. A visualization of the Ant agent is shown in Figure 14. The action space has 8 dimensions, corresponding to the torques applied to the Ant's joints to coordinate leg movements (compared to 2 dimensions for the Point agent). We train and evaluate our method against DeepLTL, the strongest baseline in our main results. For a fair comparison, all training procedures, training parameters, and neural network sizes were kept the same (note that the last one in particular could result in some performance drop as the state dimension and action dimension are bigger for the ant agent). Both methods are trained until convergence. The evaluation results are shown in Table 7. We can observe that our method consistently outperforms the baseline across all metrics, demonstrating an even larger performance gain in environments with more complex agent dynamics. Note that for the Ant agent, in addition to the LTL specifications, there is a built-in safety constraint that the agent should not fall headfirst, enforced as a hard constraint with immediate episode termination. Although a negative reward is assigned in such cases, as is done when a specification is

violated for DeepLTL, it does not explicitly model safety constraints as our method does through HJ reachability, making it less effective in handling such cases. This impedes the learning of coordinated locomotion and then the satisfaction of LTL specifications, leading to degraded performance.

Zone size = 0.2

Zone size = 0.3

Zone size = 0.4

Zone size = 0.5

Zone size = 0.6

Zone count = 1

Zone count = 2

Zone count =3

Zone count = 4

Zone count = 5

D Visualizations

We provide visualizations for the experiments in Section 5.1 and Section 5.2, including infinitehorizon tasks (Figure 15), increasing environment complexity (Figure 16) and increasing numbers of atomic propositions (Figure 12). For infinite-horizon tasks, our method exhibits a clear recurring pattern that visits the specified regions in the correct order, while the baseline struggles to do so and suffers from safety constraint violations. For different environment complexities, we vary the zone size and zone count per atomic proposition. Importantly, we increase the zone count only for propositions the agent must avoid, since doing so for propositions the agent need to satisfy would simplify the task. Larger zones or more zones per proposition force the agent to navigate around constrained regions to reach the target, and the resulting trajectories demonstrate that our method satisfies the safety constraints. For experiments with an increasing number of atomic propositions, we incrementally introduce new propositions, up to 10 in total, each corresponding to a new colored region with associated lidar observations. This makes zero-shot generalization challenging for baseline methods, as they explicitly encode atomic propositions and cannot generalize to unseen ones without retraining. Even substituting one proposition for another, for example, replacing blue with red , causes these baselines to fail, due to their dependence on fixed encodings. In contrast, our method uses subgoal-induced observation reduction, which removes this dependency when atomic

propositions share the same property. This allows our method to focus on the semantics of the propositions, i.e., "reach" or "avoid", rather than the specific proposition symbols, enabling zero-shot generalization to unseen atomic propositions.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the paper's goals, methods, and contributions, which are consistently supported and developed throughout the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include the limitations in the conclusion section and the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: While the paper does not include theoretical results, it clearly states the necessary assumptions underlying the proposed method.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides sufficient details about the experimental setup, including environment configurations, evaluation metrics, and baselines, to support reproducibility of the main results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided in the supplementary materials, along with instructions sufficient to reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies key details of the experimental setup, including environment settings, training and evaluation protocols, hyperparameters, and optimizer configurations in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports the mean and variance of evaluation metrics across multiple runs to support the statistical reliability of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides details on compute resources, including hardware specifications and runtime, in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and confirm that the research complies with all its guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses potential positive and negative societal impacts of the proposed work in a section provided before the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Our work does not involve the release of high-risk data or models. However, we note that the potential risks stem from how users specify LTL tasks, and we emphasize in the broader impacts that careful specification design is essential to avoid unintended or unsafe behaviors.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All external assets used in this work, including code and environments, are properly cited in the paper. We respect the terms of use and licenses associated with these assets, and license information is provided where applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release new code assets as part of this work. The code is accompanied by documentation that describes usage, dependencies, training details, and environment integration.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve research with human subjects and therefore does not require IRB approval or equivalent review.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were only used for writing and editing purposes and did not contribute to the core methodology or scientific content of the research.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GenZ-LTL, a method enabling RL agents to achieve zero-shot generalization for tasks specified in Linear Temporal Logic formulas. The algorithm decomposes LTL requirements into a series of subgoals derived from Buchi automata, which are then solved sequentially using safe RL. The authors confirm that GenZ-LTL works well in simulated navigation environments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper's core idea of solving LTL tasks from a goal-conditioned RL perspective is novel, and the results align with the expected benefits of goal-conditioned RL: better zero-shot and OOD generalization.\n- Treating avoid subgoals as hard constraints is also technically impressive.\n- Clear and high-quality writing.\n\n## Weaknesses\nThe primary weakness I find is that the experimental validation, while extensive, is limited to programmatic navigation-style environments. These environments are particularly favorable for LTL, because tasks map cleanly onto rule-based constraints (go here, don't go here). However, one cannot do everything in one paper, so I simply recommend the authors acknowledge and address this in an explicit limitations section, separate from conclusion or broader impacts (more below).\n\nQUESTIONS:\n1. Suggestion: a more in-depth explanation of the environments, starting in line 250, would be helpful. What is the goal of the agent in these environments? Figure 7 in the Appendix was helpful.\n2. Line 54 oversells slightly, in that the environments are focused on navigation. \n3. Line 239: Did you ever observe the greedy assumption to be an issue? I suppose if the agent were in an environment where it had to actively plan, then this assumption might be?\n4. Line 180: Moving some detail up from appendix or having an algorithm block in the main text might be helpful for understanding here.\n\nThank you for your time!\n\nLIMITATIONS:\nI think the paper would benefit from an explicit limitations section. Talking points that comes to mind:\n1. The two environments tested both are fairly programmatic, and thus map cleanly onto the LTL specification. How do the authors envision this work generalizing to more realistic settings? \n2. Can one have a perfect LTL specification? We assume here that the agent receives an LTL formula, if I understand correctly, but achieving this formula or description of the environment in the first place seems like a challenge. \n3. How does this method interact with soft constraints or preferences? Do these come up in the intended use case, or can we assume clean alignment with LTL?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a multi-task RL framework for tasks given as LTL specifications, combining goal-conditioned RL and safe RL. Specifically, for a given LTL task, they perform a DFS search to identify accepting cycles on the induced Buchi automaton and extract reach and avoid subgoals; then, if applicable, apply their proposed subgoal-induced observation reduction method to reduce the state size; and finally, they sequentially (and myopically) pass individual reach-avoid problems to a goal conditioned policy. They incorporate Hamilton-Jacobi reachability to their goal-conditioned policy by simultaneously learning a value function estimating the feasible set and using this value function with a state-dependent Lagrangian multiplier to select the subgoal that achieves the best trade-off between the task progression and the safety constraint. To account for the fact that some subgoals might be unsatisfiable, they implement a subgoal switching mechanism that is triggered if the current subgoal is not reached within a timeout threshold; if the current subgoal times out, then they switch to the next best subgoal, selected based on the value functions. During these attempts, they make sure that the current reach-avoid problem does not violate another reach-avoid problem by respecting all avoid subgoals of the current automaton state. An experimental evaluation of the method is presented, comparing the proposed framework against a selected set of previous work and showing its efficacy.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper combines the existing approaches for multi-task RL with LTL objectives and Hamilton-Jacobi-based safe RL in an intuitive and clear manner. Their extensive experimental evaluation demonstrates the benefits of their proposed approach compared to the previous work. Their subgoal-induced observation reduction technique is a novel contribution that stems from a simple observation about the local reach-avoid nature of LTL specifications; however, I found the explanation given in Sec 4.2 a bit too complicated. I wonder if that explanation can be simplified, or the authors might even consider adding a figure to assist the reader with the intuition that they are trying to communicate. Also, an experimental comparison with [54] is needed, as that work outperforms [50]. Furthermore, as also mentioned in the paper, the proposed approach is myopic since the goal-conditioned policy doesn't know what the future subgoals are. So, the authors might consider including an experimental evaluation (at least in the appendix) tailored to show this limitation to demonstrate how the proposed approach behaves compared to [50, 54] and DeepLTL in terms of its myopia. That being said, I think the approach presented in this paper is orthogonal to techniques given in [50, 54], in the sense that one can also include LTL formula embeddings [50] or automaton embeddings [54] in the product state and overcome myopia while still leveraging the benefits of the proposed framework. Overall, this is a well-written paper with good experimental evaluation, and such additional experiments and/or further discussion on these issues could make it a stronger contribution and help better position it in the literature.\n\nQUESTIONS:\n1. Have you tried comparing against [54]? I think this empirical comparison is needed, as [54] is a newer technique that outperforms [50]. Also, do you agree with my comment that LTL formula embeddings [50] and automata embeddings [54] can be incorporated into your technique to overcome the myopia issue? If so, I think the paper can benefit from a discussion highlighting how these different approaches are orthogonal and can be incorporated to get the best of both worlds, i.e., overcome myopia via embeddings and the benefits of your safe RL formulation.\n2. In Figure 6, what were the network sizes and number of training steps for \"Ours (w/o s^\\sigma)\" and \"Ours\"? Were they the same? If so, then do you think that given more compute, \"Ours (w/o s^\\sigma)\" could perform as well as \"Ours\"?\n3. Does the benefit of the state reduction technique stem from the compression it provides through exploiting the symmetries in the observation? Is it possible to get the same performance with more compute when this state reduction technique is not used? I think a discussion or an experiment on this matter might be helpful to further highlight what this state reduction technique provides. Especially given the fact that it is an environment-specific approach that cannot be used in any environment, it would be good to have a further discussion of how to make the overall method work as well in cases where this state reduction technique cannot be used.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GenZ-LTL, a reinforcement learning (RL) framework that enables zero-shot generalization to arbitrary Linear Temporal Logic (LTL) specifications. In its proposed method, the LTL task is first decomposed into a sequence of reach-avoid subgoals using the corresponding Büchi automaton. Instead of conditioning on full subgoal sequences, GenZ-LTL processes subgoals one at a time to better generalize to unseen tasks. Besides, the method treats avoid conditions as hard constraints and incorporates Hamilton-Jacobi reachability to enforce safety.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper proposes to solve the generalization of temporal logic tasks, which is an important topic in many applications.\n\nWeakness:\n1. The major concern of this paper is its novelty. The main idea of this paper, i.e., solving or generalizing LTL tasks by decomposing LTL tasks into reach-avoid sub-tasks, has been proposed and investigated by many previous works [1,2,3,4]. Especially in [2,4], generalizing temporal logic tasks by training goal-conditioned RL policies to solving arbitrary reach-avoid sub-tasks has been proposed. \n\nAuthors say that safety constraints were not modeled in [4]. However, both target and safety constraints of sub-tasks/subgoals are clearly modeled in [2]. The method proposed in [2] also trains a general value function for arbitrary reach-avoid sub-tasks composed of propositions, but authors claim this as their innovation. \n\nIn [3], authors even derive the probability lower-bound for solving reach-avoid sub-tasks. Although [3] didn’t mention generalization, RAMS in [3] can be trivially applied into the generalization of LTL tasks. \n\nIn addition, the subgoal-induced observation reduction just trivially extracts components about propositions to reach and to avoid from LIDAR observations, since every proposition already has an independent component in the LIDAR observation. \n\n2. The writing of this paper needs to be improved a lot. \n\nThe introduction part does not present some key concepts clearly, e.g., subgoal-induced observation reduction, which makes it difficult for people to understand the author's contribution. \n\nIn the method section, a concise algorithm table should be provided, summarizing key steps of the proposed framework. Authors only introduce the algorithm in text and diagram, but these are not enough. Besides, there are a lot of redundant sentences in the method section, making people confused when reading this part. In section 4.2, a diagram should be provided to describe the process of observation reduction.\n\n3. Authors propose to use the constrained policy optimization [5] to solve the reach-avoid sub-task. However, there are many other safe RL approaches to achieve the same goal. Authors did not clearly explain why they chose the method in [5] and not use others. RASM in [3] is another method with probabilistic guarantee. Authors could compare their method with RASM for solving any reach-avoid sub-tasks.\n\n[1] Wang, Yuning, and He Zhu. \"Deductive Synthesis of Reinforcement Learning Agents for Infinite Horizon Tasks.\" 37th International Conference on Computer Aided Verification (CAV), 2025.\n\n[2] Xu, Duo, and Faramarz Fekri. \"Generalization of Compositional Tasks with Logical Specification via Implicit Planning.\" arXiv preprint arXiv:2410.09686 (2024).\n\n[3] Žikelić, Đorđe, et al. \"Compositional policy learning in stochastic control systems with formal guarantees.\" Advances in Neural Information Processing Systems 36 (2023): 47849-47873.\n\n[4] Qiu, Wenjie, Wensen Mao, and He Zhu. \"Instructing goal-conditioned reinforcement learning agents with temporal logic objectives.\" Advances in Neural Information Processing Systems 36 (2023): 39147-39175.\n\n[5] Yu, Dongjie, et al. \"Reachability constrained reinforcement learning.\" International conference on machine learning. PMLR, 2022.\n\nQUESTIONS:\nPlease address the novelty concerns I raised in Weakness section.\t\n\nIn the last paragraph of section 3, it says “q and q' are consecutive states along some accepting run, and q≠q' if q'∉ F”. Why q≠q' if q'∉ F? When the agent is going back and forth between two different accepting states, this statement is not correct.\n\n\tAuthors use reach-avoid subgoals throughout the paper. Is this same as sub-task? If it is about reach-avoid, “sub-task” should be used, not subgoal.\n\n\tI don’t quite understand the observation reduction part. It says the original size of proposition-dependent part is k×2^(|AP|). The lidar observation has an independent component for every proposition. So, is the original size k×|AP|?\n\n\tWhat do you mean by this sentence “For simplicity of deriving a fusion operator that preserves task-relevant semantics such as distances to avoid zones, we assume that the os are coordinate-wise monotonic” in section 4.2?\n\tIn section 5.3, authors evaluate the effects of the observation reduction and safe RL approach. However, training a separate Q and V function for safety constraints can cost a lot of environmental samples. Could you evaluate the increase of environmental samples used to train Q and V functions for safety constraints?\n\nLIMITATIONS:\nAuthors need to include sufficient discussion about the scalability of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GenZ-LTL, a reinforcement learning (RL) method that achieves zero-shot generalization to unseen tasks defined by Linear Temporal Logic (LTL) specifications. The core approach is to decompose LTL goals into reach-avoid subgoals through Büchi automata and subgoal-induced observation reduction, and solve them sequentially using a safe RL framework guided by Hamilton-Jacobi (HJ) reachability constraints. The subgoal-induced observation reduction technique also mitigate exponential scaling with atomic propositions. To cope with this issue when a selected subgoal is unsatisfiable in the environment, there is a timeout-based subgoal-switching mechanism to identify unsatisfiable subgoals and allow the agent to switch to alternative ones. Empirical results across several environments demonstrate generalization, safety, and sample efficiency compared to the baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. First method to combine state-wise safety constraints, HJ reachability, with goal-conditional RL on LTL tasks.\n2. The reach-avoid formulation and observation reduction technique enhance the generalization while keep the training efficiency.\n3. Comprehensive evaluations across finite and infinite-horizon tasks, with strong baselines (DeepLTL, LTL2Action, GCRL-LTL)\n4. The method and experiments are described clearly, with good figures and examples.\n\nWeaknesses:\n1. The method handles only one subgoal at a time, which could be suboptimal in tasks requiring global planning. The authors acknowledge this in their discussion, but it's still a core limitation.\n2. The Assumptions for observation reduction(e.g., decomposability of observations, monotonic sensor fusion) might not generalize to all real-world settings.\n3. It would be beneficial to include additional experimental results on benchmarks that closely resemble real-world scenarios and robot agents, such as robotic arms or quadrupeds in simulated environments in MuJoCo [1].\n\n[1] MuJoCo benchmark: https://github.com/google-deepmind/mujoco\n\nQUESTIONS:\n1. Is there any comparison that highlights the performance of GenZ-LTL versus the baselines on satisfiable LTL specifications that may admit sub-optimal solutions?\n2. How well does GenZ-LTL perform in scenarios that closely resemble real-world environment?\n3. See above in Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}