Title: Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

goal-conditioned reinforcement learning

formal methods

graph embeddings

representation learning
Abstract: Goal-conditioned reinforcement learning is a powerful way to control an AI agent's behavior at runtime. That said, popular goal representations, e.g., target states or natural language, are either limited to Markovian tasks or rely on ambiguous task semantics. We propose representing temporal goals using compositions of deterministic finite automata (cDFAs) and use cDFAs to guide RL agents. cDFAs balance the need for formal temporal semantics with ease of interpretation: if one can understand a flow chart, one can understand a cDFA. On the other hand, cDFAs form a countably infinite concept class with Boolean semantics, and subtle changes to the automaton can result in very different tasks, making them difficult to condition agent behavior on. To address this, we observe that all paths through a DFA correspond to a series of reach-avoid tasks and propose pre-training graph neural network embeddings on "reach-avoid derived" DFAs. Through empirical evaluation, we demonstrate that the proposed pre-training method enables zero-shot generalization to various cDFA task classes and accelerated policy specialization without the myopic suboptimality of hierarchical methods.

Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning

Abstract

Goal-conditioned reinforcement learning is a powerful way to control an AI agent's behavior at runtime. That said, popular goal representations, e.g., target states or natural language, are either limited to Markovian tasks or rely on ambiguous task semantics. We propose representing temporal goals using compositions of deterministic finite automata (cDFAs) and use cDFAs to guide RL agents. cDFAs balance the need for formal temporal semantics with ease of interpretation: if one can understand a flow chart, one can understand a cDFA. On the other hand, cDFAs form a countably infinite concept class with Boolean semantics, and subtle changes to the automaton can result in very different tasks, making them difficult to condition agent behavior on. To address this, we observe that all paths through a DFA correspond to a series of reach-avoid tasks and propose pre-training graph neural network embeddings on 'reach-avoid derived' DFAs. Through empirical evaluation, we demonstrate that the proposed pre-training method enables zero-shot generalization to various cDFA task classes and accelerated policy specialization without the myopic suboptimality of hierarchical methods.

gt

AND

GATv2

Convolutional

Concatenate

0

(st I gt)

90

q2

q2

Policy

91

91

Network

Gotogreen

Gotoorangewhileseeing

while avoiding red

blue even numberof times

cDFA EmbeddingSpace

Niklas Lauffer ∗

Sanjit A. Seshia

1 Introduction

Goal-conditioned reinforcement learning (RL) [33] has proven to be a powerful way to create AI agents whose task (or goal) can be specified (conditioned on) at runtime. In practice, this is done by learning a goal encoder, i.e., a mapping to dense vectors, and passing the encoded goals as inputs to a policy, e.g., a feedforward neural network. This expressive framework enables the development of flexible agents that can be deployed in a priori unknown ways, e.g., visiting states never targeted during training. The rise of large language models has popularized leveraging natural language as an ergonomic means to specify a task, e.g., 'pick up the onions, chop them, and take them to stove.'

While incredibly powerful, tasks specified by target states or natural language have a number of shortcomings. First and foremost, target states are necessarily limited to non-temporal tasks. On the other hand, natural language is, by definition, ambiguous, providing little in the way of formal guarantees or analysis of what task is being asked of the AI agent.

To this end, we consider conditioning on tasks in the form of Boolean combinations of deterministic finite automata (DFAs). We refer to this concept class as compositional DFAs (cDFAs). The choice of cDFAs as the concept class is motivated by three observations. First and foremost, DFAs offer simple and intuitive semantics that require only a cursory familiarity with formal languagesif one can understand a flow chart, one can understand a DFA . Moreover, recent works have demonstrated that DFA and cDFA can be learned in a few shot manner from expert demonstrations and natural language descriptions [41]. As such, DFAs offer a balance between the accessibility of natural language and rigid formal semantics. The addition of Boolean combinations to cDFA, e.g., perform task 1 AND task 2 AND task 3, provides a simple mechanism to build complicated tasks from smaller ones.

Second, DFAs represent temporal tasks that become Markovian by augmenting the state-space with finite memory. Further, they are the 'simplest' family to do so since their finite states are equivalent to having a finite number of sub-tasks, formally Nerode congruences [19]. This is particularly important for goal-conditioned RL which necessarily treats temporal tasks differently than traditional RL. For traditional RL, because the task is fixed, one can simply augment the state space with the corresponding memory to make the task Markovian. In goal-conditioned RL, this is not, in general, possible as it is unclear what history will be important until the task is provided . Instead, the encoded task must relay to the policy this temporal information. Third, existing formulations like temporal logics over finite traces and series of reach-avoid tasks are regular languages and thus are expressible as DFAs [11]. This makes DFAs a natural target for conditioning an RL policy on temporal tasks.

The expressivity of DFAs introduces a number of challenges for goal-conditioned RL. First, DFAs form a countably infinite and exponentially growing concept class where subtle changes in the DFA can result in large changes in an agent's behavior. Notably, this means that any distribution over DFAs is necessarily biased with many 'similar' DFAs having drastically different probability. Thus, to generally work with DFAs, one cannot simply match finite patterns, but need to learn to encode details necessary for planning. Second, as with traditional goal-based objectives, DFAs provide a very sparse binary reward signaldid you reach the accepting state or not? Together with non-trivial dynamics, naïve applications of RL become infeasible due to the lack of dense reward signal. Finally, the exponentially-expanding concept class presents computational limitations for encoding. For example, many interesting DFAs may be too large to be feasibly processed by a graph neural network.

To address these issues of reward sparsity and the need to encode planning, we introduce a distribution of DFAs, called reach-avoid derived (RAD) . This concept class is inspired by the observation that all paths through a DFA correspond to a series of (local) reach-avoid problems. We argue in Section 4 that RAD encourages learning to navigate a DFA's structure. Our first key result is that pre-training DFA encoders on RAD DFAs enables zero-shot generalization to other DFAs. Next, we treat the problem of DFA size. Many problems are naturally expressed compositionally, e.g., a sequence of rules that must all hold or a set of tasks of which at least one must be accomplished. Due to their Boolean semantics, i.e., did you reach the accepting state or not, DFAs offer well-defined semantics under Boolean compositions. Our second key insight is to encode conjunctions 2 of DFAs (called cDFAs). This is done by using a graph attention network [9] (GATv2) to encode the individual DFA graph structure as well as the conjunction (AND) relationships between a collection of tasks. Recalling that the conjunction of any two DFAs grows (worst-case) quadratically, we observe that cDFAs offer an exponential reduction in the size of temporal tasks passed to GATv2.

1.1 Contributions. Our main contributions are: (1) we propose compositional DFAs (cDFAs), which balance formal semantics, accessibility, and expressivity, as a goal representation for temporal tasks in goal-conditioned RL; (2) we propose encoding cDFAs using graph attention networks (GATv2); (3) we propose pre-training the cDFA encoder on reach-avoid derived (RAD) DFAs, and (4) we perform experiments demonstrating strong zero-shot generalization of pre-training on RAD DFAs.

1.2 Related Work. Our work explores a temporal variant of goal-conditioned RL [33] where goals are represented as automata. While traditionally focused on goals as future target states, this has since been extended to tasks such as natural language [7, 21, 26, 37] and temporal logic [38]. While not as expressive as natural language, we believe cDFAs offer a balance between the ergonomics of language while maintaining unambiguous semantics-something of increasing importance due to the seemingly inevitable proliferation of AI agents to safety-critical systems. Moreover, DFA inference has a rich literature [6, 15, 17, 24, 27, 28, 45] with recent works have even shown the ability to learn DFA and cDFA from natural language and expert demonstrations-bridging the gap even more between natural language and automata specified goals [41, 42, 43].

Operating in a similar space, previous work, LTL2Action [38], has shown success with using (finite) linear temporal logic (LTL), a modal extension of propositional logic, to condition RL policies on. In fact, the pre-training and test domains used in this paper are directly derived from that work.

Our choice to focus on DFA rather than LTL is two-fold. First and foremost, over finite traces, LTL is strictly less expressive than DFA. For example, LTL cannot express tasks such as 'the number of times the light switch is toggled should be even.' Second, like DFA, LTL tasks constitute a countably infinite set of tasks. This again means that any distribution over LTL is necessarily biased to certain subclasses. On the one hand, the syntactic structure makes separation of subclasses very easy. On the other, it remains unclear how to generalize to 'common' LTL formula. By contrast, we argue that the local reach-avoid structure of DFAs offers a direct mechanism for generalization. Finally, we note that while LTL is exponentially more succinct than DFA, this is largely mitigated by supporting boolean combinations of DFAs (cDFAs).

Moving away from goal-conditioned RL, recent works have proposed performing symbolic planning on the DFA and cleverly stringing together policies to realize proposed paths [16, 22, 23, 30]. However, their method can still suffer from sub-optimality due to their goal-conditioned value functions being myopic. Specifically, if there are multiple ways to reach the next sub-goal of a temporal task, the optimality of the next action depends on the entire plan, not just the next sub-goaldestroying the compositional structure used for planning. For example, Figure 4 shows a simple example in which hierarchical approaches will find a suboptimal solution due to their myopia. In contrast, conditioning on cDFA embeddings allows our policies to account for the entire task.

On the single-task LTL-constrained policy optimization side, several works have tackled the problem in varying settings [10, 13, 36, 44]. Adjacent to these efforts, various approaches have explored LTL specifications and automaton-like models as objectives in RL [2, 3, 8, 12, 14, 20, 29, 32, 46, 48]. A different line of work considers leveraging quantitative semantics of specifications to shape the reward [1, 22, 25, 36]. However, all of these lines of work are limited to learning a single, fixed goal.

Finally, in a previous work [47], we used hindsight experience replay [5] to solve the reward sparsity problem for DFA-conditioned off-policy learning of DFA task classes. We observe that our RAD pre-training pipeline has similar sample efficiency while generalizing to larger classes of DFAs.

2 Preliminaries

In the next sections, we will develop a formalism for conditioning agent behavior on temporal tasks represented as DFAs. To facilitate this exposition, we quickly review goal-conditioned RL and DFA.

2.1 Goal-Conditioned RL. We start with technical machinery for goal-conditioned RL.

Definition 2.1. A Markov Decision Process 3 (MDP) is a tuple M = ( S, A, T ) , where S is a set of states , A is a set of actions , and T is a map determining the conditional transition probabilities, i.e. Pr( s ′ | s, a ) def = T ( s ′ , a, s ) for s, s ′ ∈ S and a ∈ A . We assume that M contains a sink

̸

state, $ , denoting the end of an episode. Any MDP, M , can be given γ -temporal discounting semantics by asserting that each transition can end the episode with probability 1 -γ . Formally, T ′ ( s ′ , a, s ) = (1 -γ ) · T ( s ′ , a, s ) , for s, s ′ = $ .

Definition 2.2. Let G be a set of states called goals . A goal-conditioned policy 4 , π : S ×G → ∆ A , is a map from state-goal pairs to distributions over actions. Together, an MDP M and a goal-conditioned policy, π , define a distribution of sequences of states, called paths .

A goal-conditioned reinforcement learning problem is a tuple consisting of a (possibly unknown) MDP and a distribution over goals, Pr( g ) for g ∈ G . The objective of goal-conditioned RL is to maximize the probability of generating a path containing g .

2.2 Deterministic Finite Automata. Next, we provide a brief refresher on automata.

Definition 2.3. A Deterministic Finite Automaton (DFA) is a tuple A = ( Q, Σ , δ, q 0 , F ) , where Q is a finite set of states , Σ is a finite alphabet, δ : Q × Σ → Q is the transition function , q 0 ∈ Q is the initial state , and F ⊆ Q are the accepting states . The transition function can be lifted to strings via repeated application, δ ∗ : Q × Σ ∗ → Q . One says A accepts x if δ ∗ ( q 0 , x ) ∈ F . Finally for simplicity, we abuse notation and denote by DFA the set of all DFA.

Visually, we represent DFAs as multi-graphs whose nodes represent states and whose edges are annotated with a symbol that triggers the corresponding transition. Omitted transitions are assumed to stutter , i.e., transition back to the same state. DFAs have a number of interesting properties.

Up to a state isomorphism, all DFAs can be minimized to a canonical form, e.g., using Hopcroft's algorithm [18], denoted by minimize ( A ) . Further, DFAs are closed under Boolean combinations. For example, the conjunction of two DFAs, A 1 and A 2 , denoted A 1 ∧ A 2 , is constructed by taking the product of the states, Q = Q 1 × Q 2 , and evaluating the transition functions element-wise, δ (( q 1 , q 2 ) , σ ) = ( δ 1 ( q 1 , σ ) , δ 2 ( q 2 , σ )) . Also, ( q 1 , q 2 ) ∈ F if q 1 ∈ F 1 and q 2 ∈ F 2 . While DFAs are closed under conjunction, each pair-wise operation results in a O ( n 2 ) increase in the number of states.

Definition 2.4. A compositional DFA (cDFA) is a finite set of DFA, C = {A 1 , . . . , A n } . The monolithic DFA associated to cDFA, C , is given by monolithic ( C ) def = ∧ A∈C A . The semantics of a cDFA are inherited from its monolithic DFA, i.e., a string x is accepted by cDFA C if and only if it is accepted by monolithic ( C ) . Graphically, cDFA are represented as trees of depth 1 where the root corresponds to conjunction and each leaf is itself a DFA.

2.3 Augmenting MDPs with cDFAs. Observe that DFAs can be viewed as deterministic MDPs leading to a natural cascading composition we refer to as DFA-augmention of a MDP. Informally, one imagines that the MDP and DFA transitions are interleaved. First, the MDP transitions given an action. The new state is then mapped to the DFA's alphabet. This symbol then transitions the DFA. The goal is to reach an accepting state in the DFA before the MDP reaches its end-of-episode state.

Formally, let A and M be a DFA and MDP, respectively and let L : S → Σ denote a labelling function mapping states to symbols in the DFAs alphabet. The new MDP is constructed as a quotient of the cascading composition of A and M . First, take the product of the state spaces, S × Q . The action space is M 's action set. The transition of the MDPs state is as before and the DFA transitions are given by: δ ( q, L ( s )) . In order for there to be a unique goal and end-of-episode state, we quotient S × Q as follows: (i) ($ , q ) are treated as a single end of episode state and (ii) for all accepting states, q ∈ F , the product states ( s, q ) are treated as a single accepting (goal) state.

3 Compositional Automata-Conditioned Reinforcement Learning

We are now ready to formally define (compositional) automata-conditioned RL as a variant of goal-conditioned RL where goals are defined in the augmented MDPs.

Definition 3.1. A DFA-conditioned policy , π : S × DFA → ∆ A is a mapping from state-DFA pairs to action distributions. A DFA-conditioned RL problem is a tuple ( M , P ) , where M is an MDP and P is a distribution over DFAs. The objective is to maximize the probability of generating a path containing the accepting state. cDFA-conditioned RL is defined via the underlying monolithic DFA.

q

0

q

2

q

1

q

0

q

2

q

1

AND

g

t

The proposed architecture for conditioning RL policies on temporal tasks encoded as cDFA is given in Figure 1. At step t , given an observation o t and a cDFA C t , we compute an embedding s t of the observation o t using a convolutional neural network and embedding g t of C t (constituting a goal ) using a message-passing neural network (MPNN). We then concatenate s t and g t and feed it to a policy network to get an action a t . After taking a t , we use the next observation o t +1 to update the initial state of each A t ∈ C t based on the transition taken. We then minimize each updated DFA to form the next cDFA C t +1 , which is used to get the next action a t +1 .

3.1 cDFA Featurization. In order to pass a cDFA C t to an MPNN, we construct a featurization ˆ G C t = ( ˆ V C t , ˆ E C t , h C t ) . Details of this process are given in Appendix A. Essentially, we apply four sequential modifications to the graph structure trivially induced by C t : (i) add new nodes for every transition in each DFA in the composition, remove these transitions to connect the new nodes to the source and target nodes of the removed transitions, and in the feature vectors of the new nodes, encode all symbols triggering that transition using positional one-hot encodings, (ii) reverse all the edges, (iii) add self-loops to all nodes, and (iv) add edges from the nodes corresponding to the initial states of DFAs to the 'AND' node. Figure 2 shows the featurization of the cDFA given in Figure 1.

3.2 cDFA Embedding. Given a cDFA featurization ˆ G C t = ( ˆ V C t , ˆ E C t , h C t ) , we construct an embedding of the cDFA using a graph attention network (GATv2) [9] performing a sequence of message passing steps to map each node to a vector. For ease of notation, we refer to the features of a node v ∈ ˆ V C t as h v def = h C t ( v ) . At each message passing step, node features are updated as:

where N ( v ) denotes neighbors of v , W tgt is a linear map, and α vu is the attention score between v and u computed as:

where a is a vector and W src is a linear map. After message passing steps 5 , the feature vector of the cDFA's 'AND' node (which has received messages from all A t ∈ C t ) represents the latent space encoding g t of the temporal task given by C t .

4 Pre-training on Reach-Avoid Derived (RAD) Compositional Automata

In this section, we introduce pre-training on reach-avoid derived (RAD) cDFAs. This pre-training is designed to solve three important problems. First, we wish to avoid simultaneously learning a control policy and embeddings for the cDFAs. Second, we wish to learn a domain-independent cDFA encoder that performs well across cDFA distributions. Third, we wish to have either a domain-specific or a general cDFA-conditioned policy that performs well across different cDFA distributions.

To alleviate the first problem, i.e., decoupling of learning of control and cDFA embeddings, we follow [38] and pre-train our cDFA encoding GNN on a ' dummy ' MDP containing a single nonend-of-episode state. To solve this dummy MDP, we connect a single linear layer to the output of the GNN which maps cDFA embeddings to the symbols in its alphabet. We then take these symbols and

Mutate and minimize

q

0

q

1

q

2

q

3

q

0

q

3

q

1

q

2

q

4

SRA DFA

RAD DFA

take a transition (either a stuttering one or one that changes the state of the DFA) in each DFA of the composition. The episode ends either when all DFAs end up in their corresponding accepting states (reward 1 ) or when one of the DFAs in the composition ends up in a non-accepting sink state (reward -1 ). The cDFA-augmentation then only models the dynamics of the cDFA being conditioned on. The resulting cDFA-conditioned RL problem can thus focus on only learning to encode the cDFAs.

4.1 Sequential reach-avoid as the basis for planning. What remains then is to support generalization to other cDFA distributions. In particular, recall that cDFAs (and DFAs) lack a canonical 'unbiased' distribution making it a priori difficult to pre-select a cDFA distribution. To begin, observe that realizing any individual state transition q → q ′ through a DFA corresponds to a reach-avoid task: (i) eventually transition to state q ′ and (ii) avoid symbols that transition to a state different than q and q ′ . Further, a path through a DFA, q 1 , . . . , q n , corresponds to a sequence of reach-avoid (SRA) tasks. Notably, as illustrated in Figure 3, SRA tasks can be represented as DFA. Thus, in order to generally reason about satisfying a DFA, our neural encoding must be able to solve SRA tasks.

4.2 Reach-avoid derived DFAs. For general DFAs, there can be arbitrarily many (often infinitely many) paths leading to an accepting state, resulting in multiple SRA tasks interacting and interleaving. Thus, the overall structure of a DFA might be much richer than an SRA task even though it consists of local reach-avoid problems. Building on the observation that paths of a DFA correspond to SRA tasks, we define reach-avoid derived DFAs as a generalization of SRA.

Definition 4.1. We define a mutation of a DFA A to be the process of randomly changing a transition of A , removing the outgoing edges of the accepting state, and then minimizing. To change a transition, we uniformly sample a state-symbol-state triplet ( q, σ, q ′ ) and define δ ( q, σ ) def = q ′ . A DFA A is said to be m -reach-avoid derived ( m -RAD) if it can be constructed by mutating an SRA DFA m -times.

We define the ( m,p ) -RADdistribution over DFAs as follows: (i) sample a SRA DFA, A , with k +2 states where k ∼ Geometric ( p ) ; each transition stutters with probability 0.9 and is otherwise made a reach or avoid transition with equal probability; (ii) sequentially apply m mutations where mutations leading to trivial one-state DFAs are rejected. To avoid notational clutter, we shall often suppress m,p and say RAD DFAs. Finally, we define the RAD cDFA distribution by sampling n RAD DFAs, where n is also drawn from a geometric distribution. See Appendix B for the pseudocode. Figure 3 demonstrates how mutations and minimization can turn an SRA DFA into a RAD DFA. To get the RAD DFA, we apply two mutations to the SRA DFA, adding two new transitions: (i) from q 0 to q 3 on green and (ii) from q 4 to q 1 on blue, and then the minimization collapses q 0 and q 4 to a single state.

5 Experiments

We explore the following 6 questions on several task classes in discrete and continuous environments:

RQ1 Do our policies overcome the limitations of hierarchical approaches?

RQ2 Does pre-training of the cDFA encoder on RAD cDFAs accelerate cDFA-conditioned RL?

RQ3 Does freezing the cDFA encoder negatively impact cDFA-conditioned RL performance?

RQ4 Do the cDFA-embeddings pre-trained on RAD generalize to other task classes?

RQ5 How does the RAD pre-trained cDFA encoder represent the corresponding cDFA?

RQ6 Do RAD pre-trained cDFA-conditioned policies generalize across task classes?

0.8

Discounted return

0.6

0.4

Hierarchicallimit

Optimal

0.2

RAD

0.0

0

10

20

30

40

50

60°

Frames (thousands)

91

I3

Letterworld Environment. Introduced in [4, 38], Letterworld is a 7x7 grid where the agent occupies one square at a time. Some squares have randomly assigned letters which cDFA tasks are defined over. The agent can move in the four cardinal directions to adjacent squares. Moving into a square with a letter satisfies the corresponding symbol. The agent observes its position and the letters in each grid. Each layout has 12 unique letters (each appearing twice) and a horizon of 75.

Zones environment. Introduced in [38] (building on Safety Gym [31]), Zones is as an extension to Letterworld with continuous state and action spaces. There are 8 colored (2 of each color) circles randomly positioned in the environment. cDFA tasks are defined over colors. The agent's action space allows it to steer left and right and accelerate forward and backward while observing lidar, accelerometer, and velocimeter data. The environment has a horizon of 1000.

Task classes. In addition to RAD, we consider the following task classes in our experiments.

Reach (R): Defines a total order over k ∼ Uniform (1 , K ) symbols, e.g., q 0 and q 1 in Figure 1.

Reach-Avoid (RA): Defines an SRA task of length k ∼ Uniform (1 , K ) , e.g., first DFA in Figure 1.

Reach-Avoid with Redemption (RAR): Similar to RA, but they have transitions from their avoid state to the previous state, e.g., the sub-DFA with states q 0 , q 1 , q 3 in Figure 3.

Parity: Asub-class of RAR where the symbol that leads to the avoid state also triggers a transition back to the previous state, e.g., the DFA on the right in Figure 1. These tasks are not expressible in LTL due to their languages not being star-free regular [40].

Given a bound N , using these task classes, we generate their compositional counterparts by sampling n ∼ Uniform (1 , N ) many DFAs to form a cDFA. As a short-hand notation, for example, we refer to RAR cDFAs with a maximum number of conjunctions 3 where each DFA has a task length of also 2 as cRAR.1.2.1.3 and refer to its associated monolithic (see Definition 2.4) class as RAR.1.2.1.3, i.e., without the 'c' prefix. We use the same abbreviation convention for other task classes as well. We use cRAD for RAD cDFAs with truncated distributions and NT-cRAD for the not-truncated version.

Finally, for comparisons with [38], we include the task classes defined in that work, i.e., partially ordered (PO) and avoidance (AV) finite LTL tasks, represented as cDFAs.

Pre-training procedure. We pre-train GATv2 on RAD cDFAs in the dummy MDP as described in Section 4. We set both geometric distribution parameters of the RAD cDFA sampler to 0 . 5 and truncate these distributions so that n ≤ 5 and 2 + k ≤ 10 , i.e., we make sure that the maximum number of DFAs in a composition is 5 , and the maximum number of states of each DFA is 10 . As a baseline for GATv2, we also pre-train a relational graph convolutional network (RGCN) [34].

Training procedure. We experiment with training policies three ways: Training without pretraining ( no pretraining ), with frozen MPNNs pre-trained on RAD as described in Section 4 ( pretraining (frozen) ), and allowing the pre-trained MPNN to continue receiving gradient updates ( pretraining ).

We use proximal policy optimization (PPO) [35] for all reinforcement learning setups, giving a reward of +1 when a rollout reaches the accepting state of the cDFA task, -1 when a rollout reaches

Discounted return

Letterworld

Zones

0.8

0.8

GATv2nopretraining

0.6

0.6

GATv2pretraining

0.4

0.4

GATv2pretraining (frozen)

RGCNnopretraining

0.2

0.2

RGCN pretraining

0.0

0.0

RGCN pretraining (frozen)

0

5

10

15

20

0

5

10

15

20

Frames

(millions)

Frames

(millions)

a rejecting sink state of the cDFA task, and 0 if the environment reaches the timeout horizon without achieving the cDFA task. See Appendix D for the hyperparameters and compute used.

RQ1 cDFA-conditioned policies do not suffer from myopia. First, we present the experiment results highlighting the myopia of hierarchical approaches. To this end, we construct a simplified variation of the Letterworld environment, given in Figure 4, where the task is to visit the orange square before the green one. Observe that any hierarchical approach, like [23, 30], based on high-level planning over some automaton-like structure suffers from myopia. Specifically, such approaches do not account for the overall task while planning for the intermediate goals. In this example, hierarchical approaches fail to differentiate between the two orange squares in the context of the given task. On the other hand, our approach quickly converges to the optimal policy as given in Figure 4.

RQ2,RQ3: Pretraining helps policies perform better faster. Figure 5 shows training curves for learning on RAD tasks using RGCN (dashed curves) and GATv2 (solid curves) as the MPNN module in Letterworld. Without pre-training in the dummy environment (blue curves), the policies barely learn how to accomplish the tasks, likely because of the difficulty in trying to simultaneously learn an embedding of the cDFA task and learn the dynamics of the environment required to reach certain symbols. Frozen pre-training (green curve) helps tremendously in learning the RAD task class in the Letterworld regardless of the MPNN module architecture. The policies learned are able to satisfy sampled RAD tasks on average over 95% of the time and within 5 timesteps. Non-frozen pre-training (red curve) also helps the policies learn more quickly, but with a significant difference between GATv2 and RGCN. Notably, non-frozen pre-training performs significantly worse for RGCN than its frozen counterpart, showing that freezing the MPNN module after pre-training actually helps performance. We further note that the wall clock training time was significantly faster due to not propagating gradients through the MPNN (about 20% faster).

RQ4: Pretrained cDFA embeddings generalize well. We tested pre-trained GATv2 and RGCN models on various new cDFAs and their associated monolithic task distributions in the dummy DFA environment. Figure 10 in Appendix C.2 presents experiment results averaged over 10 seeds and 50 episodes per seed. The left y-axis illustrates the satisfaction likelihood while the right y-axis presents the number of steps needed to complete the task. The figure shows that GATv2 achieves approximately 90% accuracy across all tasks, with nearly 100% accuracy in most cases. Although RGCN generalizes comparably to GATv2 in most instances, it achieves around 80% accuracy for both cRA.1.5.1.5 and cRA.1.1.1.10 whereas GATv2 performs at nearly 90% and 100%, respectively.

Considering that the maximum number of DFAs in a composition seen during training is 5 (sampled with low probability) and the DFA with the maximum task length is 9 (with a maximum of 10 states, sampled with even lower probability), these generalization results are quite remarkable. GATv2 can generalize to 10 DFAs in composition as well as DFAs with a task length of 10 with nearly 100% accuracy. Furthermore, Figure 10 shows that the embeddings produced by GATv2 lead to shorter episodes compared to RGCN, indicating that GATv2 learns better representations for cDFAs. Overall, regardless of the MPNN model used, pre-training on the RAD cDFAs provides generalization.

Reach-AvoidDerived

Reach-AvoidwithRedemption

2-ConjunctionCollapse

Reach-Avoid

Parity

■ 1-StepAdvance

Reach

No Op

1-StepAdvanceLeadingAccept

0.6

0.7

0.8

0.9

1.0

0.00.51.01.52.02.53.0

(a) GATv2-2-D t-SNE

RQ5: cDFA embedding space is well-clustered. Our GATv2 cDFA encoders learn 32-dimensional embeddings, and, as shown in the previous answer, generalize well to other cDFA classes. To study the cDFA embeddings, we sample cDFAs from RAD, RA, RAR, reach, and parity task classes. For each class, we also introduce two variants: 2-conjunction collapse and 1-step advance . The former randomly selects two DFAs of a cDFA and collapses them down to a single DFA, and the latter takes a random non-stuttering transition on the cDFA. We study the embedding space in two different ways. One projects the embeddings down to 2D using t-SNE [39]. The other computes pairwise cosine similarities and Euclidean distances of the embeddings. See Appendix C.3 for the RGCN results.

The results are given in Figure 6. We first observe that modifications like 2-conjunction collapse and 1step advance result in similar embeddings, showcasing the robustness provided by pre-training GATv2 on RAD. Moreover, due to the 1-step advance operation, some cDFAs end up in their accepting states, and we see that accepting cDFAs are well-isolated inside the green region in Figure 6a. Similarly, we observe that parity samples are mapped very close to RAR samples (its parent class).

The same figure also demonstrates that GATv2 successfully separates different cDFA classes into distinct clusters. This separation is further confirmed by the cosine similarity (c.f. Figure 6b) and Euclidean distance heat maps (c.f. Figure 6c). On the other hand, both heat maps also show that the RAD cDFA class is very rich in terms of various DFAs it contains since it has both similar and different samples compared to other task classes. Notably, we see that the vast majority of RAD cDFA we sample are out-of-distribution for the other classes implying robust generalization.

RQ6: Policies Trained on RAD generalize well. Figure 7 (blue) shows how well the policies trained on RAD generalize to other task distributions. Both frozen and non-frozen (see Appendix C.4) pre-trained policies generalize well across the board with a slight edge to frozen pre-training. Notably, the policies suffers almost no performance loss on non-truncated RAD tasks even though it was only trained on the truncated distribution. The policy sees the largest dip in satisfaction likelihood on

Satisfaction

1.0

0.5

RAD

CRA.1.1.1.5

CRAR.1.2.1.2

0.0

3

2

√

2

3

3

CRAD

2

3

3

RA.1.2.1.2

RA.1.3.

RAR.1.2.

RAR.1.3.

P.1.3.

R.1.3.1.3

CRA.

CRA

CRA.

CRA.

CRAR.

CRAR.

CRAR.

CRAR.1.

CRAR.1.

10

8

CR.

CR

CR.

1.0

Satisfaction

0.5

RADPretraining

LTL

PartiallyOrdered

LTLAvoidance

0.0

cPO.1.5.1.4

cPO.1.6.1.5

cPO.1.8.1.7

CPO.1.10.1.1

CAV.1.3.1.2

CAV.1.2.1.3

longer compositional reach avoid (cRA) tasks. Interestingly, even very long cRA tasks of up to 10 symbols can be satisfied above 90% of the time, even though such long tasks put the satisfying state in cDFA past the depth that the message passing in our MPNNs reach. Overall, the performance of the non-pre-trained models (see Appendix C.4) follows a similar (albeit dramatically worse) trend as the pre-trained models. They do see some success in the easier task classes such as reach and parity but perform poorly in more difficult task classes that contain rejecting sink states. Our generalization experiments comparing GATv2 to RGCN show that even though RGCN performs comparably, GATv2 outperforms it in all cases (see Appendix C.6).

Figure 7 shows that RAD pre-training allows policies trained in Letterworld to zero-shot generalize far out of distribution. If you take a RAD pre-trained cDFA encoder and train a policy on some small subclass (cRA.1.1.1.5 and cRAR.1.2.1.2), the policies can still handle much longer tasks within their class and even tasks outside of their class. In the case of cRA.1.1.1.5, the training of the policy module never gets to see a multistage task, only multiple conjunctions, yet it's able to satisfy tasks from the full RAD distribution 90% of the time.

We also compare against policies trained using LTL syntax embeddings [38]. Figure 21 in Appendix C.8 shows training curves for these policies where pre-training was performed similar to RAD pre-training except only on the specialized class that randomly samples between PO.1.5.1.4 and AV.1.3.1.2 (task classes from [38]). Figure 8 shows how well these policies and our RAD pre-trained policy generalize in the Letterworld to harder version of PO and A V . The results show that our method generalizes well across all tasks. Furthermore, in some cases, it outperforms the LTL policy on the task it is specifically trained on, e.g., even though the LTL policy shown by the green bar is trained on cAV.1.3.1.2, our method outperforms it. Similarly, observe that although our method was not specifically trained on cPO.1.5.1.4 tasks, it achieves 100% accuracy just like the LTL policy that is specifically trained on this task class. In addition to all of the earlier generalization results, this figure shows that RAD pre-training generalizes to these out of distribution tasks just as well as the policies from [38] even though it is not specifically trained on these tasks.

6 Conclusion

We introduced compositional deterministic finite automata (cDFA) as an expressive and robust goal representation for temporal tasks in goal-conditioned reinforcement learning (RL). Addressing the limitations of hierarchical approaches and the challenge of extending goal-conditioned RL to non-Markovian tasks, we proposed using cDFAs to leverage formal semantics, accessibility, and expressivity. By employing graph attention networks (GATv2) to construct embeddings for cDFA tasks and conditioning the RL policy on these representations, our method encourages the policy to take the entire temporal task into account during decision-making. Additionally, we introduced reach-avoid derived (RAD) cDFAs to enable the learning of rich task embeddings and policies that generalize to unseen task classes. Our experiments show that pre-training on RAD cDFAs provides strong zero-shot generalization and robustness. Using cDFAs to encode temporal tasks enhances explainability and enables automated reasoning for interpretable and safe AI systems.

6.1 Limitations. A limitation of this work is the assumption of a labeling function mapping MDP states to the task alphabet. Also, further research is needed to understand how cDFA-encoders learn to represent tasks and how much further their generalization capabilities can be pushed.

Acknowledgments and Disclosure of Funding

This work is partially supported by DARPA contracts FA8750-18-C-0101 (AA) and FA8750-23-C0080 (ANSR), by Nissan and Toyota under the iCyPhy Center, and by C3DTI. Niklas Lauffer is supported by an NSF graduate research fellowship.

References

Derya Aksaray, Austin Jones, Zhaodan Kong, Mac Schwager, and Calin Belta. Q-learning for robust satisfaction of signal temporal logic specifications. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 6565-6570. IEEE, 2016.

Rajeev Alur, Suguman Bansal, Osbert Bastani, and Kishor Jothimurugan. A framework for transforming specifications in reinforcement learning. In Principles of Systems Design: Essays Dedicated to Thomas A. Henzinger on the Occasion of His 60th Birthday , pages 604-624. Springer, 2022.

Rajeev Alur, Osbert Bastani, Kishor Jothimurugan, Mateo Perez, Fabio Somenzi, and Ashutosh Trivedi. Policy synthesis and reinforcement learning for discounted ltl. In International Conference on Computer Aided Verification , pages 415-435. Springer, 2023.

Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International conference on machine learning , pages 166-175. PMLR, 2017.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Dana Angluin. Learning regular sets from queries and counterexamples. Inf. Comput. , 75(2):87106, 1987.

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946 , 2018.

Alper Kamil Bozkurt, Yu Wang, Michael M Zavlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 10349-10355. IEEE, 2020.

Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491 , 2021.

Mingyu Cai, Shaoping Xiao, Zhijun Li, and Zhen Kan. Optimal probabilistic motion planning with potential infeasible ltl constraints. IEEE transactions on automatic control , 68(1):301-316, 2021.

Alberto Camacho, Meghyn Bienvenu, and Sheila A McIlraith. Finite ltl synthesis with environment assumptions and quality measures. In Sixteenth International Conference on Principles of Knowledge Representation and Reasoning , 2018.

Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Anthony Valenzano, and Sheila A McIlraith. Ltl and beyond: Formal languages for reward function specification in reinforcement learning. In IJCAI , volume 19, pages 6065-6073, 2019.

Xuchu Ding, Stephen L Smith, Calin Belta, and Daniela Rus. Optimal control of markov decision processes with linear temporal logic constraints. IEEE Transactions on Automatic Control , 59(5):1244-1257, 2014.

Jie Fu and Ufuk Topcu. Probably approximately correct mdp learning and control with temporal logic constraints. arXiv preprint arXiv:1404.7073 , 2014.

E Mark Gold. Language identification in the limit. Information and control , 10(5):447-474, 1967.

Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Deep reinforcement learning with temporal logics. In Formal Modeling and Analysis of Timed Systems: 18th International Conference, FORMATS 2020, Vienna, Austria, September 1-3, 2020, Proceedings 18 , pages 1-22. Springer, 2020.

Marijn Heule and Sicco Verwer. Exact DFA identification using SAT solvers. In ICGI , volume 6339 of Lecture Notes in Computer Science , pages 66-79. Springer, 2010.

John Hopcroft. An n log n algorithm for minimizing states in a finite automaton. In Theory of machines and computations , pages 189-196. Elsevier, 1971.

John E. Hopcroft and Jeffrey D. Ullman. Introduction to Automata Theory, Languages and Computation . Addison-Wesley, 1979.

Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A McIlraith. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research , 73:173-208, 2022.

Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. Advances in Neural Information Processing Systems , 32, 2019.

Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, and Rajeev Alur. Compositional reinforcement learning from logical specifications. Advances in Neural Information Processing Systems , 34:10026-10039, 2021.

Niklas Lauffer, Beyazit Yalcinkaya, Marcell Vazquez-Chanlatte, Ameesh Shah, and Sanjit A Seshia. Learning deterministic finite automata decompositions from examples and demonstrations. In Conference on Formal MEthods in Computer-Aided Design-FMCAD 2022 , page 325, 2022.

Xiao Li, Cristian-Ioan Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3834-3839. IEEE, 2017.

Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926 , 2019.

José Oncina and Pedro Garcia. Inferring regular languages in polynomial updated time. In Pattern recognition and image analysis: selected papers from the IVth Spanish Symposium , pages 49-61. World Scientific, 1992.

Rajesh Parekh and Vasant Honavar. Learning dfa from simple examples. Machine Learning , 44:9-35, 2001.

Mateo Perez, Fabio Somenzi, and Ashutosh Trivedi. A pac learning algorithm for ltl and omegaregular objectives in mdps. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 21510-21517, 2024.

Wenjie Qiu, Wensen Mao, and He Zhu. Instructing goal-conditioned reinforcement learning agents with temporal logic objectives. Advances in Neural Information Processing Systems , 36, 2024.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

Dorsa Sadigh, Eric S Kim, Samuel Coogan, S Shankar Sastry, and Sanjit A Seshia. A learning based approach to control synthesis of markov decision processes for linear temporal logic specifications. In 53rd IEEE Conference on Decision and Control , pages 1091-1096. IEEE, 2014.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning , pages 1312-1320. PMLR, 2015.

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, proceedings 15 , pages 593-607. Springer, 2018.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Ameesh Shah, Cameron Voloshin, Chenxi Yang, Abhinav Verma, Swarat Chaudhuri, and Sanjit A Seshia. Deep policy optimization with temporal logic constraints. arXiv preprint arXiv:2404.11578 , 2024.

Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems , 36, 2024.

Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcilraith. Ltl2action: Generalizing ltl instructions for multi-task rl. In International Conference on Machine Learning , pages 10497-10508. PMLR, 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Moshe Y Vardi. From church and prior to psl. In 25 Years of Model Checking: History, Achievements, Perspectives , pages 150-171. Springer, 2008.

Marcell Vazquez-Chanlatte, Karim Elmaaroufi, Stefan J Witwicki, and Sanjit A Seshia. L ∗ lm: Learning automata from examples using natural language oracles. arXiv preprint arXiv:2402.07051 , 2024.

Marcell Vazquez-Chanlatte, Ameesh Shah, Gil Lederman, and Sanjit A Seshia. Demonstration informed specification search. CoRR, vol. abs/2112.10807 , 2021.

Marcell Jose Vazquez-Chanlatte. Specifications from Demonstrations: Learning, Teaching, and Control . University of California, Berkeley, 2022.

Cameron Voloshin, Hoang Le, Swarat Chaudhuri, and Yisong Yue. Policy optimization with linear temporal logic constraints. Advances in Neural Information Processing Systems , 35:1769017702, 2022.

Wojciech Wieczorek. Grammatical Inference . Springer, 2017.

Eric M Wolff, Ufuk Topcu, and Richard M Murray. Robust control of uncertain markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference on decision and control (CDC) , pages 3372-3379. IEEE, 2012.

Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, and Sanjit Seshia. Automata conditioned reinforcement learning with experience replay. In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning , 2023.

Cambridge Yang, Michael Littman, and Michael Carbin. On the (in) tractability of reinforcement learning for ltl objectives. arXiv preprint arXiv:2111.12679 , 2021.

A Details of cDFA Featurization

To construct the featurization of a cDFA C t , we first build individual graph featurizations for each DFA in the composition and then combine them in a single featurization for C t . For each A t ∈ C t , we construct a directed graph G A t = ( V A t , E A t , P A t ) , where v i ∈ V A t represents the states of A t , e vu = ( v, u ) ∈ E A t represents the transitions of A t , and P A t : E A t ↦→ 2 Σ t maps each edge e vu to a set of symbols in A t 's alphabet, denoting different symbols triggering a transition from v to u . Using the graph representation G A t , we construct the DFA featurization ˆ G A t = ( ˆ V A t , ˆ E A t , h A t ) . The sketch of the iterative construction of ˆ G A t is given in the following paragraph.

Initially, ˆ G A t has all the nodes of G A t , and it does not have any edges. For each e vu = ( v, u ) ∈ E A t , add a new node e vu to ˆ V A t and two new edges to ˆ E A t - one from v to the new node e vu and another one from e vu to u . Then, add self-loop edges for each node. Once ˆ G A t is constructed this way, we reverse its edges to ensure that message passing transmits information to the initial state of the DFA. To include the details of individual nodes, each v ∈ ˆ V A t is associated with an input node feature h A t ( v ) . For v ∈ V A t , h A t ( v ) indicates whether v is the initial, an accepting, or a rejecting state. For each e vu ∈ ˆ V A t \ V A t , h A t ( e vu ) has one-hot positional encodings of each symbol in P A t ( e vu ) .

Once all featurizations are constructed for each A t ∈ C t , we construct the composition's featurization ˆ G C t = ( ˆ V C t , ˆ E C t , h C t ) . It is constructed by taking the union of all ˆ G A t for each A t ∈ C t , adding an 'AND' node v AND to the graph, and including new edges from the nodes corresponding to the initial states of all A t ∈ C t to v AND. Notice that taking this union is trivial since all nodes and edges of a DFA in the composition are unique. Through one-hot encoding of a unique index in the feature vector h C t ( v AND ) of v AND, the 'AND' node is encoded as a different kind of node that does not share any features with other nodes in the graph, making it distinguished compared to nodes representing DFA states and transitions. Figure 2 shows the featurization of the cDFA in Figure 1.

B RAD cDFA Sampler Algorithm

Algorithm 1 shows the pseudocode for RAD cDFA sampling.

In the experiments, we bounded the size of sampled cDFAs using truncated geometric distributions for n and k , with upper bounds of 5 and 10 , respectively, and k was fixed per rollout environment. Using multiple environments with different sampler instantiations, combined with mutations applied to the sampled DFAs, allowed policies to encounter a diverse range of DFAs of varying sizes during training.

C Results

C.1 Pretraining learning curves: GATv2 vs RGCN. Figure 9 presents the learning curves for pre-training of GATv2 and RGCN on RAD cDFAs in the dummy MDP. The figure illustrates that GATv2 converges faster than RGCN, potentially highlighting the benefits of the attention mechanism.

GAT

RGCN

1.0

1.0

return

0.8

0.8

0.6

re

0.6

Discounted :

0.4

d

0.4

t

scoun

0.2

0.2

0.0

S

0.0

-0.2

-0.2

0

2

4

6

8

10

0

2

4

6

8

10

Frames (millions)

Frames (millions)

Pretraining

C.2 Generalization of pre-trained cDFA embeddings: GATv2 vs RGCN. Figure 10 presents the generalization experiment results in the dummy MDP. It shows that pre-trained GATv2 consistently outperforms RGCN with almost 100% accuracy in most cases. It also learns to solve the dummy MDP with less steps compared to RGCN. Overall, we observe that pre-training on RAD cDFAs provides good generalization to other cDFA task classes for both GATv2 and RGCN.

.10

1.0

25

Satisfaction

Steps

0.5

GATV2

RGCN

0.0

0

CRAD

RA.1.3.1.3

RAR.1.3.1.3

P1.3.1.3

R.1.3.1.3

NT-CRAD

CRA.

CRA.

CRAR.

CRAR.

CRAR.

CP.1.3.1.3

CR.

CR.

CR.1.1.1.10

C.3 Embedding space analysis: GATv2 vs RGCN. Figure 11 presents the embedding space analysis results for both GATv2 and RGCN. Results show that RGCN demonstrates competence in effectively separating different classes, albeit with certain limitations. While it succeeds in distinguishing reach from other classes, it struggles to differentiate RA tasks from parity and RAR. Even though parity and RAR cDFAs share structural similarities with RA cDFAs, this lack of separation is concerning as there is a semantic difference between these task classes-RA cDFAs have rejecting states but parity and RAR cDFAs do not. However, it still successfully clusters cDFAs that are 2-conjunction collapsed and 1-step advanced while also isolating accepting ones in the green region shown in Figure 11d, illustrating the robustness provided by pre-training on RAD cDFAs.

Another point to note is the scales of the Euclidean distance heat maps given in Figure 11c and Figure 11f. The distance scale for RGCN is from 0 to 1.5 whereas the scale for GATv2 is from 0 to 3. Moreover, we see darker colors in Figure 11c, suggesting that GATv2 captures semantic nuances by effectively leveraging the embedding space for robust representations of cDFAs. The analysis results suggest that GATv2 learns a richer representation of the cDFA embedding space, likely due to its attention mechanism, which enables it to attend to relevant features and relationships within the data.

Reach-AvoidDerived

Reach-Avoid with Redemption

x 2-Conjunction Collapse

Reach-Avoid

Parity

■ 1-StepAdvance

Reach

No Op

1-StepAdvance Leading Accept

0.6

0.7

0.8

0.9

1.0

0.00.51.01.52.02.53.0

(a)GATv2-2-Dt-SNE

0.6

0.7

0.8

0.9

1.0

0.0

0.5

1.0

1.5

(d)RGCN-2-Dt-SNE

(e) RGCN - Cosine similarity heat map

(f) RGCN -Euclidean distance heat map

C.4 Generalization of policies: pre-trained vs not pre-trained. Figure 12 compares the generalization strengths of policies with not-pre-trained, pre-trained-and-frozen, and pre-trained GATv2 cDFA encoders. It shows a similar pattern to the learning curves given in Figure 5. Specifically, we see that the frozen one outperforms others while the pre-trained one performs comparably. However, we see that not-pre-trained one does not come close to the others, indicating that learning embedding spaces through pre-training is essential for good performance and generalization to different task classes. Figure 13 presents the number of steps taken in Letterworld to solve the task for the same experiment. We see a similar trend where without pre-training model needs to take a lot of steps to end the episode (either successfully or unsuccessfully). Both pre-trained models take significantly fewer steps while the frozen one solves the task fastest. Combining both figures, we see that the frozen version generalizes well while also learning to solve tasks faster.

Satisfaction

frozen

pretrain

nopretrain

1.0

0.5

0.0

Q

Q

8

R

R

of Steps

60

frozen

pretrain

nopretrain

40

Number

20

0

Q

Q

Q

8

8

8

C.5 Training curves and steps generalization for other task classes. We first present the training curves for RA.1.1.1.5 and RAR.1.2.1.2 using RAD pre-trained cDFA encoders. Figure 14 shows that policies trained on RAD and RAR.1.2.1.2 converge relatively faster than the one trained on RA.1.1.1.5. Notice that RA.1.1.1.5 samples cDFAs with more than 3 DFAs with high probability (since the number of conjunctions sampled uniformly at random from the given interval) whereas the probability of sampling such big compositions is lower in RAD (as it sampled from a geometric distribution). Thus, the figure indicates that it is relatively harder to learn big conjunctions.

In Figure 7, we shared the satisfaction generalization results for these policies, testing how well these policies trained on specialized task distributions generalize to other task distributions. Here, we provide the number of steps taken to finish the episode during the generalization experiments in Figure 15. Observe that training the policy on RAD cDFAs results in shorter episodes. However, it is still worth noting that other policies perform comparably and provide good generalization on unseen tasks.

1.0

Discounted return

0.8

0.6

0.4

0.2

0.0

-0.2

0

2

5

7

10

12

15

17

20

Frames (millions)

RAD

ReachAvoid_1_1_1_5

—ReachAvoidRedemption_1_2_1_2

Number of Steps

60

RAD

CRA.1.1.1.5

CRAR.1.2.1.2

40

20

0

CRAD

.5.1.5

1.5.1.5

RA.1.2.1.2

RA.1.3.1.3

RAR.1.2.1.2

RAR.1.3.1.3

P.1.2.1.2

P.1.3.1.3

R.1.2.1.2

R.1.3.1.3

NT-CRAD

CRA.1.2.1.2

CRA.1.3.1.3

CRA.1

CRA.1.

CRAR.1.2.1.2

CRAR.1.3.1.3

CRAR.1.

CP.1.2.1.2

CP.1.3.1.3

CP.1.5.1.5

CP.1.1.1.10

CR.1.3.1.3

CR.1.5.1.5

CR.1.1.1.10

CRAR.1.

C.6 Generalization of pre-trained policies: GATv2 vs RGCN. Figure 16 present the results. It compares policies with frozen and pre-trained GNNs (both GATv2 and RGCN). In line with the training curves, we can see that GATv2 outperforms RGCN across the board, with the most significant differences on harder task classes with many symbols or many conjunctions. Figure 17 shows the number of steps taken in the same experiments, indicating that GATv2 also learns to complete the tasks with fewer steps compared to RGCN. The results highlight the superior performance of GATv2.

CRA

Satisfaction

1.0

0.5

GATV2

RGCN

0.0

8

R

Q

Q:

R

CRA

Q

8

8

7

8

Steps

40

GATv2

RGCN

Numberofs

20

0

Q

CRA.

8

8

8

R

C.7 Generalization of pre-trained policies in continuous domains. Figure 18 shows the generalization results for a policy (with a frozen and pre-trained GATv2) trained on RAD DFAs in Zones. The results show that our method can generalize to cDFAs with up to 10 DFAs in the composition as long as each DFA's task length is relatively short. We see that it does not perform as well in tasks with length 5 and more. This is probably due to the fact that control and navigation are much harder in continuous domains compared to discrete ones. Therefore, the low accuracies we are seeing for tasks with more length might be due to timeouts. It is worth noting that the policies can generalize up to 10 conjunctions in a cDFA, which is completely out of distribution for the policy. Overall, the results show that training on RAD DFAs provide zero-shot generalization in continuous domains as well.

Satisfaction

1.0

0.5

0.0

？

3

？

5

5

？

2

3

？

CRAD

.1.1.10

RA.1.2.1.2

RA.1.3.

RAR.1.2.1.2

RAR.1.3.1.3

P.1.2.1.2

P.1.3.1.3

R.1.2.1.2

R.1.3.1.3

NT-CRAD

CRA.1.2.1.2

CRA.1

CRA.1.

CRA.1.

CRAR.1.2.1.

CRAR.1.3.

CRAR.1.5.1.

CRAR.1.1.1.10

CP.1.2.1.

CP.1.3.1.3

CP.1.5.1.5

CP.1.1.1.10

CR.1.2.1.

CR.1.3.

CR.1.1.1.10

Figure 19 shows the number of steps taken to complete the episodes in the same experiment. We see that when the policy satisfies a task class with a high probability, it can also complete it with fewer steps. On the other hand, in almost all cases where it could not satisfy the task with high probability, it also took more steps in those episodes, suggesting that the policy might have timed out.

Num.of Steps

1000

500

0

？

3

？

心

3

？

CRAD

RA.1.2.1.2

RA.1.3.1.3

RAR.1.2.1.2

RAR.1.3.1.3

P.1.2.1.2

P.1.3.1.3

R.1.2.1.2

R.1.3.1.3

CRA.1.2.1.2

CRA.1.3.1.3

CRA.1.5.1.5

CRA.1.10.1.1

CRA.1.1.1.10

CRAR.1.2.1.

CRAR.1.3.1.3

CRAR.1.5.1.

CRAR.1.1.1.10

CP.1.2.1.2

CP.1.3.1.3

CP.1.5.1.5

CP.1.10.1.1

CP.1.1.1.10

CR.1.3.1.3

CR.1.5.1.5

CR.1.10.1.1

CR.1.1.1.10

C.8 Generalization of pre-trained policies: RAD cDFAs vs LTL2Action. To compare our approach to the closest work in the literature, we design an experiment with LTL2Action [38]. Recall that one of our main contributions is our RAD pre-training procedure that provides generalization across different tasks. Defining such a general LTL task class is not a trivial problem and is a contribution on this own. Moreover, the intuitions used to develop the RAD cDFA class are not applicable to LTL as it is not as easy to generate interesting and satisfiable LTL formulas. Therefore, for LTL2Action, we define a joint task distribution consisting of partially ordered and avoidance tasks, which are task classes defined in [38]. To sample LTL tasks, we flip a coin and pick a samplers.

1.0

Discounted return

0.8

0.6

0.4

0.2

0.0

0

2

4

6

8

10

Frames (millions)

Figure 20 present LTL2Action's learning curve for the pre-training of an RGCN in the joint LTL task distribution. As the figure shows, the model can easily learn to distinguish these LTL tasks.

Eventually.1.5.1.4

Until.1.3.1.2

1.0

1.00

0.8

0.75

ret

re

iscounted1

0.6

d

0.50

e

0.4

t

coun

0.25

0.2

S

0.00

-0.25

0

2

5

7

10

12

15

17

20

0

2

5

7

10

12

15

17

20

Frames (millions)

Frames (millions)

Nopretraining

Pretraining

Pretraining (frozen)

We then take the pre-trained RGCN and use it to train a policy that is exclusively trained on one of these task classes, i.e., partially ordered or avoidance LTL tasks. Figure 21 presents the learning curves for this training. The figure shows that if we let the RGCN get gradient updates during training, then pre-training does not help as much as freezing the network. We saw a similar behavior for RGCN in Figure 5, which further justifies our choice of using a network with an attention mechanism. Furthermore, in [38], authors show that pre-training on specific task classes helps training even when we let the RGCN continue getting gradient updates. Combining these results from [38] with what we are seeing on Figure 21 suggests that when pre-trained on rich classes with multiple distinct task patterns, RGCN need to forget about the other task classes it learned during pre-training and has to focus its capacity on the specific task class it is being trained on. We do not see such a gap in the learning curves when we let GATv2 get gradient updates, see Figure 5, suggesting that GATv2 does not suffer from the same issue as RGCN, and it learns more robust cDFA embeddings.

Steps

60

RADPretraining

LTLPartiallyOrdered

LTLAvoidance

of

Number

40

20

0

cPO.1.5.1.4

cPO.1.6.1.5

CPO.1.8.1.7

cPO.1.10.1.1

CAV.1.3.1.2

CAV.1.2.1.3

Figure 22 presents the number of steps taken for the generalization experiments, showing that in all cases our policy trained on RAD cDFAs perform comparably to the specialized LTL policies.

D Hyperparameters and Compute Usage

Both for the RGCN and GATv2 cDFA encoders, we use a hidden dimension size of 32 and perform 8 message passing steps. Also, for GATv2, we use 4 heads for the multi-head attention.

For both Letterworld and Zones, we implemented the same actor and critic architectures. The actor network comprises three fully connected layers, each with 64 units and ReLU activations, while the critic network has three fully connected layers with 64, 64, and 1 units, using a Tanh activation. For pretraining, we simplified the actor and critic to single-layer models without hidden units, allowing the cDFA encoder to learn a rich embedding space. For environments with discrete actions, the actor's output was passed through a log softmax and a logit layer. In continuous action settings, we used a Gaussian distribution, parameterizing the mean and standard deviation with two separate linear layers from the actor's output. To construct observation embeddings, for Letterworld, we employed a 3-layer convolutional network with 16, 32, and 64 channels, 2×2 kernels, and a stride of 1. In Zones, a 2-layer fully connected network with 128 units per layer and ReLU activations was used.

Table 1 shows the hyperparameters used for every training run in the experiments section. Each seed in the experiments section was run as an individual Slurm job with access to 4 cores of an AMD EPYC 7763 running at 2.45GHz and access to at most 20gb of memory. RAD pre-training experiments took approximately 14 hours each for 10 million timesteps, policy training in Letterworld took approximately 55 hours (50 hours for the frozen GNN version) for 20 million timesteps, and policy training in Zones took approximately 120 hours (70 hours for the frozen GNN version).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the introduction, we clearly list our contributions, and in the experiments section, we list the research questions we explore. We then relate these research questions and their answers to the listed contributions while discussing the experiment results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Throughout the paper and explicitly in the conclusion, we clearly state the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include any theorems or proofs. But, we still give a rigorous formalism mapping cDFA-conditioned RL to the well-known goal-conditioned RL problem.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the information needed to reproduce the main experimental results. In the corresponding section, we define all the RL environments, all the task classes, the pretraining procedure, the training procedure, and the algorithms used in the experiments. We also provide all the hyperparameter values in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In a footnote on the first page, we include a link to the webpage of the project. This page includes all data and code with instructions to reproduce the experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the training and test details in the main body of the paper as well as in the supplementary material included in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All plots in the paper report error bars whenever applicable. We also provide other appropriate information (e.g., number of seeds, sample size, etc.) about the statistical significance of the reported experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the details regarding computer resources are given in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the societal impacts of our work in the context of explainable and interpretable AI systems.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We clearly credit and cite all the original papers that produced the assets we used as baseline, benchmark, etc.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The RAD DFAs can be considered a new asset as it is a sampling distribution over DFAs. Its complete definition is given in both the main body and the appendix.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper involves neither crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper involves neither crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles Goal-conditioned Reinforcement Learning (RL) when used with Temporal Logic Objectives. The benefits of directly considering Deterministic Finite Automatons (DFAs) as the task definition leads to a new class of Compositional DFAs introduced that cover a conjunction of several tasks. These cDFAs can be encoded using a Graphical Neural Network that is pretrained on a novel task distribution (RAD). Experimental results show the approach generalizes to several task classes.\n\nSTRENGTHS:\n- Compositional DFAs are a more general form of multi-task descriptions. The proposed method is a straightforward way to encode these objectives and solve them.\n- The range of task classes considered is extensive and covers a variety of problem settings.\n\nWEAKNESSES:\n- The approach follows the similar structure as LTL2Action [41] but replaces the LTL Module with a different GNN (GATv2) and considers cDFAs or compositions/conjunctions of DFAs. The modifications w.r.t. LTL2Action apart from this and the pre-training step over a new randomly sampled (RAD) task distribution are novel yet arguably incremental.\n- Baselines appear lacking which bring into question the merits of the approach. A direct comparison to LTL2Action [41] and GCRL-LTL [32] using the monolithic DFA of the cDFA in the experiment could be possible and if not, the reason should be justified.\n- The paper is at times hard to follow and overloaded with notations and conventions. Some improvements could be made in the description of RAD pre-training (see questions) and the task nomenclature 1.X.1.X in the experiments.\n\nQUESTIONS:\n1. Should the statement in L35 be better quantified or validated with references? \"DFAs represent all temporal tasks that become Markovian by augmenting the state-space\nwith finite memory”. Does this hold when Signal Temporal Logic (STL) is considered a temporal task language as well? \n2. Is Reference [7] a concurrent submission or a typo?\n3. In Figure 5, what are the black elements in the Legend? Why are they not present in the figures?\n4. What do the envelope colors in Figure 2 indicate? Does the encoding generalize to conjunctions of the predicates on edges (e.g., Red & Green)? Are these considered during pretraining?\n5. Defn. 4.1. could use some clarity. How is a random transition defined in a mutation? Is this simply changing the target node of a given transition to a random state in the DFA? What is being “minimized\" here?\n6. Minor grammatical error - Should DFA and DFA(s) be used interchangeably? I would think the following references to DFA should be changed to DFAs (L31, L33, L80, L85-87, and more)\n\nLIMITATIONS:\nThe limits of cDFA encoding are not fully explored (mentioned).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the framework of goal-conditioned reinforcement learning to support temporally-extended goals specified as (compositions of) deterministic finite automata (cDFA), avoiding the limitations of state-based goals and the ambiguities of natural language specifications. The authors introduce a graph neural network architecture for embedding cDFAs to vectors via message passing. They then develop a pretraining distribution for learning cDFA embeddings, which is constructed by leveraging the insight that paths in a DFA correspond to a series of reach-avoid tasks, motivating a distribution of cDFAs that are derived from sequential reach-avoid DFAs. In experiments, the authors show that pretraining on this distribution produces generalizable embeddings that capture the structure of DFA space, while also enabling the training of generalizable cDFA-conditioned policies. The authors run these experiments on two labeled MDPs, Letterworld and Zones, showing that embeddings or policies trained on only RAD DFAs or other DFA subsets often achieve high goal satisfaction rates even when presented with cDFAs with longer task lengths and higher composition numbers. cDFA conditioned policies using a frozen cDFA embedder trained on the RAD distribution also lead to high satisfaction rates across the board in Letterworld, with reasonable strong performance in Zones (a harder continuous MDP).\n\nSTRENGTHS:\nThis is a very well-written and thorough paper that made for a pleasant and insightful read. Despite not working much with DFAs or GNNs, I found both the overall idea and the low-level details very easy to understand, because all key concepts are carefully but concisely explained, with figures that clearly illustrate what a cDFA is, how it can fed into a GNN, and what the structure of SRAs and RADs look like. The design of the RAD pretraining distribution was also very insightful, demonstrating careful thinking about what motifs ought to be repeated across a large class of DFAs (namely, reach-avoid tasks), and how that could be exploited to enable generalization. Experiments were very clearly laid out and thoroughly executed, covering a wide range of DFA classes to test generalizability, and demonstrating that the proposed method actually does lead to cDFA-conditioned policies that generalize quite well across the board.\n\nIn terms of motivation and impact, I think it's great that people are exploring more expressive and well-defined notions of goals in the context of goal-conditioned RL. While working with formal specifications is not very popular in machine learning these days, I think a strong paper like this one helps to elevate the profile of using such specifications to enable more interpretable and generalizable RL systems. As such, I think this paper could have a decently large impact within the fields of both goal-conditioned RL and people working on formal methods in RL / ML.\n\nWEAKNESSES:\nAs the authors themselves note, probably the main weakness / limitation of work that relies on symbolic specifications like cDFAs or LTL formulae is that it requires MDPs with a labeling function. This limitation isn't insurmountable --- for e.g., it should be possible to use computer vision algorithms to segment and label states of the world from pixels -- but in the final version of the paper, I think it'd be good to discuss this a bit more, since it might help readers appreciate how work like this could be applicable without a oracle labeling function.\n\nAnother slight weakness of the paper is that most of the experiments focus on Letterworld, with weaker average performance in the more realistic environment of  Zones. This raises questions about how applicable the method will be to more realistic tasks. With the extra page in the final version, I think it'd be good to show images of both environments, and examples of harder cDFAs in those environments, just to illustrate how difficult it might be to solve those tasks. Otherwise it might be easy to write-off e.g. Letterworld as a toy domain even though with cDFAs the tasks can be quite long-horizon. It might also be good to say a bit bout how performance could be improved for harder tasks like Zones.\n\nFinally, I think one thing that bears discussing in the final version is how this framework might enable safety/correctness of the policy, not just interpretability of the specification. While it's no doubt a safety benefit to have more interpretable specifications that say exactly what correct behavior is, this on its own doesn't guarantee the resulting learned policies are safe. I wonder if the authors could say a bit more on this point, perhaps by connecting to work on formal verification of NNs, or by combining cDFA-conditioned RL with constrained planning methods, using the learned policy to accelerate provably safe planning.\n\nQUESTIONS:\nThere were a couple of minor things I didn't fully grasp, which it'd be great to clarify:\n\n- When constructing/sampling a SRA DFA (e.g. RHS of Figure 3) how are reach or avoid transitions created? Is the reach transition always a transition to the accepting state? Or is it to a a state that eventually leads to an accepting state? Is it the case that there is a single long path to the accepting state? And is the avoid transition always to a single sink state? I couldn't fully understand this from Appendix B.\n\n- In Figure 4, in the panel for Zones, where is the (non-frozen) GATv2 pretraining line baseline? And am I right to say that the RGCN baseline was not trained for Zones?\n\n- For each MDP with its own labeling function and alphabet, do you basically to do the RAD pretraining phase for specifically that alphabet of symbols? Or is there hope of training a cDFA embedder that generalizes across different MDPs with different labeling functions and alphabets?\n\nLIMITATIONS:\nThe authors have adequately discussed a number of limitations of their work. It would be good to also discuss the fact that interpretable specifications (in the form of cDFAs) do not immediately lead to verifiably safe behavior.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on goal-conditioning policies using DFAs. This leverages the ability of DFAs to be composed - in this case the work focuses on conjunction. Two main pieces are introduced in leverging DFAs: 1) a GATv2 model which provides task embeddings from the DFA, 2) pre-training of the GATv2 on Reach Avoid Derived compositional automata which trains the embeddings with a basis representation in terms of the RAD embeddings which can then be used to represent other DFA embeddings. Once the  DFA embeddings is obtained it is appended to the environment state space and used to train a policy. Experiments demonstrate the benefit of GATv2 over an alternative RGCN (when pre-training and not freezing the embeddings) and also demonstrate the benefit of pre-training the task embeddings, particularly when freezing the embeddings after.\n\nSTRENGTHS:\n# Originality\nThe use of DFAs and similar representations of tasks is a popular idea in RL at the moment. However, this work focuses on the use of the DFA within a longer pipeline to obtain the embeddings. This use appears novel as most works I am aware of tend to use fairly literal and inflexible representations of the DFAs or state machines. This work then also puts some interesting pieces together - DFAs for task representations, GATv2 to represent a graph, RAD pretraining. Overall this all supports the novelty of this work and I do believe this is a strength.\n\n# Quality\nThe aim and hypothesis of this work is clear. Each step of obtaining the task embedding is justified and validated with experiments. The ablation study does demonstrate the utility of both GATv2 and the pretraining from RAD tasks. Moreover the additional experiments and visualisations support the fact that GATv2 is learning to produce reusable and semantically meaningful task embeddings.\n\n# Clarity\nOverall the paper is well written, notation is used sparingly and intuitive. This is particularly useful as the combination of DFAs and MDPs can often become very notation heavy. I appreciate the effort placed in making this work clear. Images and visualization are also neat and clear and support understanding.\n\n# Significance\nEnabling generalization in RL, and specifically task generalization is an important problem. It has particular utility for enabling safe and adaptable RL agents. This work provides a new perspective and step towards this goal. I do think that this work would lead to future work and new ideas.\n\nWEAKNESSES:\n# Clarity\nFigure captions could elaborate more. As it stands they are vague and do not add any explanation or support understanding of the figure. Secondly, Section 3.1 is unclear due to the omission of an explanation for the DFA modifications. As a result it is unclear why any of these steps are needed, but this seems to be an important and early part of the pipeline. Thus, it leaves the entire pipeline feeling unclear or at least on an uneasy grounding.\n\n# Significance\nUnfortunately, I think the significance of this work may be its biggest weakness. Firstly, it is not clear to me that it is trivial to extend the findings on conjunctions will generalize to general boolean combinations as is mentioned in lines 144 and 145. Secondly, the curse of dimensionality which results from conjunctions of DFAs is never addressed or touched on. Perhaps the vector representation of task embedding does support generalization - and the visualizations would even support this - but it is not discussed. This is even more worrying when considering that a fixed dimension for the task embeddings is used. Thirdly, the experiments do support the answering of the research questions posed. However, the experiments do not compare to any other baselines from the literature, which makes it difficult to contextualize and build from this work. Finally, and I acknowledge that the authors point this out in the limitations section and so I will take this point lightly, but there is no insight into how the task embeddings support generalization. This does unfortunately limit the significance of the work.\n\nQUESTIONS:\n1. How would the results of the generalized agents with task embeddings compare to specialized agents?\n2. In Figure 4, why do the Zones results only have two models?\n3. At a high level, how would you extend the approach to incorporate general boolean combinations\n\nIf the authors could address my concerns with regard to the curse of dimensionality, and answer these questions I would likely advocate for acceptance.\n\nLIMITATIONS:\nLimitations are mentioned and I do not see any omissions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the multi-task setting where each task is a temporal logic task specified by a conjunction of deterministic finite automata (cDFA). To address the sample efficiency and generalisation problems present in this setting, they propose a method for generating good cDFA embeddings which can then be used by a universal value function approximator to learn general policies. Precisely, they propose converting a given cDFA into a graph which can then be passed to a graphical neural network (GNN) like GATv2 to create a vector embedding of the task. Similarly to prior work that demonstrated the benefit of pretraining GNN embeddings on LTL tasks (LTL2Action), this paper also proposes the *reach avoid derived* DFA (RAD-DFA) task distribution to pretrain the GNN embeddings. They then demonstrate that using frozen pretrained embeddings leads to better performance than finetuning, and no pretraining leads to abysmal performance. Finally, they also demonstrate that their pretraining procedure leads to reasonable cDFA embeddings and help with generalisation.\n\nSTRENGTHS:\n## Originality\n- This work combines three established directions of research, namely: temporal logic task specifications (in the form of DFAs specifically), goal conditioned RL (using UVFAs specifically), and graph neural networks (GATv2 specifically).\n- The intersection of these three subfields is of more recent interest and only just emerging in the literature. Thus, there is an element of originality in the specific manner in which they are combined in this work with the aim of improving sample efficiency and generalisation in RL. \n- Also the specific manner in which a GNN is pretrained to generate good DFA embeddings (by introducing the RAD pretraining task set) appears to be fairly new.\n\n## Quality and Clarity\n- The work provides mathematical formulations for the main concepts needed to understand the approach, which aid the quality of the work. \n- Assumptions, while not formally stated are mentioned too. \n- The explanation of the framework is also detailed and coherent which makes the contributions of this work more apparent.\n- The paper includes a number of Figures which are helpful, clear and in the case of Figure 1, aids the message of the work quite a lot.\n- The paper includes a number of experiments that illustrate the goal embeddings learned by the proposed approach, and the resulting sample efficiency and generalisation over specific task distributions.\n\n## Significance\n- Temporal logic instruction following is an important direction of work and has wide applications---for example in multitask, lifelong learning and safe RL. \n- So leveraging GCRL (with UVFAs) and pretrained GNN embeddings to improve sample efficiency and optimality---important issues in general in RL---for DFA specified tasks can be particularly impactful and widely adopted. In this sense, this work could guide future work and ideas.\n\nWEAKNESSES:\n# Major\n\n## Originality\n- Overall, this work is very similar to LTL2Action (Vaezipoor et al. 2021) and does not deviate particularly far from the work presented there beyond considering DFA specified tasks instead of LTL ones, which is just a choice of task specification language (regular/DFA vs LTL/BuchiAutomata). Also the paper claims that LTL2Action is limited to \"finite\" LTL (line 82). It is not clear what is meant by \"finite\" here. If it means finite trace, then that is incorrect. LTL2Action is applicable to any LTL specification.\n- In addition, relying only on a UVFA for sample efficiency and generalisation is a widely known idea that has also been explored in prior works. While the exact implementation here may be new (such as the specific way the goal/task embedding is obtained), the ideas are familiar from a function approximation front. \n- The paper introduces the cDFAs, but this does not seem novel since they don't seem meaningfully different from DFAs and the distinction does not seem relevant for the proposed approach.  Hence this just adds more notations and terminologies. Also, the footnote on page 2 says \"With negations and disjunctions omitted as straightforward extensions\". It is not clear how they are straightforward, unless cDFAs are indeed just DFAs as I mentioned previously.\n\n## Quality and clarity\n### Mathematical formulations\n- I found it very hard to fully understand the proposed framework and judge the soundness because there are a number of incorrect statements and missing information.\n- The MDP definition (Def 2.1) is wrong. An MDP is a five tuple M = (S,A,T,R,$\\gamma$) where $R$ is the reward function and  $\\gamma$ is the discount factor (optional when $\\gamma=1$). It is not a triple M = (S,A,T). The objective of goal-conditioned RL is not to maximize the probability of generating a path containing g (lines 127-128), but instead to maximise the value function (the expected cumulative rewards). \n- It is fine if the authors want to focus on the maximisation of the success probability, but then the paper needs to describe how that will be achieved in the RL setting defined.\n- The reward function and discount factor for the problem formulation are never defined. The only reward function mentioned in the mathematical formulations is the one used for pretraining the GNN. Since the experiments uses the same reward scheme for learning policies, I suggest the authors formally state that in their problem formulation. Additionally, Table 1 shows that the experiments use a discounting, meaning that the learned policies *do not* correspond to the maximisation of success probability.\n- The paper says that goals are defined in the augmented MDPs (line 164), but the DFA-conditioned policy uses the environment states and not the augmented ones (Def 3.1).   \n - The paper uses message-passing neural network (MPNN) (line 173) as a core factor in their approach, but never define nor describe what these are.\n- The paper proposes cDFAs as a novel goal-representation for temporal tasks in goal-conditioned RL. Beyond the fact that I am not convinced that they are meaningfully different from DFAs, the paper gives no theory nor conducts any experiment to demonstrate that they \"balance formal semantics, accessibility, and expressivity\" as claimed in their first contribution. In general, since the authors claim it as a main contribution of this work, they should justify why it is better than other representations like RMs, LTL, GTL, TLTL, SPECTRL, etc. I think such rigorous justifications are not needed if they did not claim it as a contribution of this work.   \n\n### Experiments\n- It is not clear why the main paper contains no experiment comparing with prior works. The only such experiment is left to the appendix, and only uses LTL2Action as baseline which is not state-of-the-art (there is also no sample-efficiency comparison). Without state-of-the-art baselines like [1] or [2] in the main paper, it is hard to tell how good the proposed approach is. Given that [2] drastically outperforms LTL2Action in their experiments in the same domains, this work does not look promising as it only shows similar performance to LTL2Action. \n- The paper claims that the proposed approach is applicable to any DFA, but the DFAs used for pretraining the goal encoder (Algorithm 1) and the ones used for the experiments (page 7) only have one atomic proposition per edge (e.g. $red$, $square$, etc). These are the simplest DFA transitions possible. In general the edge of a DFA is a Boolean expression over the atomic propositions (e.g. $red \\wedge \\neg square$). It is unclear if the approach generalises to more complex DFAs (where the edges/transitions can be arbitrary Boolean expressions).  \n- All the example DFAs used are generally not satisfiable in the environments considered since their states lack self-transitions (e.g. in Fig 1). Since all the task distributions used for the experiments are defined similarly, it is unclear why the results show high success rates on them (they should be generally unsolvable). This suggests that important details of the implementation of the approach for the experiments may be different from what has been described in the paper.\n- It is also unclear what the tasks sampled from the defined distributions look like. It would have helped if the authors included examples of the simplest and most complex DFAs sampled during training and during testing. \n- It is unclear why freezing the pretrained weights performs better than finetuning. This is counter-intuitive but is never explained in the paper.\n- Given how the performance of the proposed approach is heavily reliant on the goal embeddings obtained from pretraining (the performance is abysmal without it), it is concerning that the paper did not stress test it to understand when it fails and why. For example, what if the number of conjunctions goes up to 100 where each DFA has a task length that also goes up to 100 (relates to the temporal curse of dimensionality)? What if the DFA transitions are more complex, that is when they are arbitrary Boolean expressions over the atomic propositions (relates to the spatial curse of dimensionality)?\n\n## Significance\n- This work does not make a very large step forward from prior work (the addition of state-aware planning), but it is one. So I do not want to be overly pessimistic on the front of significance, but it is necessary to note that the contributions of this work are relatively incremental. If I understood the paper correctly, the only main contribution is the approach for pretraining a *good* goal (DFA) embedding. Additionally Figures 20-21 also only shows similar performance to non-state-of-the-art prior work (LTL2Action). \n- The absence of theory does not help. Theory guaranteeing generalisation and convergence to optimal policies (as claimed) under some reasonable assumptions would have helped. The lack of experiments comparing with state-of-the-art baselines like [1,2] also does not help.\n- Relying only on a UVFA for generalisation has clear limitations (e.g. in terms of the spatial and temporal curses of dimensionality present in temporal logic tasks), since no optimality and generalisation guarantees can be given even in finite MDPs (like grid-worlds). This is why the field has largely moved away from such approaches. More recent works such as [1,2] (and the large majority of prior works like [3,4]) break down the problem into a set of reach-avoid sub-tasks (goals), then use planning (or HRL) to determine how to sequence those goals (addressing the temporal curse). These works use UVFA for the sub-goal policies only when the goal space is sufficiently large. When the goal space isn't too large, such as in the letter grid-world, these methods can still learn and generalise without relying on a UVFA (by learning the policies/options for each goal independently).\n- This reliance on end-to-end function approximation for generalisation hurts the significance of this work.\n\n\n[1] Tasse, Geraud Nangue, et al. \"Skill Machines: Temporal Logic Skill Composition in Reinforcement Learning.\" The Twelfth International Conference on Learning Representations. (2024)\n\n[2] Qiu, Wenjie, Wensen Mao, and He Zhu. \"Instructing goal-conditioned reinforcement learning agents with temporal logic objectives.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Araki, Brandon, et al. \"The logical options framework.\" International Conference on Machine Learning. PMLR, 2021.\n\n[4] Jason Xinyu Liu, Ankit Shah, Eric Rosen, George Konidaris, and Stefanie Tellex. Skill transfer for temporally-extended task specifications. arXiv preprint arXiv:2206.05096, 2022.\n\n# Minor\n\n- It is not clear what \"concept class\" means or how it is different from a \"task class\". Are they the same?\n- It is not clear what GATv2 in the caption of Figure 1 means. Please add in the caption that it is a GNN.\n- It is seems like \"task\" and \"goal\" are mean the same thing in this work (line 16), i.e. they mean a DFA. Is that correct? That wasn't clear. \n- Adding the cDFA in Fig 1 to Fig 2 will\n- There are a couple minor typos or writing mistakes. For example:\n - There are numerous places where shorthands like \"i.e.\" and \"e.g.\" are used (e.g. first paragraph of the introduction). It is not recommended in a formal writing. Please spell them out.\n - The sentence on Line 184 doesn't end.\n\nQUESTIONS:\nPlease refer to the weaknesses above.\n\nLIMITATIONS:\nThe authors have discussed some of the limitations. However, there seems to be several other ones. For example,\n\n- Just like *some* prior works, the paper assumes that all tasks are defined over a fixed set of given atomic propositions. If the set increases, everything needs to be retrained.\n- The paper demonstrates their approach only on DFAs with one atomic proposition per edge. It is unclear how it will behave when the DFA transitions are more complex (i.e. arbitrary Boolean expressions over the atomic propositions)?\n- The paper relies on UVFAs to learn general policies. Hence, it is likely that it can only generalise to tasks *similar* to those it is trained on, and not to tasks that are *significantly* out of distribution. The paper does not investigate when the model fails to generalise and why. That would have helped the reader have some justified intuition of what types of tasks are *significantly* out of distribution for function approximation to fail for proposed approach.\n\n\nI recommend the authors carefully think of the various significant limitations of their approach, such as the one I mentioned above, and explicitly state and discuss them in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-24"}