SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The weight encodes the capability of a goal-condition agent to transit from  $q_{i}$  to  $q_{j}$  from an observation at environment state  $s$ . In Fig. 4(d), we restrict  $\mathcal{L}(q_i)$ ,  $\psi$ , and  $\psi'$  to conjunctions of atomic propositions or negated propositions. In Appendix G.3, we extend this definition to include formulas as conjunctions of propositions and their negations to support more complex LTL tasks, such as  $\mathbf{F}(g_1 \wedge \neg g_2)$ .
In Fig. 4(e), the transition is similar to that of Fig. 4(d), and the agent can also take as many steps as necessary to reach the goal region specified by  $\psi$  by utilizing the self transition on the source node  $q_{i}$ . However, there are no specific avoidance requirements to consider. We use Equation 6 to weigh this kind of transition. In Fig. 4(f), the goal-conditioned agent is required to reach the goal region  $\psi$  in exactly one step as there is not any self transition that can be taken upon  $q_{i}$ . We set  $\mathcal{W}(s,(q_i,\psi ,q_j)) = 0$  (the full capacity) if  $\mathcal{L}(q_i)\Leftrightarrow \psi$  i.e. the agent can directly "jump" to  $q_{j}$  (e.g. see the rightmost non-self transition in Fig. 4(b)). Otherwise, we assign it the lowest capacity  $\mathcal{W}(s,(q_i,\psi ,q_j)) = \infty$ . This is because our approach cannot explicitly pick an action to reach a goal region in a single step (we instead depend on the goal-conditioned agent to take as many steps as needed for goal reaching).
3.2 Algorithm Summary
Our technique generates policies for a LTL specification  $\varphi$  based on a goal-conditioned RL agent  $\pi$  and a learned goal value function  $\mathcal{V}$  as follows:
1. We first convert  $\varphi$  to a Buchi automaton  $\mathcal{B}$ , which is subsequently converted to a graph representation  $G_{\mathcal{B}}$  using the technique illustrated in Sec. 3.1.
2. Associate each transition  $(q_i, \psi, q_j)$  on  $G_{\mathcal{B}}$  with weight  $\mathcal{W}(s, (q_i, \psi, q_j))$  according to Equation 6 to measure the capability of the goal-conditioned agent to make the transition. In our implementation, we set  $s$  as the initial state of an episode. A more advanced planning strategy may conduct task planning every  $h$  timesteps from a current environment state for the remaining sub-goals on  $G_{\mathcal{B}}$  and update the weight of any transition based on the current state before replanning.
3. Decompose  $G_{\mathcal{B}}$  into strongly connected components (SCCs) using Tarjan's algorithm Tarjan [1972].
4. To find an optimal path on  $G_{B}$  for task execution to satisfy the LTL  $\varphi$ , we follow these steps:
- For each accepting state  $s_a$  within a maximal SCC Geldenhuys and Valmari [2004], we use Dijkstra's algorithm to find the shortest path from the initial state to  $s_a$ , denoted as  $p$ .
- Next, we apply Dijkstra's algorithm to find the shortest cycle from  $s_a$  back to  $s_a$  in the maximal SCC, denoted as  $q$ .
- The optimal path for the accepting state  $s_a$  is  $pq^\omega$  where  $\omega \to \infty$  represents the number of times the shortest cycle is executed. The cost of the optimal path is calculated as  $w(p) + w(q)^\omega$  where  $w(p)$  or  $w(q)$  is the sum of the weights of the transitions on  $p$  or  $q$ . In the implementation, we use  $\omega = 5$  to estimate the path cost.
- Finally, we select the optimal path on  $G_{\mathcal{B}}$  as the least-cost path to any accepting state of  $G_{\mathcal{B}}$ .
5. For task execution, on the searched optimal path  $pq^{\omega}$ , we use a goal-conditioned agent to subsequently achieve the goals along the path  $p$  and reach the goals on  $q$  iteratively in a loop to satisfy  $\varphi$ . For any transition  $(q_i,\psi ,q_j)$  on a searched path, the agent employs its goal-conditioned policy  $\widetilde{\pi} (\cdot |s,\psi)$  to reach the targeted goal region  $\psi$  (Fig. 4(d),(e)).
Justification. Recall that our objective of learning a LTL specification satisfying policy (Equation 3) is finding  $\pi^{*} = \arg \max_{\pi}\mathbb{E}_{\tau \sim \pi (\cdot ,\cdot ,\varphi)}[\mathbb{1}[\tau \models \varphi ]]$ , which generates the maximum number of LTL-satisfying runs for a given LTL property  $\varphi$ . Our formalization of goal-condition RL uses a sparse
binary reward function (Equation 1) - a reward of 1 is provided only when the specified goal is successfully achieved by the end of a training episode. In this setting, when the discounted factor  $\gamma \rightarrow 1$ , the weight  $w$  of a transition on  $G_{\mathcal{B}}$ , which is determined by the learned value function  $\mathcal{V}$  (e.g.  $w = -\log \mathcal{V}(\cdot)$ ), is inversely proportional to the probability of reaching the goal region represented by the target node from that represented by the source node. As such, based on the capability of the goal-conditioned agent (i.e. the learned value function  $\mathcal{V}$ ), the task planning algorithm seeks the optimal path as the one the agent is most likely to succeed.
3.3 Handling Avoidance
As depicted in Fig. 4(d), our method uses the goal-conditioned agent policy  $\widetilde{\pi}$  to fulfill a reach-avoid subtask defined as  $\widetilde{\pi}(\cdot | s, \bigwedge_{j} g_{j}, \bigwedge_{k} \neg g_{k})$  - from a state  $s$  an agent is required to avoid several regions  $g_{k} (k \geq 1)$  before reaching the target region  $\bigwedge_{j} g_{j}$  in the goal space. Our method only performs avoidance in situations where there is a high likelihood of colliding with any  $g_{k}$ . The likelihood of a collision occurring with  $g_{k}$  from a current state  $s$  can be assessed by evaluating the value function  $V(s, g_{k})$  of the goal-conditioned agent. If  $V(s, g_{k})$  exceeds a threshold  $\sigma$ , it indicates a high likelihood of collision. We formally define the reach-avoid policy  $\widetilde{\pi}$  as follows:
The strategy is to find the most dangerous zone  $g_{k}$  to avoid. A predicted value  $V(s, g_{k})$  below the threshold  $\sigma$  (e.g., 0.85) signifies that  $g_{k}$  is not considered dangerous and does not need to be avoided. We then take a goal-reaching action  $\arg \max_{a} \min_{j} Q^{\pi}(s, g_{j}, a)$  Tasse et al. [2020] for reaching the overlapped goal space covered by  $\wedge_{j} g_{j}$ . Otherwise, we select a safe action  $s = \arg \min_{a} Q(s, g_{k}, a)$  that moves the agent away from the dangerous zone  $g_{k}$  and identify a dangerous action  $d = \arg \max_{a} Q(s, g_{k}, a)$  that moves the agent directly to the dangerous zone  $g_{k}$ . We then select a "blocked" goal-reaching action  $t = \arg \max_{a \neq d} \min_{j} Q^{\pi}(s, g_{j}, a)$  to eliminate the impact from the dangerous action  $d$ . For continuous environments, we generate the combined action  $s + t$  that accounts for both safety and reachability. For discrete environments, we simply return  $t$  to block the effect of unsafe actions.
$$
\widetilde {\pi} \left(\cdot | s, \bigwedge_ {j} g _ {j}, \bigwedge_ {k} \neg g _ {k}\right) \equiv
$$
$$
\text {l e t} k = \arg \max  _ {k} V (s, g _ {k}) \text {i n}
$$
$$
i f i f V (s, g _ {k}) <   \sigma
$$
$$
\text {t h e n} \underset {a} {\arg \max } \underset {j} {\min } Q ^ {\pi} (s, g _ {j}, a)
$$
$$
\mathrm {e l s e}
$$
$$
\text {l e t} d = \arg \max  _ {a} Q (s, g _ {k}, a)
$$
$$
\text {l e t} s = \arg \min  _ {a} Q (s, g _ {k}, a)
$$
$$
\text {l e t} t = \underset {a \neq d} {\arg \max } \underset {j} {\min } Q ^ {\pi} (s, g _ {j}, a)
$$
$$
s + t
$$
Figure 5: Handling reach-avoid subtasks by goal-conditioned agents in GCRL-LTL.
We implemented our algorithm in a tool called GCRL-LTL² that extends goal-condition RL to task-oriented RL where tasks are specified by LTL objectives.
We evaluate GCRL-LTL in the ZoneEnv environment shown in Fig. 1 and the Ant-16rooms environments depicted in Fig. 3. We also include a  $7 \times 7$  discrete grid-based environment LetterWorld from Andreas et al. [2017]. In this environment, 12 unique letters occupy 24 positions out of 49 squares, where each letter is presented twice. This denotes the agent can satisfy an atomic proposition by reaching any of the two. The initial positions of the robots and the zones or letters are random in every episode. More information about these environments is provided in Appendix G.1.
Primitives. As shown in Fig. 5, our agent policy requires enumerating the action space to perform an avoidance maneuver when a collision is imminent. To support efficient enumeration over continuous action space, we pretrain four neural primitive skills that can move the Point robot in ZoneEnv and Mujoco ant in Ant-16rooms along the four cardinal directions UP, DOWN, LEFT, RIGHT. Each of the skills is trained for 0.5 million steps. More details for primitive training are given in Appendix G.2. To ensure a fair comparison, both our learning algorithm and the baselines get access to the neural primitives, which enable a discrete action space for both.
$^{2}$ GCRL-LTL is available at https://github.com/RU-Automated-Reasoning-Group/GCRL-LTL
Figure 6: (a) and (b) shows success rates and discounted rewards of LetterWorld on partially-ordered tasks, respectively. (c) and (d) shows success rates and discounted rewards on avoidance tasks, respectively. We report the average results on 5 random seeds, where each data point is acquired by collecting results from 100 episodes.
(a)
(b)
Figure 7: Comparisons of (a) success rate and (b) discounted rewards in ZoneEnv on avoidance tasks. We report the average results on 5 random seeds, where each data point is acquired by collecting results from 50 episodes. The initial 2 million steps are omitted due to the training of the dynamic primitives.
We extend Proximal Policy Optimization (PPO) (Schulman et al. [2017]) to a goal-conditioned RL algorithm to train the goal-conditioned agents for LetterWorld and ZoneEnv. We use an extension of the GCSL Ghosh et al. [2021] algorithm to train a goal-conditioned agent for Ant-16rooms. More details about the training algorithms are provided in Appendix B.
4.1 Evaluation in Multi-Task Settings
Baseline. We compared GCRL-LTL with LTL2Action ( Vaezipoor et al. [2021]). This baseline exemplifies the state-of-the-art learning algorithms for LTL-satisfying policies that use advanced neural architectures to encode LTL task formulas for policy decision-making. In contrast, GCRL-LTL employs weighted graph search to decompose LTL tasks into basic reach-avoid tasks that can be accomplished by goal-conditioned agents.
We consider the following LTL task spaces to evaluate GCRL-LTL and the baseline.
Partially-Ordered Tasks. Such a task consists of multiple streams of sub-tasks. A sub-task contains a sequence of goals that must be satisfied in a specified order. For instance, a possible task can be  $\mathbf{F}(p\wedge \mathbf{F}(q\wedge \mathbf{F}r))\wedge \mathbf{F}(s\wedge \mathbf{F}q)$ . This task can be described as "satisfy  $p$ ,  $q$ ,  $r$  in that order; and satisfy  $s$ ,  $q$  in that order". One valid solution is to reach  $s$ ,  $p$ ,  $q$ ,  $r$  in order to satisfy all sub-tasks.
Avoidance Tasks. Similar to partially ordered tasks, a sequence of propositions must be satisfied in order, while some propositions must be avoided. Visiting a prohibited position leads to failure. An example avoidance task is  $\neg j\mathbf{U}(w \wedge (\neg y\mathbf{U}r))$  (Fig. 2), which can be interpreted as a two-stage task: "reach  $w$  while avoiding  $j$  in the first stage, then reach  $r$  while avoiding  $y$  in the second stage".
During training, LTL2Action has access to a task sampler that produces a random LTL task from the large set of possible task spaces described above for each training episode. GCRL-LTL does not use the task sampler and is only trained to reach random goals. In evaluation, tasks are randomly sampled in each episode for both agents. We choose  $\sigma = 0.85$  in Fig. 5 and perform ablation study on this value in Appendix G.3.
Figure 6 shows the evaluation results across training iterations on both partially ordered and avoidance tasks in LetterWorld. Fig. 7 demonstrates the results for avoidance tasks in ZoneEnv. GCRL-LTL outperforms LTL2Action by a large margin despite having no access to the task sampler during training. Furthermore, we assess the generalizability of GCRL-LTL and the baseline by testing
(b)
Figure 8: Figures (a) and (b) depict the performance of trained goal-conditioned agents in satisfying the  $\omega$ -loop and  $\omega$ -stability tasks in the ZoneEnv environment. Each data point represents the average performance obtained from evaluation on 20 episodes, and the results are averaged over 5 random seeds.
Environment	Tasks	Timeout	Ours	LTL2Action
LetterWorld	Partially-Ordered	75	0.996(0.423)	0.661(0.129)
Partially-Ordered + Depth ↑	225	0.986(0.259)	0.477(0.033)
Partially-Ordered + Conjuncts ↑	225	0.992(0.410)	0.567(0.049)
ZoneEnv	Avoidance	1000	0.958(0.632)	0.934(0.601)
F(a ∧ X(¬bUc)) ∧ G(¬d)	1000	0.928(0.543)	0.342(0.304)
F(a ∧ F(b ∧ F(c ∧ Fd))	2000	0.926(0.220)	0.060(0.020)
Table 1: Performance of trained agents on original and out-of-distributions tasks. We report the success rate and discounted return(in parentheses). Results are averaged over 1000 episodes.
them on more complex tasks that are out-of-distribution to the task sampler used in baseline training. Specifically, for partially ordered tasks in LetterWorld, we increase the maximum depth from 5 to 15, and the number of conjunctions from 4 to 12. Similarly, we extend the avoidance tasks in ZoneEnv to a more difficult setting where the agent needs to avoid one extra zone in the second stage  $(\mathbf{F}(a \wedge \mathbf{X}(\neg b\mathbf{U}c)) \wedge \mathbf{G}(\neg d))$ . Additionally, we test if agents trained in ZoneEnv can efficiently solve tasks in which a long sequence of consecutive zones must be reached ( $\mathbf{F}(a \wedge \mathbf{F}(b \wedge \mathbf{F}(c \wedge \mathbf{F}d)))$ ). The experimental results, as detailed in Table 1, indicate that GCRL-LTL significantly outperforms the baseline in its ability to generalize to unseen tasks.
$\omega$  -regular LTL Tasks. We conducted experiments to evaluate the performance of GCRLLTL in satisfying  $\omega$  -regular LTL objectives.  $\omega$  -Stability Tasks: such a task requires the agent to satisfy a task proposition by reaching a goal region and remaining in the goal region infinitely.  $\omega$  -Loop Tasks: such a task in the form  $\mathbf{GF}(a\wedge \mathbf{XF}b)\wedge \mathbf{G}(\neg c)$  requires the agent to continuously cycle between two specific zones,  $a$  and  $b$  while always avoiding another zone,  $c$  . For example, in the task  $\mathbf{GF}(r\wedge \mathbf{XF}y)\wedge \mathbf{G}(\neg w)$  , the agent is expected to traverse between the Red and Yellow zones indefinitely, while never visiting any White zone. We evaluate the satisfiability of  $\omega$  -loop and  $\omega$  stability tasks in the ZoneEnv environment during the goal-conditioned agent training process. The results are presented in Figure 8. The  $x$  -axis
represents the number of environment interaction steps, while the  $y$ -axis represents the number of  $\omega$  occurrences. In the  $\omega$ -loop tasks it represents the number of rounds completed by the agent in looping between specific zones, and in the  $\omega$ -stability tasks it represents the number of transitions the agent successfully remains in the desired zone, within the first 1500 timesteps (higher values indicate better performance). The results confirm that GCRL-LTL can handle  $\omega$ -regular LTL specifications even though the underlying goal-conditioned agents have never seen such specifications during their training time. Figures 9a and 9b illustrate agent trajectories that solve the above  $\omega$ -LTL task. In both
(a)
Figure 9: Figures (a) and (b) illustrate agent trajectories that solve the  $\omega$ -loop task  $\mathbf{GF}(r \wedge \mathbf{XF}y) \wedge \mathbf{G}(\neg w)$  in the ZoneEnv environment. The initial positions of the robot are represented by orange diamonds  $\diamond$ .
trajectories, the robot quickly reaches the Red zone and subsequently oscillates between the Red and Yellow zones. More examples can be found in Appendix E.
4.2 Comparison with Compositional RL
We also compare our approach with DiRL (Jothimurugan et al. [2021]), which is a state-of-the-art compositional RL algorithm for LTL-satisfying policies. DiRL simultaneously learns a low-level policy for each sub-goal transition and a high-level policy to plan over the sub-goal space. However, it is restricted to a small fragment of LTL, cannot handle  $\omega$ -regular LTL properties (e.g. Fig. 1), and cannot be applied to multi-task RL because the low-level policies are specific to a single environment setting and do not generalize across different environments. We used 8 LTL task specifications. These tasks have increasing levels of difficulty as they require the agent to sequentially reach a growing number of sub-goals. An example instruction is given in Fig. 3. DiRL has to train a separate agent for each of the LTL specifications. In contrast, GCRL-LTL trains a single goal-conditioned agent and evaluates this agent over all 8 LTL specifications. Our agent achieves a success rate of approximately  $90\%$  in fulfilling all specifications after being trained with 3e6 environment steps, whereas the DiRL method has to exercise 3e6 steps for each of the specifications to match the success rate of GCRL-LTL. More comprehensive evaluation results are given in Appendix F.
There exists a large amount of previous work in using linear temporal logic (LTL) for specifying or shaping the reward functions for reinforcement learning (RL) Aksaray et al. [2016], Littman et al. [2017], Sadigh et al. [2014], Hasanbeig et al. [2018], Cai et al. [2021], Hahn et al. [2018], Camacho et al. [2019], Hasanbeig et al. [2020], Jothimirugan et al. [2020], Icarte et al. [2018, 2022]. These methods primarily concentrate on learning a single, specific task defined by an LTL objective. In contrast, our approach involves learning task-oriented policies akin to goal-conditioned RL agents within a multitasking framework. In this setup, a new arbitrarily complex LTL task is sampled as a goal for each episode.
Past research efforts have also aimed to empower RL agents to handle previously unseen multi-task instructions expressed in LTL. One common solution is decomposing complex LTL tasks into smaller, independent subtasks to simplify the learning process. For example, Araki et al. [2021] develops a hierarchical options framework that learns various neural networks each specialized for one subtask, and optimally composes these learned options via value iteration to solve temporal logic specifications. However, subtask execution does not consider what the agent must do afterward and hence might perform sub-optimally in solving the full task. León et al. [2022] progresses towards completion of an LTL task by greedily identifying subtasks of the original LTL instruction that remain to be addressed, which may result in myopic behaviors. Recent works show that the agents' neural architecture to embed LTL specifications is the key to improving the performance of RL agents in unseen environments. Kuo et al. [2020] presents a novel network architecture framework to compose neural networks one for each LTL operator and environment object to mirror the formula structure. In a similar vein, Vaezipoor et al. [2021] uses neural encodings to interpret full LTL specifications to guide an RL agent to learn task-conditioned policies in multitask settings. Voloshin et al. [2023] developed eventual discounting that exploits the fact that optimally satisfying the LTL specification does not depend on the length of time it takes to reach accepting states (e.g., "eventually always reach the goal") to find LTL-satisfying policies with the highest achievable probability. We discuss the other related work in a more general context in Appendix C.
This paper presents GCRL-LTL to learn task-oriented policies where tasks are specified by Linear Temporal Logic (LTL) instructions. Unlike existing methods that sample a vast set of LTL instructions during training, our approach enables simple goal-conditioned RL agents to follow arbitrary and complex LTL specifications without the need for additional training over the LTL task space. GCRL-LTL is capable of handling diverse and intricate LTL specifications including  $\omega$ -LTL objectives over infinite time horizons. Through extensive experiments, we demonstrated the effectiveness of GCRL-LTL to seamlessly satisfy complex temporal logic task specifications in a zero-shot manner.
We thank the anonymous reviewers for their comments and suggestions. This work was supported by NSF Award #CCF-2124155 and NSF Award #CCF-2007799.
Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, and Wei Yang. Mapgo: Model-assisted policy optimization for goal-oriented tasks, 2021.
Lewis Hammond, Alessandro Abate, Julian Gutierrez, and Michael J. Wooldridge. Multi-agent reinforcement learning with temporal logic specifications. CoRR, abs/2102.00582, 2021. URL https://arxiv.org/abs/2102.00582.
Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BkxUvnEYDH.
Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Program-guided transformer for program-guided tasks, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012. doi: 10.1109/IROS.2012.6386109.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/7634ea65a4e6d9041cdf3f7de18e334a-Paper.pdf.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html.
Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=6Tk2noBdvxt.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801.01290, 2018. URL http://arxiv.org/abs/1801.01290.
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
A GCRL-LTL Limitations 15
B Pseudo Code for Goal-Conditioned Reinforcement Learning 15
B.1 Pseudo Code for the Goal-Conditioned PPO Algorithm 15
B.2 Pseudo Code for the Goal-Conditioned Supervised Learning Algorithm 15
C Additional Related Work 16
C.1 Goal-conditioned Reinforcement Learning 16
C.2 Reinforcement Learning Meets Logic and Specifications 17
D Demonstrations of the LTL Tasks in Sec. 4 18
D.1 Avoidance Tasks 18
D.2 Avoiding More Tasks 18
D.3 Goal Chaining Tasks. 19
E Demonstration on  $\omega$ -LTL Tasks 20
E.1  $\omega$  -Loop Tasks 20
E.2  $\omega$  -Stability Tasks 20
F Demonstrations of the LTL Tasks on Ant 16-room Environments 20
F.1 Tasks: LTL Specifications 21
F.2 Results. 22
F.3 Ablation Study 23
G Implementation Details 24
G.1 Environment Details 24
G.2 Primitive Policies Details 26
G.3 Handling Transitions 27
G.4 Hyperparameters 28
A GCRL-LTL Limitations