SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The main limitation of our method is that we assume the atomic propositions in LTL properties can only be goals within the goal space of the underlying goal-conditioned policy e.g. colored zones in ZoneEnv navigation. We do not allow other sources of atomic propositions e.g. external environment signals that are out of the agent's control. For example, our current algorithm does not apply when the agent needs to pursue different tasks based on an external signal.
A potential solution to the aforementioned limitation is using a task monitor, which acts as an external memory, to maintain a record of completed sub-goals and past external environment signals. During task execution, when receiving a new environmental signal, our task planning algorithm can dynamically revise the high-level path for the remaining sub-goals that the goal-conditioned agent needs to achieve. We leave it for future work.
The other limitations of GCRL-LTL include (1) the overapproximation of the true optimal probability by our task-planning algorithm for  $\omega$ -regular LTL properties by bounding  $\omega$  to a finite number in Sec. 3.2; (2) dynamic handling of avoidance during test time instead of during planning - our task-planning algorithm does not consider the goal-conditioned agent's ability to stay safe before reaching a sub-goal; (3) the lack of an alarm system for unsatisfiable task specifications. We experimented with unsatisfiable task specifications. For properties such as  $\mathbf{F}(a \wedge b)$ , when the goal regions  $a$  and  $b$  are close but not overlapping the agent's behavior mirrors that of  $\mathbf{GF} a \wedge \mathbf{GF} b$ , oscillating between  $a$  and  $b$ . However, if these goal regions are far apart, our agent does not exhibit good-looking behavior.
B Pseudo Code for Goal-Conditioned Reinforcement Learning
B.1 Pseudo Code for the Goal-Conditioned PPO Algorithm
Algorithm 1 illustrates the pseudocode for our goal-conditioned proximal policy optimization (PPO) algorithm. This algorithm builds upon the standard PPO algorithm (Schulman et al. [2017]). Notable variations in the training process include the following: (1) when gathering experiences from the environment, it is crucial to synchronize the environment, goal representation, and the reward function; (2) an extended version of the value function  $V_{\phi}(s,g)$  and policy  $\pi_{\theta}(s,g)$  is necessary, which take both an environment state  $s$  and one of the goal representations  $g$  as inputs.
Including the goal as an input in goal-conditioned reinforcement learning algorithms enables the agent to explicitly incorporate the desired objectives into its decision-making process and to learn a policy that is explicitly aware of the desired goals. As such, the policy provides a mechanism for the agent to condition its behavior on the desired outcome, allowing it to generate actions that are tailored to the specific goal at hand. When the goal is included as an input to the value function, it helps the agent estimate the value of being in a particular state while aiming for a specific goal. This assists in determining the quality or potential of a state.
As discussed in the paper, we leverage goal-conditioned RL agents to interact with low-level environments to learn how to achieve basic goals without considering higher-level tasks. To solve an unseen LTL instruction, we engage in high-level reasoning over the difficulty of achieving the specified goals in the goal space to optimally achieve the sequences of tasks specified in the LTL objective. To this end, we learn a function  $\mathcal{V}: S \times \mathcal{G} \times \mathcal{G} \to \mathbb{R}$  to estimate the capability of taking the goal-conditioned policy  $\pi$  to go from one goal to another goal on average. We regress  $\mathcal{V}$  towards the value function of the a goal-conditioned agent by minimizing:
$$
\min  _ {\mathcal {V}} \left(V \left(s _ {t}, g _ {k}\right) - \mathcal {V} \left(s _ {j}, g _ {t}, g _ {k}\right)\right) ^ {2} \quad \text {w h e r e} g _ {k} \in L \left(s _ {k}\right) \wedge g _ {t} \in L \left(s _ {t}\right)
$$
with  $\tau \sim B$  (a replay buffer),  $t \sim \{0 \dots t_{\text{max}}\}$ ,  $s_t, a_t, s_{t+1} \sim \tau$ ,  $j \sim \{0, t\}$ ,  $k \sim \{t + 1 \dots t_{\text{max}}\}$ , and  $L$  being given as the goal labeling function. In Algorithm 1, we train  $\mathcal{V}$  together with the goal-conditioned PPO algorithm.
B.2 Pseudo Code for the Goal-Conditioned Supervised Learning Algorithm
Algorithm 2 extends goal-conditioned supervised learning (GCSL) (Ghosh et al. [2021]), following its iterative procedure of sampling trajectories, relabeling them, and training a policy until convergence. It trains an agent to achieve specific goals in a given environment. The primary enhancement
Algorithm 1 Goal-Conditioned Proximal Policy Optimization Algorithm
Require: Initial policy parameters  $\theta_0$ , initial value function parameters  $\phi_0$ , initial Q function parameters  $\beta$ , initial goal value function parameters  $\omega$ , goal-labeling function  $L:S\to \mathcal{G}$  from state space  $S$  to goal space  $\mathcal{G}$ , discount factor  $\gamma$ , clip range  $\epsilon$
Ensure: Goal-conditioned policy  $\pi_{\theta}(s,g)$
for  $k = 0,1,2,\dots ,N$  do
$$
\mathcal {D} _ {k} = \{\}
$$
for  $m = 0,1,2,\ldots ,M$  do
Sample  $g_{m}\sim \mathcal{G},s_{0}\sim S$
$\mathcal{D}_k^m = \{\tau_m = (\dots ,(s_i,g_m,a_i,r_i),\dots)\}$
$\mathcal{D}_k = \mathcal{D}_k\cup \mathcal{D}_k^m$  Acquire trajectories by running policy  $\pi_k(s,g_m) = \pi (\theta_k)$
Compute rewards-to-go  $\tilde{R}_t$
Compute advantage function  $\hat{A}_t$  based on the current value function  $V_{\phi_k}$
$\triangleright$  Collect goal-conditioned trajectories
$\triangleright$  Update the policy  $\pi_{\theta}$
$$
\theta_ {k + 1} = \arg \max _ {\theta} \frac {1}{| \mathcal {D} _ {k} | T} \sum_ {\tau \in \mathcal {D} _ {k}} \sum_ {t = 0} ^ {T} \min  (\frac {\pi_ {\theta} (a _ {t} | s _ {t} , \tau_ {g})}{\pi_ {\theta_ {k}} (a _ {t} | s _ {t} , \tau_ {g})} A ^ {\pi_ {\theta_ {k}}} (s _ {t}, \tau_ {g}, a _ {t}), f (\epsilon , A ^ {\pi_ {\theta_ {k}}} (s _ {t}, \tau_ {g}, a _ {t}))),
$$
where  $\tau_{g}$  is the goal of rollout  $\tau$  and  $f(\epsilon, A) = \left\{ \begin{array}{ll} (1 + \epsilon)A, & A \geq 0 \\ (1 - \epsilon)A, & A < 0 \end{array} \right.$
$\triangleright$  Update the  $Q$  function  $Q_{\beta}$
$$
\beta_ {k + 1} = \arg \min  _ {\beta} \frac {1}{| \mathcal {D} _ {k} | T} \sum_ {\tau \in \mathcal {D} _ {k}} \sum_ {t = 0} ^ {T} \left(Q _ {\beta} \left(s _ {t}, \tau_ {g}, a _ {t}\right) - \left(r _ {t} + \gamma V _ {\phi_ {k}} \left(s _ {t + 1}, \tau_ {g}\right)\right)\right) ^ {2}
$$
$\triangleright$  Update the goal value function  $\nu_{\omega}$
$$
\omega_ {k + 1} = \arg \min  _ {\omega} \frac {1}{| \mathcal {D} _ {k} | T} \sum_ {\tau \in \mathcal {D} _ {k}} \sum_ {t = 0} ^ {T} \sum_ {j = 0} ^ {t - 1} \sum_ {k = t + 1} ^ {T} \left(V _ {\phi_ {k}} \left(s _ {t}, g _ {k}\right) - \mathcal {V} _ {\omega} \left(s _ {j}, g _ {t}, g _ {k}\right)\right) ^ {2}
$$
where  $g_{k} = L(s_{k}), g_{t} = L(s_{t})$ , and  $s_0$  is the initial state of  $\tau$
$\triangleright$  Update the value function  $V_{\phi}$
$$
\phi_ {k + 1} = \arg \min _ {\phi} \frac {1}{| \mathcal {D} _ {k} | T} \sum_ {\tau \in \mathcal {D} _ {k}} \sum_ {t = 0} ^ {T} \left(V _ {\phi} \left(s _ {t}, \tau_ {g}\right) - \hat {R} _ {t}\right) ^ {2}
$$
involves organizing past goals into a graph structure  $G$  based on their reachability to improve agent exploration. This graph represents connections between goals, with nodes representing past goals and edges indicating that one goal can be reached from another. Our algorithm explicitly uses the goal value function  $\mathcal{V}(s_0, g_1, g_2)$  (which is learned together with the policy) to measure the capability of reaching a goal node  $g_2$  from a goal node  $g_1$  from the viewpoint at an initial state  $s_0$ .
Assume  $L$  is a goal-labeling function from state space  $S$  to goal space  $\mathcal{G}$ . Algorithm 2 works as follows. (1) Its samples a goal from a goal space  $\mathcal{G}$  and an initial state from a state space  $S$ . (2) It collects a trajectory  $\tau = (s_0, a_0, s_1, \ldots, s_t)$  to a goal node  $g'$  on  $G$  that is the closest node to  $g$  (distances between goal nodes are measured by  $\mathcal{V}$ ) until either  $L(s_t) = g'$  or  $t$  exceeds the maximum horizon. During this step, at each current state  $s$ , it finds the shortest past  $\rho$  from  $L(s)$  to  $g'$ , where  $\rho = L(s), g_{i+1}, g_{i+2}, \ldots, g'$  and then calling the policy  $\pi(\cdot | s, g_{i+1})$  to reach the goal  $g_{i+1}$ . (3) It executes the current policy  $\pi$  for  $T - t$  steps in the environment to collect a trajectory  $\tau'$  to reach the sampled goal  $g$ . (4) It concatenates  $\tau$  and  $\tau'$  and relabels the trajectory to add new expert tuples  $(s_t, g = s_{t+h}, a_t, R^t)$  for  $t, h > 0, t + h \leq T$  to the training dataset. (5) It performs supervised learning on the entire dataset to update the goal value function  $\mathcal{V}$  and the policy  $\pi$  via maximum likelihood.
C Additional Related Work
C.1 Goal-conditioned Reinforcement Learning
Goal-conditioned Reinforcement Learning(GCRL) is a popular topic that has been studied for a long time in many previous works. To characterize Goal-Augmented MDP, Schaul et al. [2015] provides a straightforward method that extends the standard value function to a goal-conditioned value function,
Algorithm 2 GCSL with organizing past goals into a graph structure  $G$
Require: Initial goal value function  $\mathcal{V}(s_0,g_1,g_2)$  , initial policy  $\pi (\cdot |s,g)$    
Require: Maximal horizon  $T$  , discounted factor  $\gamma$  ,(unknown) environment transition relation  $\mathcal{P}$    
goal-labeling function  $L:S\to \mathcal{G}$  from state space  $S$  to goal space  $\mathcal{G}$    
Require: Goal graph  $G(\mathcal{G},E)$  with nodes in  $\mathcal{G}$  edges  $E$  , and edge weight  $W_{e}$  for  $e\in E$    
for  $k = 1,2,3\dots$  do Sample  $g\sim \mathcal{G}$ $s_0\sim S$ $g^{\prime} = \arg \min_{g^{\prime}\in V}\mathcal{V}_{k}(s_{0},g^{\prime},g)$  Log a trajectory  $\tau = (s_0,a_0,s_1,a_1,\dots ,s_t)$  until  $L(s_{t}) = g^{\prime}$  or  $t\geq T$  with  $s_{i + 1}\sim \mathcal{P}(s_i,a_i)$ $a_i\sim \pi_k(s_i,g_{i + 1})$  where  $g_{i + 1}$  is on the shortest path  $\rho$  from  $L(s_{i})$  to  $g^{\prime}$  in G w.r.t edge weight  $W_{g_1\rightarrow g_2} = \mathcal{V}_k(s_0,g_1,g_2)$  for any nodes  $g_{1},g_{2}\in \mathcal{G}$  , and  $\rho = L(s_i),g_{i + 1},g_{i + 2},\ldots ,g'$  Log trajectory  $\tau^{\prime} = (s_{t},a_{t},\ldots ,S_{T})$  where  $s_{i + 1}\sim \mathcal{P}(s_i,a_{i + 1})$ $a_{i + 1}\sim \pi_k(s_i,g)$ $\tau = \tau @\tau '$  D Concatenate  $\tau$  and  $\tau^\prime$  D Update goal graph  $G$  For each state  $s_i$  on  $\tau$  , add  $L(s_i)$  to vertices  $V$  , add  $(L(s_i),L(s_{i + 1}))$  to edges  $E$  of  $G$ $D_{\tau} = \{(s_t,g = s_{t + h},a_t,\hat{R}_t):t,h > 0,t + h\leq T\}$  with  $\hat{R}_t = \sum_{k = 0}^{h}\gamma^k r(s_{t + k},g,a_{t + k})$ $D_{k + 1} = D_{\tau}\cup D_k$ $\mathcal{V}_{k + 1} = \underset {\mathcal{V}}{\arg \min}\mathbb{E}_{(s_t,g_{-,}\hat{R}_t)\in \mathcal{D}_{k + 1}}(\mathcal{V}(s_0,L(s_t),g) - \hat{R}_t)^2$  D Update the goal value function   
 $\pi_{k + 1} = \arg \max_{\pi}\mathbb{E}_{(s_t,g,a_t, - )\in \mathcal{D}_{k + 1}}[\log_\pi (a_t|s_t,g)]$  D Update the policy   
end for
called universal value function approximation (UVFA). UVFA serves as a basis in many following research works. How to define an appropriate reward is one of the basic challenges in GCRL. Some work reshapes the reward with the distance between the achieved goal and the desired goal but without any knowledge of the environment, i.e. Trott et al. [2019]. However, the local optimal policy it produces cannot finish many of the tasks in different environments. To solve long-horizon tasks, many planning methods on the abstract level above the robot control level are proposed. Nasiriany et al. [2019] introduced a goal-conditioned value function to measure the reachability of the next sub-goal given the previous sub-goal by analyzing how close the goal-conditioned policy gets to the goal after a specific number of time steps. Eysenbach et al. [2019] broke apart the task of reaching a distant goal into a sequence of way-points selected by learning a distributional Q-value function which indicates how many steps the current state and goal are away from one another. For the data as images, Chane-Sane et al. [2021] imagines possible sub-goals in a self-supervised fashion and uses them to facilitate training. By embedding goals into a latent space that captures some notion of temporal distance between goals, Zhang et al. [2021] learns a set of latent landmarks scattered across the goal space to enable scalable planning.
More generally, motivated by the fact that human learns from failure experience, Andrychowicz et al. [2018] provided a famous approach called Hindsight Experience Replay (HER), which can relabel the desired goals in the replay buffer with any reached goal within the same trajectory. Moreover, HER mitigates the sparse reward issue. In addition, many works try to extend the idea of HER. For instance, Rauber et al. [2019] demonstrates how hindsight can be introduced to policy gradient methods and generalize this idea to a successful class of reinforcement learning algorithms; Drawn intuition from the observation that any trajectory is a successful trial for reaching its final state, Ghosh et al. [2021] proposed a self-imitation algorithm named goal-conditioned supervised learning (GCSL). To enhance the diversity of relabeled goals, Zhu et al. [2021] develop a model-based reinforcement learning approach with a new relabeling strategy that relabels the goals by looking into the future with a learned dynamics model.
C.2 Reinforcement Learning Meets Logic and Specifications
In recent years, instructing agents using LTL formulas has been studied in a wide range of research works (Aksaray et al. [2016], Littman et al. [2017], Sadigh et al. [2014], Hasanbeig et al. [2018], Cai et al. [2021], Hahn et al. [2018], Camacho et al. [2019]). Among these approaches, retrieving information from and handling Q values in reinforcement learning is the mainstream implementation.
However, these Q-learning based approaches are not suitable for complex environment with continuous action space, until it is extended by Hasanbeig et al. [2020]. The above methods typically analyze a given LTL specification and then formulate a (usually sparse) reward function correspondingly. Then it could be used by any RL algorithm to learn a policy. Specifically, Jothimurugan et al. [2020] designed and implemented a domain-specific language for specifications called SPECRL. Users may decompose a complex task with sequences, disjunctions and (or) conjunctions of simpler sub-tasks, and can define the safety properties. Based on the above information, they construct a finite state machine named task monitor that generates shaped rewards. Icarte et al. [2018] build a model based on automaton called reward machines (RM). The RMs are used to represent and decompose complex high-level LTL specifications, so that a curated RL algorithm called QRM may be applied to exploit the structure of the RMs. In their following work of Icarte et al. [2022], they propose a hierarchical RL algorithm called HRM that acquires policies using RMs. Besides research works focusing on classic single-agent RL environments, there are a few recent works on RL with logic specifications in multi-agent RL environments (Hammond et al. [2021]). Still, experiments show training RL agents using classic RL algorithms without considering the formulation and semantics of LTL may lead to sub-optimal myopic policies (Vaezipoor et al. [2021], Icarte et al. [2022]). To this end, policy sketches may be used in achieving the LTL formula, where a sequence of sub-tasks is relatively easier to solve than a complex task as a whole. Andreas et al. [2017] show that such sketches are useful when solving long-horizon tasks. Sun et al. [2020] and Zhao et al. [2021] show that injecting semantics to a programmatic agent can speed up learning for complex long-horizon tasks.
D Demonstrations of the LTL Tasks in Sec. 4
In this section, we provide a detailed explanation of the general format for each Linear Temporal Logic (LTL) task presented in the paper. Additionally, we present specific examples of trajectories that illustrate how our method addresses a diverse range of challenging tasks across various environments. These trajectories serve as concrete demonstrations of our method's effectiveness in tackling these tasks.
D.1 Avoidance Tasks
The avoidance task follows a general format represented as  $\neg a\mathbf{U}(b \wedge (\neg c\mathbf{U}d))$ , where  $a, b, c, d$  correspond to propositions or goals relevant to task completion in our context. In Figures 10a and 10b, we showcase two trajectories generated by our trained agent to solve a specific avoidance task:  $\neg y\mathbf{U}(j \wedge (\neg w\mathbf{U}r))$ . This task can be intuitively understood as a two-stage process: the agent needs to reach the "Jetblack" zone while avoiding the "Yellow" zone in the first stage, and subsequently reach the "Red" zone while avoiding the "White" zone in the second stage. Figures 10a and 10b serve as clear demonstrations of the agent's ability to reach the desired zones while never entering the prohibited zones.
D.2 Avoiding More Tasks
The avoiding more task follows a general format represented as  $\mathbf{F}(a\wedge \mathbf{X}(\neg b\mathbf{U}c))\wedge \mathbf{G}(\neg d)$ , where  $a,b,c,d$  are task propositions or goals in our context. Figures 10c and 10d illustrate trajectories generated by our trained agent to solve an avoiding more task:  $\mathbf{F}(j\wedge \mathbf{X}(\neg y\mathbf{U}r))\wedge \mathbf{G}(\neg w)$ . Similar to avoidance tasks, avoiding more tasks requires the agent to traverse a sequence of zones in two stages while avoiding specific zones. The distinction is that avoiding more tasks require the agent to avoid one additional zone in the second stage, making it more challenging. Informally, the example can be interpreted as follows: in the first stage, the agent needs to reach the "Jetblack" zone while avoiding the "White" zone; in the second stage, the agent must reach the "Red" zone while avoiding the "White" and "Yellow" zones. The trajectories showcased in Figures 10c and 10d demonstrate the ability of our trained agent to effectively navigate through the required zones, avoiding the designated zones as specified by the avoiding more task.
From Figures 10a and 10c, we can observe how the presence of additional zones to avoid affects the trajectories, given identical initial states and sequences of goals (Jetblack and Red). Specifically, in the second stage of the avoidance task  $\neg y\mathbf{U}(j\wedge (\neg w\mathbf{Ur}))$ , the agent is expected to reach the Red zone while visiting a Yellow zone is not prohibited. However, in the second stage of the avoid more task  $\mathbf{F}(j\wedge \mathbf{X}(\neg y\mathbf{Ur}))\wedge \mathbf{G}(\neg w)$ , it is not allowed to visit any Yellow zone. In order to avoid the Yellow
(d)
Figure 10: Figures (a) and (b) depict trajectories that solve the avoidance task:  $\neg y\mathbf{U}(j \wedge (\neg w\mathbf{U}r))$ . On the other hand, Figures (c) and (d) showcase trajectories that solve the avoiding more task:  $\mathbf{F}(j \wedge \mathbf{X}(\neg y\mathbf{U}r)) \wedge \mathbf{G}(\neg w)$ . The initial positions of the robot are represented by orange diamonds  $\diamond$ . The different colors of the zones, namely Red, Yellow, White, and Jetblack, correspond to the task propositions  $r, y, w, j$ , respectively. It is important to note that the initial positions of the robot and the zones are randomized. The initial states of Figures (a) and (c) are the same, while the initial states of Figures (b) and (d) are identical.
zones in the second stage of the avoid more task, the agent takes additional steps (as shown in Figure 10c) to bypass a Yellow zone and eventually reach a Red zone. Figures 10b and 10d demonstrate that identical trajectories can solve different LTL tasks. This occurs when the extra zones to avoid are located far away from the robot throughout the entire execution, rendering no impact on the trajectories.
D.3 Goal Chaining Tasks.
The goal chaining task follows a general format expressed as  $\mathbf{F}(a\wedge \mathbf{F}(b\wedge \mathbf{F}(c\wedge \mathbf{F}d))$  , where  $a,b,c,d$  represent propositions or goals that are relevant to task completion in our context. Figures 11a and 11b display the trajectories for solving goal chaining tasks, specifically  $\mathbf{F}(j\wedge \mathbf{F}(w\wedge \mathbf{F}(r\wedge \mathbf{F}y)))$  . In this particular LTL task, the agent is required to navigate through zones with the colors Jetblack, White, Red, and Yellow in that specific order. The trajectories depicted in Figure 11 effectively demonstrate how our methods enable the robot to smoothly and efficiently reach the desired zones in the prescribed order. These trajectories showcase that our trained agent can efficiently adjust the robot's state after completing a sub-task and promptly proceed toward the next goal zone.
(b)
Figure 11: Figures (a) and (b) depict trajectories that solve the goal chaining task:  $\mathbf{F}(j\wedge \mathbf{F}(w\wedge \mathbf{F}(r\wedge \mathbf{F}y))$ . The initial positions of the robot are represented by orange diamonds. The different colors of the zones, namely Red, Yellow, White, and Jetblack, correspond to the task propositions  $r,y,w,j$  respectively. It is important to note that the initial positions of the robot and the zones are randomized.
E Demonstration on  $\omega$ -LTL Tasks
E.1  $\omega$ -Loop Tasks
An  $\omega$ -loop task, expressed in the form  $\mathbf{GF}(a \wedge \mathbf{XFb}) \wedge \mathbf{G}(\neg c)$ , intuitively requires the agent to continuously loop between two specific zones,  $a$  and  $b$ , while always avoiding another zone,  $c$ . For example, in the task  $\mathbf{GF}(r \wedge \mathbf{XFy}) \wedge \mathbf{G}(\neg w)$ , the agent is expected to traverse between the Red and Yellow zones indefinitely, while never visiting any White zone. Figures 12a and 12b illustrate trajectories that solve this  $\omega$ -LTL task. In both trajectories, the robot quickly reaches the Red zone and subsequently oscillates between the Red and Yellow zones. It is important to note that the LTL formula only requires the avoidance of White zones. Therefore, the trajectory shown in Figure 12b, which visits a Jetblack zone, is still valid. Theoretically, to satisfy an  $\omega$ -LTL task, our trained agent should run indefinitely if not falsified. However, in Figure 12, we only plot trajectories for the first 1500 steps to demonstrate that our trained agent is capable of solving  $\omega$ -loop tasks.
E.2  $\omega$ -Stability Tasks
An  $\omega$ -stability task requires the agent to satisfy a task proposition by reaching a goal region and remaining in the goal region infinitely. Figures 13a and 13b depict two trajectories that (approximately) satisfy the stability task:  $\mathbf{FG}y$ . In both figures, the agent can eventually reach a desired Yellow zone and then stabilize at that zone. Since the stability task runs infinitely, we report the first 1500 steps to demonstrate that our agent is capable of stabilizing at zones of the specified color.
F Demonstrations of the LTL Tasks on Ant 16-room Environments