SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct experiments to assess the effectiveness of our approach to navigating an Ant through a challenging environment consisting of 16 rooms separated by thick walls. In this environment, each room has the same size  $8 \times 8$  divided by walls and corridors with thickness 1 (Fig. 16). There are two obstacles denoted by black squares in the environment. We place a Mujoco (Todorov et al. [2012]) Ant robot in this environment for navigation. To solve the task, the ant should depart from the center of the bottom-left room to reach the desired goal positions. The initial state of the ant is created with random noise, which can keep the position of the ant around the center of the bottom-left room.
Here an atomic proposition  $g \in \mathcal{G}$  is in the form of  $(r, c)$  denoting the room in the  $r$ -th row and  $c$ -th column. The bottom-left corner is room  $(0, 0)$ . Given an environment state  $s$ , define  $s \vDash (r, c)$  is true if the position of the Mujoco ant in  $s$  is close to the center of the  $(r, c)$ -th room within a threshold.
Figure 12: Figures (a) and (b) illustrate trajectories that solve the  $\omega$ -loop task:  $\mathbf{GF}(r \wedge \mathbf{XF}y) \wedge \mathbf{G}(\neg w)$ . The initial positions of the robot are represented by orange diamonds  $\diamond$ . The different colors of the zones, namely Red, Yellow, White, and Jetblack, correspond to the task propositions  $r, y, w, j$ , respectively. It is important to note that the initial positions of the robot and the zones are randomized.
(a)
(b)
Figure 13: Figures (a) and (b) illustrate trajectories that solve the  $\omega$ -stability task: FGy. The initial positions of the robot are represented by orange diamonds  $\diamond$ . The different colors of the zones, namely Red, Yellow, White, and Jetblack, correspond to the task propositions  $r, y, w, j$ , respectively. It is important to note that the initial positions of the robot and the zones are randomized.
F.1 Tasks: LTL Specifications
We evaluate our approach using five LTL specifications taken from DiRL Jothimurugan et al. [2021] denoted as  $\phi_{1}$  to  $\phi_{5}$ . These tasks have increasing levels of difficulty as they require the agent to sequentially reach a growing number of sub-goals. In order to reach each sub-goal, the ant is allowed a maximum of 1000 steps to move within the environment. In Figure 16, the sub-goals are represented by colors, starting from blue and progressing to orange, yellow, green, and purple for  $\phi_{1}$  to  $\phi_{5}$  respectively. The agent has the option to choose between two possible paths when transitioning from one sub-goal to the next. For instance, in the most complex specification, the agent must depart from the blue point and sequentially reach the sub-goals at orange, yellow, green, and finally purple. The five specifications are described below:
$$
\bullet \phi_ {1} := \mathbf {F} ((0, 2) \vee (2, 0)).
$$
It corresponds to reaching the center of either room (0,2) or room (2,0) in the Ant16rooms environment.
$\phi_2 \coloneqq \mathbf{F}((0, 2) \vee (2, 0)) \wedge \mathbf{F}(2, 2)).$
After completing the task specified by  $\phi_{1}$ , the ant should then proceed to reach the center of room (2,2) in the Ant16rooms environment.
-  $\phi_3 \coloneqq \mathbf{F}((0,2) \vee (2,0)) \wedge \mathbf{F}((2,2) \wedge \mathbf{F}(((2,1) \vee (3,2)) \wedge \mathbf{F}(3,1)))$ .
After completing the task specified by  $\phi_{2}$ , the ant should then proceed to reach the center of room (3,1) in the Ant16rooms environment. The ant has the option to go through the center of either room (2,1) or room (3,2) on its way to the target location.
$\phi_4\coloneqq \mathbf{F}((0,2)\vee (2,0))\wedge \mathbf{F}((2,2)\wedge \mathbf{F}((2,1)\vee (3,2))\wedge \mathbf{F}((3,1)\wedge \mathbf{F}(((1,1)\vee (3,3))\wedge$ $\mathbf{F}(1,3)))))))$
After successfully completing the task defined by  $\phi_3$ , the ant's next objective is to reach the center of room (1,3) in the Ant16rooms environment. The ant has the choice to pass through the center of either room (1,1) or room (3,3) while making its way to the designated location.
-  $\phi_5 := \mathbf{F}(((0,2) \vee (2,0)) \wedge \mathbf{F}((2,2) \wedge \mathbf{F}(((2,1) \vee (3,2))) \wedge \mathbf{F}((3,1) \wedge \mathbf{F}(((1,1) \vee (3,3))) \wedge \mathbf{F}((1,3) \wedge \mathbf{F}(((1,1) \vee (0,3)) \wedge \mathbf{F}(0,1))))$ ).
Upon completing the task specified by  $\phi_4$ , the ant's subsequent goal is to reach the center of room (0,1) in the Ant16rooms environment. The ant has the option to traverse through the center of either room (1,1) or room (0,3) as it moves towards the target location.
Additionally, to demonstrate the generalization capability of our approach, we incorporate two additional specifications, namely  $\phi_6$  and  $\phi_7$ , which were not included in the original study by DiRL (Jothimurugan et al. [2021]). Furthermore, we introduce an  $\omega$ -regular LTL specification, denoted as  $\phi_8$ , which is not supported by DiRL.
-  $\phi_6 := \mathbf{F}(((2,0) \vee (0,1)) \wedge \mathbf{F}((2,1) \wedge \mathbf{F}(((2,2) \vee (1,1)) \wedge \mathbf{F}((1,2) \wedge \mathbf{F}(((3,2) \vee (1,3)) \wedge \mathbf{F}(3,3))))$ ).
It corresponds to the following sequence of sub tasks: First, reaching the center of room (2,1) by passing through either room (2,0) or room (0,1). Next, reaching the center of room (1,2) by passing through either room (2,1) or room (1,1). Finally, reaching the center of room (3,3) by passing through either room (3,2) or room (1,3).
$\phi_7\coloneqq \mathbf{F}(((2,0)\lor (0,2))\land \mathbf{F}((2,2)\land \mathbf{F}(((2,1)\lor (1,2))\land \mathbf{F}((1,1)\land \mathbf{F}(((0,1)\lor (1,3))\land$ $\mathbf{F}(0,3))))$
It corresponds to the following sequence of sub tasks: First, reaching the center of room (2,2) by passing through either room (2,0) or room (0,2). Next, reaching the center of room (1,1) by passing through either room (2,1) or room (1,2). Finally, reaching the center of room (3,3) by passing through either room (3,1) or room (1,3).
-  $\phi_8 \coloneqq \mathbf{F}((2,0) \vee (0,2)) \wedge \mathbf{G}\mathbf{F}((2,2) \wedge \mathbf{X}(\mathbf{F}((2,1) \wedge \mathbf{X}\mathbf{F}(1,1))))$ .
It corresponds to the following sequence of sub tasks: First, reaching the center of room (2,2) by passing through either room (2,0) or room (0,2). Then, taking an infinite loop from room  $(2,2)\rightarrow$  room  $(2,1)\rightarrow$  room  $(1,1)\rightarrow$  room  $(1,2)\rightarrow$  room  $(2,2)\dots$
F.2 Results.
We also compare our approach with DiRL (Jothimurugan et al. [2021]), which is a state-of-the-art LTL-satisfying RL algorithm. Similar to ours, DiRL leverages the compositional structure of a task specification to enable learning. However, there are some differences between our approach and DiRL. In DiRL, a unique low-level policy is learned for each sub-goal transition, and a high-level policy plans over the sub-goal transition space by evaluating the quality of the various low-level policies. One limitation of DiRL is that it is restricted to a small fragment of LTL. Particularly, it cannot handle  $\omega$ -regular LTL properties that involve infinite state sequences, as it is not possible to learn a unique low-level policy for each transition in an infinite sequence. Additionally, DiRL is not applicable to multi-task RL scenarios because the low-level policies are specific to a single environment setting and do not generalize across different environments.
Methods	DiRL	DiRL+ GCSL	Ours	Ablation
φ1	0.910 (0.022)	0.923 (0.082)	0.967 (0.006)	0.722 (0.073)
φ2	0.770 (0.083)	0.953 (0.037)	0.925 (0.049)	0.713 (0.046)
φ3	0.367 (0.147)	0.967 (0.017)	0.935 (0.028)	0.667 (0.039)
φ4	0.183 (0.046)	0.937 (0.031)	0.875 (0.041)	0.593 (0.086)
φ5	0.043 (0.061)	0.913 (0.017)	0.868 (0.038)	0.533 (0.083)
φ6	/	/	0.857 (0.004)	0.565 (0.045)
φ7	/	/	0.882 (0.018)	0.617 (0.043)
φ8	/	/	0.903 (0.045)	0.655 (0.039)
Table 2: The success rate of the LTL specifications  $\phi_{1}$  to  $\phi_{8}$  on Ant16rooms. The success rates shown are the average values obtained from three individual training experiments (for each training experiment, we evaluate the final trained agent against a specification 200 times). Since DiRL and DiRL+GCSL cannot be applied to multi-task RL, we train a separate agent for each of the LTL specifications  $\phi_{1}$  to  $\phi_{5}$  using 3e6 environment steps. In contrast, our algorithm trains a single goal-conditioned agent using Algorithm 2 via 3e6 environment steps and evaluates this agent over all eight LTL specifications.
In our experiment, we consider both DiRL and DiRL+GCSL as our baselines. DiRL uses the Augmented Random Search (ARS) algorithm (Mania et al. [2018]) to learn the low-level policy for each sub-goal transition. DiRL+GCSL, on the other hand, replaces the ARS learning algorithm with our extended GCSL algorithm (shown in Algorithm 2) with a fixed sub-goal for learning each low-level sub-goal transition policy. This allows for a fair comparison as our method is based on this learning algorithm. Since DiRL and DiRL+GCSL cannot be applied to multi-task RL, we train a separate agent for each of the LTL specifications  $\phi_{1}$  to  $\phi_{5}$  using 3e6 environment steps. In contrast, our algorithm trains a single goal-conditioned agent using Algorithm 2 via 3e6 environment steps and evaluates this agent over all five LTL specifications.
Table 2 presents the success rates of the baseline approaches DiRL, DiRL+GCSL, and Our approach on all the LTL specifications. The success rates shown are the average values obtained from three individual training experiments (for each training experiment, we evaluate the final trained agent against a specification 200 times). As the complexity of the specifications increases, the performance of DiRL gradually deteriorates, while the other baseline approach, DiRL+GCSL, maintains a stable success rate across all LTL specifications (from  $\phi_{1}$  to  $\phi_{5}$ ). Our goal-conditioned agent achieves comparable performance to DiRL+GCSL, despite using significantly fewer samples to solve these LTL tasks (see the explanation above). In addition, our approach demonstrates immediate high success rates on the two additional LTL specifications,  $\phi_{6}$  and  $\phi_{7}$ , without the need for additional training. In contrast, the DiRL baselines require retraining from scratch for these additional tasks. This showcases the capability of our technique to use a simple goal-conditioned RL agent to follow arbitrary LTL specifications without additional training over the LTL task space. Unlike the DiRL baselines, which are designed for LTL specifications expressible as regular expressions, our technique is not limited to regular expressions and can handle more complex  $\omega$ -regular expressions. This is demonstrated by the success rate of our agents on the  $\omega$  (loop) specification  $\phi_{8}$ , where the agent is evaluated based on its ability to complete the loop twice. To further illustrate the performance of our approach, we provide examples of evaluation rollouts for specifications ranging from  $\phi_{1}$  to  $\phi_{8}$  in Figure 14.
F.3 Ablation Study
Random path selection. Recall that our algorithm uses the goal value function  $\mathcal{V}(s_0, g_1, g_2)$  to measure the capability of reaching goal  $g_2$  from goal  $g_1$  from the viewpoint of the agent at an initial state  $s_0$ . This information is crucial for performing path planning over the graph representation of a Büchi automaton, enabling the optimal execution of the sequence of tasks defined in an LTL instruction. In this ablation study, we investigate the importance of path planning in our algorithm. We create an ablated version of our approach where acceptable paths are randomly generated from the Büchi automaton of an LTL specification, and the goal-conditioned agent executes towards the goals along these randomly selected paths. The success rates for the ablated version of our algorithm across all specifications are presented in the last column of Table 2, and the example trajectories of the agent for  $\phi_1$  to  $\phi_8$  in this ablated setting are depicted in Figure 15. Our experiment reveals that without the path planning guided by the goal value function  $\mathcal{V}$ , the performance in satisfying specifications  $\phi_1$  to
(a) Example of  $\phi_1$
(b) Example of  $\phi_2$
(c) Example of  $\phi_3$
(d) Example of  $\phi_4$
(g) Example of  $\phi_7$
Figure 14: Figures (a) to (e) depict examples of agent trajectories for each specification from  $\phi_{1}$  to  $\phi_{8}$ . In these figures, the  $\star$  symbol represents the final goal position required by the respective specifications. Notably, for  $\phi_{8}$ , which is an  $\omega$ -regular LTL specification involving infinite state sequences, there is no final goal for this specification.
(e) Example of  $\phi_5$
(f) Example of  $\phi_6$
(h) Example of  $\phi_8$
$\phi_8$  significantly deteriorates. This decline can be attributed to the randomly selected paths sometimes crossing obstacles in certain rooms, making it challenging for the Mujoco ant to navigate through them. By comparing Figure 14 (which showcases our full algorithm with path planning) with Figure 15, we observe that our algorithm's ability to select obstacle-free paths (thanks to the goal value function  $\mathcal{V}$ ) significantly improves the agent's capacity to satisfy the given specifications.
G.1 Environment Details
ZoneEnv. The ZoneEnv environment is derived from OpenAI's Safety Gym (Ray et al. [2019]), and it features a continuous action and observation space. The square-shaped walled environment
(a) Example of  $\phi_1$  with random sequence of sub-goals
(b) Example of  $\phi_{2}$  with random sequence of sub-goals
(c) Example of  $\phi_3$  with random sequence of sub-goals
(d) Example of  $\phi_4$  with random sequence of sub-goals
(g) Example of  $\phi_7$  with random sequence of sub-goals
Figure 15: The examples of failed trajectories of each specification for the abated version of our algorithm. In these figures, the  $\star$  symbol represents the final goal position required by the respective specifications. Randomly selected paths sometimes cross obstacles in certain rooms, making it challenging for the Mujoco ant to navigate through them.
(e) Example of  $\phi_5$  with random sequence of sub-goals
(f) Example of  $\phi_6$  with random sequence of sub-goals
(h) Example of  $\phi_8$  with random sequence of sub-goals
has 8 zones (2 of each color). The colors correspond to the task propositions. We use a simple robot from Safety Gym called Point, with one actuator for turning and another for moving forward or backward. An agent can observe the LiDAR information of its surrounding zones. Given this indirect geographical information, it has to visit and/or avoid certain zones to satisfy sampled LTL task specifications. To acquire agents to generalize to any task, the initial positions of zones and the robot are random in every episode. Within this environment, the robot has the ability to perceive its own state, including factors such as velocity, acceleration, and LiDAR information. The observation comprises four distinct groups of LiDAR information, each corresponding to a specific color.
To capture LiDAR information for each color, the environment employs a set of 16 hypothetical light bins emitted from the robot. These bins are evenly distributed, with intervals of 30 degrees between them. Each bin is capable of detecting the presence of a desired color zone in its direction and returns a normalized distance value indicating the proximity of the zone. The returned distance values range
from 0 to 1, with lower values indicating closer zones, 0 denoting overlap between the agent and a specific color zone, satisfying the corresponding proposition, and 1 indicating the absence of a desired zone in that direction. In cases where a bin can intersect with two zones simultaneously, the returned value is determined by the zone that is closer to the robot. The state representation in the ZoneEnv environment employs a 76-dimensional vector, consisting of 12 dimensions for the robot and 64 dimensions for the LiDAR information (16 dimensions per color).
In this environment, the goal-labeling function  $L$  acts as an event detector, providing a truth assignment for all propositions related to the four colors  $\mathcal{G} = j, w, r, y$ . The goal-labeling function triggers when the propositions in  $\mathcal{G}$  hold true in the environment. For example,  $r \in L(s)$  if and only if the agent is located on top of a red zone at a specific environment state  $s$ .
Ant-16rooms. This environment with continuous observation and action space is adapted from the 16 rooms environment from Jothimurugan et al. [2021]. In this walled environment with 16 rooms, each room has the same size  $8 \times 8$  divided by walls and corridors with thickness 1 (Fig. 16). There are two obstacles denoted by black squares in the environment. We place a Mujoco (Todorov et al. [2012]) Ant robot in this environment for navigation. To solve the task, the ant should depart from the center of the bottom-left room to reach the desired goal positions. The initial state of the ant is created with random noise, which can keep the position of the ant around the center of the bottom-left room.
Figure 16: Ant 16 rooms
LetterEnv. The LetterEnv is a grid-based environment with a size of  $7 \times 7$ , featuring discrete action and observation spaces. Within this environment, the agent has a complete view of the grid, including its own position. Specifically, for each type of letter, there is a  $7 \times 7$  matrix that contains multiple integers. A value of 0 in a grid cell indicates that the letter is not present at that position, while a value of 1 denotes the presence of a letter instance. In addition to these 12 matrices representing the letters, there is an additional matrix indicating the position of the agent. By considering both the letter positions and the agent's position, the goal-labeling function  $L$  can determine which task propositions are satisfied or unsatisfied at each step.
G.2 Primitive Policies Details
Dynamic Primitives of ZoneEnv. In order to obtain a discrete action space for the Point robot in Safety Gym (Ray et al. [2019]), we trained a set of four dynamic primitives in the ZoneEnv environment using reinforcement learning. Each dynamic primitive was trained for 0.5 million steps. These four dynamic primitives are responsible for guiding the robot to move in the cardinal directions of UP, DOWN, LEFT, and RIGHT respectively. The reward function used to train these primitives is defined as follows:
$$
r _ {\text {z o n e}} = c _ {v} \cdot v _ {\text {d i r e c t i o n}} - c _ {\text {p e r}} \cdot \left| v _ {\text {p e r}} \right| \tag {7}
$$
where  $v_{\text{direction}}$  is the velocity of a given direction, and  $v_{per}$  denotes the velocity of the direction that is perpendicular to the given direction. In order to train the  $UP$  and  $DOWN$  primitives,  $v_{\text{direction}} = \pm v_x$  tracks velocities on  $\pm x$ -axis; similarly, to train the  $LEFT$  and  $RIGHT$  policies, we set  $v_{\text{direction}} = \pm v_y$  to keep track of velocities along  $\pm y$ -axis. Coefficients  $c_v, c_{per}$  are set to be 1 and 0.05, respectively. All dynamic primitives for the ZoneEnv environment were trained using the PPO algorithm (Schulman et al. [2017]) implemented in the Stable-Baselines3 RL framework (Raffin et al. [2021]).
Dynamic Primitives of Ant. For the MuJoCo ant, we follow Qiu and Zhu [2022] and train a set of four basic primitive policies, i.e.,  $UP$ ,  $DOWN$ ,  $LEFT$  and  $RIGHT$  for 1 million steps for each. The reward function to acquire the above four primitives is defined as:
$$
r _ {a n t} = c _ {v} \cdot v _ {\text {d i r e c t i o n}} + c _ {h} \cdot I (\text {I s H e a l t h y}) - c _ {a} \cdot \| a \| _ {2} - c _ {f} \cdot \| f _ {\text {c o n t a c t}} \| _ {2} \tag {8}
$$
where  $v_{\text{direction}}$  is the velocity of a given direction,  $I(\text{IsHealthy})$  is a Boolean function that determines the health condition of the agent, action  $a$  is an 8-dimension vector, and  $f_{\text{contact}}$  is a vector that encodes the contact force of the agent. In order to train the UP and DOWN primitives,  $v_{\text{direction}} = \pm v_x$  tracks velocities on  $\pm x$ -axis; similarly, to train LEFT and RIGHT policies, we set  $v_{\text{direction}} = \pm v_y$  to keep track of velocities along  $\pm y$ -axis. Coefficients  $c_v, c_h, c_a$  and  $c_f$  are set to be 1, 1, 0.5,  $5 \times 10^{-4}$ , respectively. All dynamic primitives for the MuJoCo ant are trained using the Soft Actor-Critic algorithm (Haarnoja et al. [2018]) implemented in the OpenAI Spinning Up RL framework (Achiam [2018]).
Training a Goal-Conditioned agent for ZoneEnv. We employ the goal-conditioned Proximal Policy Optimization (PPO) algorithm, as shown in Algorithm 1, to train a goal-conditioned agent that can satisfy all task propositions (goals) in the ZoneEnv environment, which involves reaching zones of different colors. For each color (Yellow, Red, White, and Jetblack), we use a random vector of 24 dimensions as the goal representation. During training, we sample the goal colors and correspondingly adapt the reward function and the goal representation of the ZoneEnv environment. It is crucial to ensure the synchronization of these elements within each episode. For instance, when instructing the agent to reach a Red zone, we use the reward function to detect the  $r$  proposition and exclusively utilize the appropriate goal representation for Red. To achieve a balanced agent that can successfully reach zones of all colors with equal rates, we sample the goal colors uniformly. The reward function for acquiring the goal-conditioned agent is defined as follows:
$$
r _ {\text {c o l o r}} = \left\{ \begin{array}{l l} 1, & \text {s a t i s f i e d} \\ 0, & \text {u n s a t i s f i e d} \end{array} \right. \tag {9}
$$
where color denotes a goal (i.e., one of  $r, y, w, j$ ) that the agent must reach in an episode. We employ a sparse reward function that only provides a reward of 1 when the desired color is satisfied, while all other situations receive a reward of zero. We do not penalize the agent for stepping into zones of different colors. Our focus is solely on whether the agent can effectively and efficiently reach the desired color.
Training a Goal-Conditioned agent for LetterEnv. We utilize the goal-conditioned Proximal Policy Optimization (PPO) algorithm, as shown in Algorithm 1, to train a goal-conditioned agent capable of satisfying all task propositions (i.e., reaching all letters) in the LetterEnv environment. To represent each letter as a goal, we employ a random matrix with a size of  $7 \times 7$ . The training process for the goal-conditioned agent follows the same methodology as in the ZoneEnv environment to ensure synchronization among goals, reward functions, and goal representations in each episode. The reward function for acquiring the goal-reaching primitive is defined as follows:
$$
r _ {\text {l e t t e r}} = \left\{ \begin{array}{l l} 1, & \text {s a t i s f i e d} \\ 0, & \text {u n s a t i s f i e d} \end{array} \right. \tag {10}
$$
where letter denotes one of the goal letter that the agent must reach. We employ a sparse reward function that provides a reward of 1 if the desired goal is achieved and 0 otherwise.
Training a Goal-Conditioned agent for Ant 16 rooms. We apply the goal-conditioned supervised learning algorithm described in Algorithm 2 to train a goal-conditioned agent that can navigate in the Ant16rooms environment. The reward function is defined as follows:
$$
r _ {g} = \left\{ \begin{array}{l l} 1, & \text {g o a l r e a c h e d} \\ 0, & \text {g o a l n o t r e a c h e d} \end{array} \right. \tag {11}
$$
where  $g$  is a randomly sampled goal in the goal space.
G.3 Handling Transitions