SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenjie Qiu*
wq37@cs.rutgers.edu
Wensen Mao*
wm300@cs.rutgers.edu
He Zhu
hz375@cs.rutgers.edu
Goal-conditioned reinforcement learning (RL) is a powerful approach for learning general-purpose skills by reaching diverse goals. However, it has limitations when it comes to task-conditioned policies, where goals are specified by temporally extended instructions written in the Linear Temporal Logic (LTL) formal language. Existing approaches for finding LTL-satisfying policies rely on sampling a large set of LTL instructions during training to adapt to unseen tasks at inference time. However, these approaches do not guarantee generalization to out-of-distribution LTL objectives, which may have increased complexity. In this paper, we propose a novel approach to address this challenge. We show that simple goal-conditioned RL agents can be instructed to follow arbitrary LTL specifications without additional training over the LTL task space. Unlike existing approaches that focus on LTL specifications expressible as regular expressions, our technique is unrestricted and generalizes to  $\omega$ -regular expressions. Experiment results demonstrate the effectiveness of our approach in adapting goal-conditioned RL agents to satisfy complex temporal logic task specifications zero-shot.
Goal-conditioned learning Liu et al. [2022] is a type of reinforcement learning (RL) task where an agent learns to achieve a specific goal in a given environment. The goal can be defined in various ways, such as a particular state of the environment or a desired outcome, and the agent learns to map its current observations to actions that bring it closer to the goal. By providing a high-level goal, the agent can effectively learn to generalize its behavior and adapt to changing conditions.
However, goal-conditioned RL agents struggle with generalizing to more complex, task-oriented policies when the goals are specified by temporally extended instructions typically written in Linear Temporal Logic (LTL) languages, which creates a much larger task space than simple goals. In such cases, as the LTL task space is vast and difficult to exhaustively sample, it becomes more challenging for the agent to identify the correct sequence of actions needed to accomplish a task, which results in slow learning and poor generalization.
We propose a novel approach that enables simple goal-conditioned RL agents to follow arbitrary LTL specifications without any additional training over the LTL task space. This is in sharp contrast to existing approaches for finding LTL-satisfying policies, which require sampling a large set of LTL instructions from the task space during training as "goals" Vaezipoor et al. [2021], Kuo et al. [2020]. Such methods have no guarantee in generalizing to unseen tasks outside their training distribution of LTL objectives. As an example, the capability of most existing learning algorithms for LTL-satisfying policies is restricted to specifications over regular expressions that can be modeled by reward machines Icarte et al. [2022, 2018]. Our technique can handle  $\omega$ -regular LTL specifications
* Wenjie Qiu and Wensen Mao contributed equally to this work.
over infinite control sequences even though the underlying goal-conditioned agents have never seen such specifications during training.
Our main contribution lies in a policy learning algorithm for LTL satisfaction, which decouples complex low-level environment interaction from high-level task planning. Specifically, we leverage a goal-conditioned RL agent to interact with low-level environments to learn how to achieve basic goals without considering higher-level tasks. The goal-conditioned RL agent has the capability of measuring the difficulty of transitioning between goals in the goal space. To solve an unseen LTL instruction  $\varphi$ , our method applies a weighted graph search algorithm that engages in high-level planning over the difficulty of achieving the specified goals to choose the optimal sequence of sub-goals in  $\varphi$  for the goal-conditioned agent to reach. We present experimental results demonstrating the effectiveness of our technique in adapting goal-conditioned RL agents to satisfy complex LTL task specifications of varying complexity zero-shot.
Goal-Augmented Markov Decision Processes (MDPs) extend standard MDPs by incorporating a set of goals  $\mathcal{G}$  within the system state space  $S$ . We assume the existence of a state-goal labeling function  $L: S \to 2^{\mathcal{G}}$  that maps each state to a set of atomic propositional symbols used to describe the observation of a system state where an atomic propositional symbol  $g \in \mathcal{G}$  is a variable that takes on a truth. Throughout the paper, the terms "goals" and "propositions" are used interchangeably.
As an example, consider ZoneEnv, a Safety Gym environment Achiam and Amodei [2019] adapted from Vaezipoor et al. [2021]. The environment (Fig. 1) is a walled 2D plane with colored zones that correspond to task propositions. Both the zones and the point robot are randomly positioned on the plane. In this example, we use a set of atomic propositions  $\mathcal{G} = \{y, w, r, j\}$  that represents
the four colors yellow, white, red, and jet black.
Figure 1: An LTL Task for ZoneEnv.
Formally, we model a goal-augmented MDP as a tuple  $(S,A,\mathcal{G},L,T,R,\gamma)$  that consists of a state space  $S$ , an action space  $A$ , a goal space defined by a set of atomic propositions  $\mathcal{G}$ , a transition function  $T:S\times A\times S\to [0,1]$  that maps each state-action-state transition to a probability, a labeling function  $L$ , and a reward function defined with the goals  $R:S\times \mathcal{G}\times A\to \mathbb{R}$  that maps each state-action transition to a scalar reward for a particular goal in  $\mathcal{G}$ .  $\gamma \in [0,1)$  is a discount factor that controls the relative importance of immediate versus future rewards.
In a goal-augmented MDP, the agent's objective is to reach a goal via a goal-conditioned policy  $\pi : S \times G \times A \to [0,1]$  that maximizes the expectation of the cumulative reward:  $J(\pi) = \mathbb{E}_{\substack{a_t \sim \pi(\cdot | s_t, g), g \sim P_g \\ s_{t+1} \sim T(\cdot | s_t, a_t)}} \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, g_t, a_t) \right]$  where  $p_g$  be a distribution over goals in  $G$ . The optimal policy  $\pi^* = \arg \max_{\pi} J(\pi)$  maximizes the expected cumulative return over all possible goals, weighted by their probability under the goal distribution  $P_g$ . This formulation encourages the policy to be effective at achieving a diverse set of goals, rather than simply optimizing for a single goal. For example, in the colored zone environment in Fig. 1 (left), a goal-conditioned agent policy  $\pi(a|s, g)$  can be trained to reach a specific colored zone  $g \in \mathcal{G} = \{y, w, r, j\}$ .
The value function for goal-conditioned reinforcement learning  $V^{\pi}(s_t, g)$  returns the expected cumulative discounted reward obtained by following policy  $\pi$  starting from state  $s_t$  with the goal  $g$  as the target:  $V^{\pi}(s_t, g) = \mathbb{E}_{\substack{a_t \sim \pi(\cdot|s_t, g), \\ s_{t+1} \sim T(\cdot|s_t, a_t)}} \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, g, a_t) \right]$  The goal-conditioned Q-function
$Q^{\pi}(s_t,g,a)$  returns the expected cumulative discounted reward obtained by taking action  $a$  in state  $s_t$ , and then following policy  $\pi$  thereafter, with the goal  $g$  as the target:  $Q^{\pi}(s_t,g,a) = r(s_t,g,a) + \gamma \sum_{s_{t+1} \in S} T(s_{t+1}|s_t,a)V^{\pi}(s_{t+1},g)$ . The goal-conditioned  $Q$  and  $V$  functions may be learned using model-free RL algorithms such as  $Q$ -learning or policy gradient methods. In training, we assume goals can be uniformly sampled. We also assume that  $t_{max}$  is the maximum horizon and a goal
conditioned agent receives a positive reward signal only when it successfully achieves the specified goal by the end of a training episode by framing a reward function:
$$
r \left(s _ {t}, g, a _ {t}\right) = \mathbb {1} [ t = t _ {\max } ] \cdot \mathbb {1} [ g \in L \left(s _ {t}\right) ] \tag {1}
$$
where  $\mathbb{1}$  is the indicator function. This reward function has the following interpretation: given a state  $s_t$ , will the goal-conditioned policy  $\pi$  get to a goal  $g$  after  $t$  time steps of attempting to reach  $g$ .
Linear Temporal Logic (LTL) is a formal language used to specify the temporal properties of a system. LTL formulas are built from a set of temporal operators, such as  $\mathbf{X}$  ("next"),  $\mathbf{F}$  ("eventually"),  $\mathbf{G}$  ("always"), and  $\mathbf{U}$  ("until"). LTL formulas are constructed over a finite set of atomic proposition symbols  $\mathcal{G}$ . Formally, the syntax of LTL formulas is defined recursively as follows:
$$
\varphi \because = g | \neg \varphi | \varphi \wedge \varphi | \varphi \vee \varphi | \mathbf {X} \varphi | \varphi \mathbf {U} \varphi | \mathbf {F} \varphi | \mathbf {G} \varphi \text {w h e r e} g \in \mathcal {G}
$$
Intuitively, the formula  $\mathbf{X}\varphi$  holds if  $\varphi$  holds at the next time step,  $\varphi_{1}\mathbf{U}\varphi_{2}$  holds if  $\varphi_{1}$  holds until  $\varphi_{2}$  holds,  $\mathbf{F}\varphi$  holds if  $\varphi$  holds at a future step, and  $\mathbf{G}\varphi$  holds if  $\varphi$  holds at the current and all future steps.
An LTL formula  $\varphi$  is said to be satisfied by a state transition system with state space  $S$  if and only if all infinite sequences of states of the system satisfy  $\varphi$ . We assume the existence of a labeling function  $L: S \to 2^{\mathcal{G}}$  that maps each state to the atomic propositions that are true in that state. Given an infinite sequence of system states  $\tau = \langle s_0, s_1, s_2, \ldots \rangle$  generated by a system,  $\tau \models \varphi$  models the satisfaction relation  $\vDash$  between an LTL formula  $\varphi$  and the sequence of system states  $\tau$ . Formally,  $\tau \models \varphi$  if and only if  $s_0 \models \varphi$  which. We can define  $s_i \models \varphi$  inductively, as illustrated on the right.
-  $s_i \models g$  if  $g \in \mathcal{G}$  is an atomic proposition and  $g \in L(s_i)$  holds.
-  $s_i \models \neg \varphi$  if  $s_i \nmodels \varphi$ .
-  $s_i \models \varphi_1 \wedge \varphi_2$  if  $s_i \models \varphi_1$  and  $s_i \models \varphi_2$ .
-  $s_i \models \varphi_1 \lor \varphi_2$  if  $s_i \models \varphi_1$  or  $s_i \models \varphi_2$ .
-  $s_i \models \mathbf{X}\varphi$  if  $s_{i+1} \models \varphi$ .
-  $s_i \models \mathbf{F}\varphi$  if there exists  $j \geq i$  such that  $s_j \models \varphi$ .
-  $s_i \models \mathbf{G}\varphi$  if for all  $j \geq i$ ,  $s_j \models \varphi$ .
-  $s_i \models \varphi_1 \mathbf{U} \varphi_2$  if there exists  $j \geq i$  such that  $s_j \models \varphi_2$ , and for all  $i \leq k < j$ ,  $s_k \models \varphi_1$ .
In the ZoneEnv example in Fig. 1 (left), the robot is tasked to oscillate between yellow zones and white zones while always avoiding red zones. Assume a set of atomic propositions  $\mathcal{G} = \{y, w, r, j\}$  that represents the four colors yellow, white, red, and jet black. The LTL objective for this task can be specified as:
$$
\mathbf {G F} (y \wedge \mathbf {X F} w) \wedge \mathbf {G} (\neg r) \tag {2}
$$
Büchi Automaton is a type of automaton used to recognize languages that consist of infinite sequences of symbols. A Büchi automaton is defined by a tuple  $(Q, \Sigma, \delta, q_0, F)$ , where  $Q$  is a finite set of automaton states,  $\Sigma$  is a finite alphabet of symbols,  $\delta: Q \times \Sigma \to 2^Q$  is a transition function that maps each state and symbol to a set of states,  $q_0 \in Q$  is the initial state, and  $F \subseteq Q$  is a set of accepting states. Informally, a Büchi automaton recognizes a language consisting of all infinite sequences of symbols (corresponding to automaton runs) that visit an accepting state infinitely often.
Büchi Automaton Construction from LTL. Büchi automata can be used to represent temporal specifications in LTL. Given an LTL formula  $\varphi$  over a set of propositional symbols in  $\mathcal{G}$ , a Büchi automaton  $\mathcal{B}$  can be constructed that recognizes all infinite sequences of system states that satisfy  $\varphi$ . The alphabet  $\Sigma$  of the Büchi automaton  $\mathcal{B}$  is over the set of atomic propositions  $\mathcal{G}$  of  $\varphi$ , i.e.,  $\Sigma = 2^{\mathcal{G}}$ . That is, an infinite sequence of system states  $s_0, s_1, s_2, \ldots$  satisfies  $\varphi$  if and only if there exists an infinite sequence of automaton states  $q_0, q_1, q_2, \ldots$  in  $\mathcal{B}$  such that  $q_0$  is the initial automaton state, for all  $i \geq 0$ , if  $s_i \models w_i$  where  $w_i \in \Sigma$  then  $\delta(q_i, w_i)$  contains at least one state, and the set of states that occur infinitely often in the sequence  $q_0, q_1, q_2, \ldots$  is a subset of the accepting states of  $\mathcal{B}$ .
For example, for the ZoneEnv task in Fig. 1 (left), the converted Büchi automaton for the LTL task in Eq. 2 is depicted in Fig. 1 (right). The only accepting state in this machine is state 2. To reach the accepting state, a yellow zone must be visited first. The Büchi automaton accepts when state 2 is reached infinitely often, meaning that the agent must always make a loop back to state 2 from state 3 and then state 1 by visiting a white zone and a yellow zone. This kind of infinite looping behavior cannot be expressed by a regular expression.
3 Instructing Goal-Conditioned Agents with LTL Objectives
Task-Augmented Markov Decision Processes. We now formalize the problem of instructing an RL agent to follow LTL objectives. To this end, we generalize Goal-Augmented MDPs to Task-Augmented MDPs  $(S,A,\mathcal{G},\Phi ,L,T)$  where the definitions for state space  $S$ , action space  $A$ , goal space  $G$ , labeling function  $L$  (from states to atomic propositions in  $\mathcal{G}$ ), state transition function  $T$  remain the same.  $\Phi$  is the universe space of LTL formulas that can be constructed from the propositions in  $\mathcal{G}$ . We would like to construct a policy  $\pi (a|s,\varphi)$  such that for any possible LTL formula  $\varphi \in \Phi$ , the policy  $\pi (a|s,\varphi)$  has the highest probability of LTL specification satisfaction with respect to  $\varphi$ :
$$
\pi^ {*} = \arg \max  _ {\pi} \mathbb {E} _ {\tau \sim \pi (\cdot |, \varphi)} [ \mathbb {1} [ \tau \vdash \varphi ] ] \tag {3}
$$
In this formulation, a task-augmented MDP does not have a reward function. Instead, the objective is to generate the most number of LTL-satisfying runs for a given LTL task. The MDP does not have a maximum horizon as the agent is expected to run over an infinite time horizon to satisfy an  $(\omega$ -regular) LTL instruction (e.g., Fig. 1).
Solving task-augmented MDPs turns out to be significantly more challenging than learning for goal-augmented MDPs due to the extremely large task space, which grows exponentially with the number of goals. The agent must identify the correct sequence of sub-goals to accomplish an LTL task. For the ZoneEnv example in Fig. 1, an agent may be instructed to accomplish a task by reaching a white zone first and then proceeding to a red zone in two distinct phases. In each phase, the zone to avoid changes (avoiding jet black and then yellow). We specify this task in Eq. 4 whose Büchi automaton representation is depicted in Fig. 2.
Figure 2: Another LTL task for ZoneEnv.
$$
- j \mathbf {U} (w \wedge (- y \mathbf {U} r)) \tag {4}
$$
The two LTL tasks in Eq. 2 and Eq. 4 express very different requirements and require distinct logic to solve. The agent is expected to successfully fulfill both tasks during inference.
3.1 LTL Task Planning on Büchi Automata
Our approach relies on the principle that an accepting run of a Büchi automaton must contain an accepting state that is reachable from the initial state and lies on a cycle (e.g., Fig. 1). This means that accepting runs of a Büchi automaton can be searched on top of a directed graph representation of the automaton. A goal-conditioned agent can then be used to subsequently achieve the goals along the search paths for task execution.
Figure 3: An LTL Task for Ant-16rooms where the red path is more feasible than the black path.
However, this simple strategy may yield suboptimal task performance. Consider a Mujoco ant navigation task in a 16-room environment depicted in Fig. 3(left). The rooms are separated by thick walls and are connected through bi-directional doors. The agent is initially positioned at the center of the bottom-left room and is given the following LTL instruction to traverse through the centers of a series of rooms with choices:
$$
\mathbf {F} \left( \right.\left((0, 2) \vee (2, 0)\right) \wedge \mathbf {F} \left(\left(2, 2\right) \wedge \mathbf {F} \left(\left(\left(2, 1\right) \vee (3, 2)\right) \wedge \mathbf {F} \left(\left(3, 1\right) \wedge \mathbf {F} \left(\left(\left(1, 1\right) \vee (3, 3)\right) \wedge \mathbf {F} (1, 3)\right)\right)\right)\right)
$$
Here an atomic proposition  $g \in \mathcal{G}$  is in the form of  $(r, c)$  denoting the room in the  $r$ -th row and  $c$ -th column. The bottom-left corner is room  $(0, 0)$ . Given an environment state  $s$ , define  $s \vDash (r, c)$  is true if the position of the Mujoco ant in  $s$  is close to the center of the  $(r, c)$ -th room within a threshold. The Büchi automaton  $\mathcal{B}$  converted from the LTL instruction is given in Fig. 3(right). There are 8 paths on  $\mathcal{B}$  to reach the accepting state 6. A graph search algorithm could yield any of these paths for the Mujoco ant to traverse along the chosen route. However, not all of them are equally optimal.
Figure 4: Graphs (a), (b), and (c) are converted from the Büchi automata in Fig. 3, Fig. 1, and Fig. 2. (d), (e), and (f) are types of transitions in a graph representation of a Büchi automaton.
Some of the routes require the ant to take a detour to avoid the black obstacles placed in certain room centers. In the worst case, the agent may be blocked by the obstacles and cannot complete the task.
In order to optimally accomplish the sequences of tasks defined in an LTL instruction  $\varphi$  with atomic propositions in  $\mathcal{G}$ , we perform high-level reasoning by evaluating the difficulty of achieving the specified goals (i.e. atomic propositions) in  $\varphi$  in the goal space  $\mathcal{G}$ . Our technique exploits the value function  $V^{\pi}(s, g)$  of a goal-conditioned agent  $\pi(\cdot | s, g)$  to capture the agent's transition capabilities. For any pair of goals, we learn a goal value function  $\mathcal{V}^{\pi}: S \times \mathcal{G} \times \mathcal{G} \to \mathbb{R}$  to estimate the capability of taking the goal-conditioned policy  $\pi$  to go from one goal to another goal on average. The function is parameterized by an environment state  $s$  because the agent's capability to transition between goals can differ depending on various environment layouts and agent states. To get the desired estimate, we regress  $\mathcal{V}$  towards the value function of  $\pi$  by minimizing:
$$
\min  _ {\mathcal {V}} \left(V \left(s _ {t}, g _ {k}\right) - \mathcal {V} \left(s _ {j}, g _ {t}, g _ {k}\right)\right) ^ {2} \quad \text {w h e r e} g _ {k} \in L \left(s _ {k}\right) \wedge g _ {t} \in L \left(s _ {t}\right) \tag {5}
$$
with  $\tau \sim B$  (a replay buffer),  $t \sim \{0 \dots t_{max}\}$ ,  $s_t, a_t, s_{t+1} \sim \tau$ ,  $j \sim \{0, t\}$ ,  $k \sim \{t + 1 \dots t_{max}\}$ , and  $L$  as the state labeling function. We train  $\mathcal{V}$  together with a goal-conditioned learning algorithm.
With the learned goal value function  $\mathcal{V}$  of a goal-conditioned agent as a measurement to assess its capability to transition between goals, we reduce the task of learning an LTL specification satisfying policy to finding the optimal accepting run in the Büchi automaton representation of the LTL specification. We convert a Büchi automaton  $\mathcal{B} = (Q, \Sigma, \delta, q_0, F)$  as a directed graph  $G_{\mathcal{B}} = (\mathcal{Q}, \Sigma, \Delta, \mathcal{Q}_0, \mathcal{F}, \mathcal{L})$  with nodes  $\mathcal{Q}$ , node transitions (edges)  $\Delta$  over the same alphabet  $\Sigma$ , a set of initial nodes  $\mathcal{Q}_0$  and accepting nodes  $\mathcal{F}$ . The conversion from  $\mathcal{B}$  to graph  $G_{\mathcal{B}}$  starts with setting  $\mathcal{Q} = Q$ ,  $\Delta = \delta$ ,  $\mathcal{Q}_0 = \{q_0\}$ , and  $\mathcal{F} = F$ . The node labeling function  $\mathcal{L}$  maps nodes in  $\mathcal{Q}$  to atomic propositions (or goals) in the alphabet  $\Sigma$ . Intuitively,  $\mathcal{L}$  approximates the specific region in the goal space where the agent should be positioned at a given node. For each non-self transition  $(q_i, \psi, q_j) \in \delta$  in  $\mathcal{B}$  where  $\psi$  encodes the goal space to which the transition is enabled, we have  $\mathcal{L}(q_j) = \psi$ . As an example, Fig. 4(a) depicts the converted graph representation  $G_{\mathcal{B}}$  from the Büchi automaton in Fig. 3(right). For the transition from the initial node to node 1, the node labeling function  $\mathcal{L}$  maps node 1 to goal room  $(0,2)$ . This transition denotes that the agent reaches room  $(0,2)$  from the initial room. If there are several incoming transitions with different alphabet symbols to  $q_j$  in  $\mathcal{B}$ , we duplicate  $q_j$  and its outgoing transitions in  $G_{\mathcal{B}}$  so that each duplicated node for  $q_j$  captures one of the possible transitions to  $q_j$ . For example, in Fig. 3, there are two non-self Büchi automaton transitions to state 1 which are encoded as node transitions from the initial room to room  $(0,2)$  and  $(2,0)$  respectively in Fig. 4(a). An exception for constructing the node labeling function  $\mathcal{L}$  occurs when  $\psi$  takes the form of  $\wedge_k - g_k$  on a non-self transition  $(q_i, \psi, q_j)$  where all atomic propositions  $g_k$  are negative, indicating a specific area in the goal space that must be avoided to activate the transition. In this case, the designated goal area outlined by  $\psi$  for  $q_j$  can be excessively large. We instead assign  $\mathcal{L}(q_j) = \mathcal{L}(q_i)$ . Intuitively, if  $g_k$  does not intersect with the current state on  $\mathcal{L}(q_i)$ , the agent can trivially fulfill the avoidance requirement by making a "jumped" transition to  $q_j$ . For example, the "avoidance" transition from state 2 to state 3 on the Büchi automaton in Fig. 1(right) is encoded as the rightmost non-self transition in the graph representation of the automaton in Fig. 4(b).
We interpret the semantics of a graph representation  $G_{\mathcal{B}}$  converted from a Büchi automaton  $\mathcal{B}$  through the different kinds of non-self transitions that  $G_{\mathcal{B}}$  may have, as visualized in Fig. 4(d), (e), and (f). In
Fig. 4(d), on a non-self transition  $(q_{i},\psi ,q_{j})$ , in order to transit to  $q_{j}$ , the goal-conditioned agent is tasked to reach the goal region  $\psi \equiv \bigwedge_{j}g_{j}$  from the goal region  $\mathcal{L}(q_i)\equiv \bigwedge_i g_i$ . The "self-transition" on a node  $q_{i}$  describes a goal-related constraint  $\psi^{\prime} = \bigwedge_{k}\neg g_{k}$  that must be maintained until the goal-conditioned agent can transit to  $q_{j}$ . The agent can take an unlimited number of steps to reach the goal region  $\psi$ , provided that each step avoids entering any undesired region  $g_{k}$  within  $\psi^{\prime}$ . For weighted graph search, we assign the weight of such a transition as:
$$
\mathcal {W} (s, \left(q _ {i}, \psi , q _ {j}\right)) = \max  _ {g _ {i}, g _ {j}} - \log \mathcal {V} (s, g _ {i}, g _ {j}) \text {w h e r e} \mathcal {L} (q _ {i}) = \bigwedge_ {i} g _ {i} \text {a n d} \psi = \bigwedge_ {j} g _ {j} \tag {6}
$$