SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ssreedh3@colostate.edu
Michael Katz
IBM T.J. Watson Research Center
michael.katz1@ibm.com
There has been an increasing interest in using symbolic models along with reinforcement learning (RL) problems, where these coarser abstract models are used as a way to provide RL agents with higher level guidance. However, most of these works are inherently limited by their assumption of having an access to a symbolic approximation of the underlying problem. To address this issue, we introduce a new method for learning optimistic symbolic approximations of the underlying world model. We will see how these representations, coupled with fast diverse planners developed by the automated planning community, provide us with a new paradigm for optimistic exploration in sparse reward settings. We investigate the possibility of speeding up the learning process by generalizing learned model dynamics across similar actions with minimal human input. Finally, we evaluate the method, by testing it on multiple benchmark domains and compare it with other RL strategies.
Project Website: https://optimistic-model-learn.github.io/
A popular trend in recent years is using symbolic planning models with reinforcement learning (RL) algorithms. Works have shown how these models could be used to provide guidance to RL agents [37, 26, 12], to provide explanations [33], and as an interface to receive guidance and advice from humans [21]. Coupled with the fact that advances in automated planning has made available a number of robust tools that RL researchers could adapt directly to their problems (cf. [11, 29, 31]), these methods have the potential to help addressing many problems faced by state-of-the-art RL methods. However, a major hurdle to using these methods is the need to access a complete and correct symbolic model of the underlying sequential-decision problems. While there have been efforts from the planning community to learn such models [19, 38], most of those methods have focused on cases where the models are synthesized from a set of plan traces, hence corresponding to the traditional offline reinforcement learning setting. Interestingly, very few works have been done in synthesizing such models in the arguably more prominent RL paradigm, namely, online RL.
To fill this gap, in this paper we propose a novel algorithm to learn relevant fragments of symbolic models in an online fashion. We show how it could be used to address one of the central problems within RL, namely effective exploration. We show how our method allows us to perform goal-directed optimistic exploration, while providing rigorous theoretical guarantees. The exploration mechanism leverages two distinct components: (a) a representation that captures the most optimistic model that is consistent with the set of observations received, and (b) the use of a fast and suboptimal diverse planner that generates multiple possible exploration paths, which are still goal-directed.
The idea of optimistic exploration is not new within the context of RL. The most prominent method being the RMax algorithm [5]. RMax modifies the reward function to develop agents that are optimistic under uncertainty. Our use of symbolic models, however, allows us to maintain an optimistic hypothesis regarding the underlying transition function. Coupled with a goal-directed planner, this lets us perform directed exploration in sparse reward settings, where we have a clear
specification of the goal state but no intermediate rewards. As we show in this work, for a finite state deterministic MDP our method is guaranteed to generate a goal-reaching policy. Additionally, we investigate the use of a structured form of generalization rule that leverages a very simple intuition, namely the effects of an action don't depend on specific object labels but only on object types. Commonly referred to as lifted representation in planning literature, we show this rule to speed up learning with minimal human input.
The rest of the paper is structured as follows. We start with related work in Section 2. Section 3 provides a formal definition of the exact problem we are investigating and Section 5 shows the empirical evaluation of our method against a set of baselines. Finally, Section 6 concludes the paper with a discussion of the methods and possible future directions.
As mentioned earlier, one of the foundational works in optimistic exploration in the context of reinforcement learning is R-max [5]. Even before the formulation in its current popular form, the idea of optimism under uncertainty has found several uses within the RL literature (cf. [20]). R-max can be seen as an instance of a larger class of intrinsic reward based learning [2], but one where the reward is tied to state novelty. Other forms of intrinsic rewards incentivizes the agent to learn potentially useful skills and new knowledge. A context where model simplification has been used in areas related to RL is in the context of stochasticity, where methods like certainty equivalence and hindsight optimization has been applied [4, 40]. In Section 6, we will see how we can also apply our methods directly in settings with stochastic dynamics. In regards to the user of symbolic models, the most common use is in the context of hierarchical reinforcement learning. Many works [26, 17, 37, 27], have investigated the possibility of using the symbolic model to generate potential options and then using a meta-controller to learn policies over such options. While most of these work assume that the model is in some way an approximation of the true model, all inferences performed at the symbolic level is performed over the original model provided as part of the problem. While in this work, we focused on cases where the symbolic model could in theory exactly capture the underlying model, the same techniques can also be applied to cases where the planning model may represent some abstraction of the true model. Another popular use of symbolic model is as source of reward shaping information (cf. [12]). In this context, works have also looked at symbolic models as a vehicle to precisely specify their objective [16, 13]. In terms of learning symbolic models, interestingly the work has mostly focused on learning plans or execution traces [38, 19, 6, 9].
In this context, it is worth noting and contrasting our proposed work with the ones focused on learning safe models [34, 19, 18, 28]. Safe models are defined to guarantee to only generate valid plans. This is an important theoretical guarantee to have if the learned model is the only source of information about the plans, or the cost of executing an invalid plan may be extremely high (e.g., because of dead-ends that might leave the AI agent stuck or if there are safety implications related to executing an incorrect action). In practice, this means learning pessimistic planning models, i.e., models that support only a subset of all possible valid plans. As we move from an offline setting to the more popular online one, such pessimism can become a burden rather than a strength. In the online setting, the agent is expected to have access to a simulator of the world, or the agent is allowed to make mistakes as part of its action execution without any irreversible damage. This means that the agent is free to try out different things until it finds a plan that works. Such methods naturally lend themselves to the use of an optimistic planning model that supports a super-set of possible plans.
In most of these works, the theoretical guarantee you are aiming for is to generate more pessimistic models that are always guaranteed to work but may overlook plausible plans. This is completely antithetical to considerations one must employ when performing explorations in common online RL settings, where the agent is either operating in a safe environment or interacting with a simulator. To the best of our knowledge, all existing online methods to acquire symbolic models [7, 24], focuses on extracting an exact representation of the true underlying model. Since our primary motivation for learning this model is to drive the exploration process, we do not have that limitation. Instead, we focus on learning a (more permissive) optimistic approximation. Also, it is worth noting that, the assumption that the system will be provided action arguments (something we will leverage to generalize learned dynamics) is one commonly made by most of these works. There are also some works that are trying to automatically acquire abstract symbolic models from an underlying MDP (including potential symbols) like that by [23]. This direction is orthogonal to our work, as symbols
produced by them may be meaningless to the human and we are explicitly trying to leverage human's intuitions about the problem.
3.1 Deterministic MDP
The central problem we are interested in addressing is that of an RL agent trying to come up with a policy for a deterministic MDP with sparse reward. We specifically chose a setting, which forefronts the problems related to exploration, while placing less emphasis on other aspects of an RL problem (though in Section 6 how one could easily apply this approach to other settings). In particular, the underlying model (which is unknown) is assumed to be of the form  $\mathcal{M} = \langle S, T, A, I, R, \gamma \rangle$ . Under this definition,  $S$  is a finite set that represents the set of possible states that the agent could find itself in. We will assume that this set includes a special absorbing state  $\bot \in S$ , allowing to capture both abnormal and normal trace termination. Additionally, there is a subset of states  $S^G \subset S$  which correspond to 'goal' states, which are desirable states for the agent.  $A$  is the set of possible actions and given the deterministic nature of the problem, the transition function is specified as  $T: S \times A \times S \to \{1,0\}$ . We will refer to the action that transitions from states in  $S^G$  to the absorbing state  $\bot$  as the goal action  $(a^G)$ . Given that we are interested in sparse reward setting, we will define the reward function as  $R(s, a^G, s') = 1$  if  $s \in S^G$  and  $s' = \bot$  and 0 otherwise. Finally,  $\gamma$  is the discount factor. Note that our work can support cases where the discount factor is equal to one.
We will refer to the transition to  $\bot$  through a non goal action, as a failure transition. Now,  $I\in S$  captures the initial state from which the agent starts. To enumerate the implications of the design choices we made in picking this model, consider the fact that reward is zero everywhere except for the goal. This means that any policy that can help reach the goal from the initial state, would be optimal for the agent (since the agent always starts from the initial state). Coupled with the fact that the transition function is deterministic, once the agent identifies a goal state, constructing an optimal policy is relatively easy as they can just reuse the path taken by the exploration strategy. Now, this setting also renders most existing methods that may use intermediate value bootstrapping or generalization mostly ineffective as there are no intermediate values to use. So it makes sense to focus on tabular methods as the RL baseline. In fact, possibly the only effective methods in the mainstream RL repertoire we can use are curiosity driven or intrinsic reward based methods and we will use such a method as a baseline. One of the central components we will leverage are state action traces we can sample from the underlying model. In particular, we say that a trace  $\langle s_0,a_1,\dots ,a_i,s_i,\dots ,a_{k + 1},s_{k + 1}\rangle$  is valid or equivalently goal-reaching if  $s_0 = I,s_k\in S_G,s_{k + 1} = \bot$ , and for every  $0\leq i\leq k$  we have  $T(s_{i},a_{i + 1},s_{i + 1}) > 0$ . The action  $a_{k + 1}$  is the goal action.
In the course of discussion, we will use the word 'original model' to refer to this true but unknown underlying MDP. The agent itself is expected to be either interacting with a generative simulator that encodes this MDP or is acting in the true environment provided that they can reset to the initial state at the end of each episode.
3.2 Symbolic Planning Models
For the symbolic model, we will be using an a representation scheme called Planning Domain Definition Language or PDDL. In particular, we will consider a version that will ignore object types [14]. Here, a planning task is defined in relational terms, i.e., states are described in terms of objects and relationships between objects and each action is described in terms of the objects involved in that action and how they may affect or be affected by the relationships between these objects. Such a model is usually defined by the tuple  $\mathcal{M}^S = \langle \mathcal{L},\mathcal{O},I,G\rangle$ , where  $\mathcal{L}$  is a first-order language,  $\mathcal{O}$  a finite set of action schemas,  $I$  and  $G$  are specifications of the initial state and the goal, respectively. The first order language describes the objects and the relationships between these objects (captured as predicates). Additionally, first order language allows specifying predicates over variables as well as actual objects. Formally, the first order language is specified as  $\mathcal{L} = \langle \mathcal{B},\mathcal{V},\mathcal{P}\rangle$ , where  $\mathcal{B}$  is the set of all objects,  $\mathcal{V}$  are the variable names and  $\mathcal{P}$  are the predicates. Each predicate  $p\in \mathcal{P}$ , will take a fixed number of arguments. For the purpose of discussion in this paper, we will either have cases where the arguments consist of only variables or only objects. We will refer to the former case as being the lifted representation of the predicate and the latter as a grounded instance of the predicate (or ground predicate). In general, however, predicates can be partially grounded, with some of the arguments being actual objects while others being variables. States of the model correspond to truth assignments to ground predicates. Each ground predicate can take either a true or a false value. Each
possible state for a given model is captured by a specific instantiation of all ground predicates. Thus, possible states can be uniquely represented by sets of ground predicates that are true (assuming the rest to be false under the closed world assumption). In this representation,  $I$  denotes a set of ground predicates, capturing the unique initial state. For our purposes,  $G$  will be captured by a subset of ground predicates, denoting all states where all of these ground predicates (and possibly some more) are true. All such states will be considered valid goal states.
Each action schema  $o \in \mathcal{O}$  provides the basic structure shared by a set of ground actions that can be actually executed by the agent. Each action schema is defined as  $o = \langle \text{params}(o), \text{pre}(o), \text{add}(o), \text{del}(o) \rangle$ , where  $\text{params}(o)$  indicates the parameters of the action schema (variables and objects), the preconditions  $\text{pre}(o)$ , the add-effects  $\text{add}(o)$ , and the delete-effects  $\text{del}(o)$ . The latter three are first-order formula over the language  $\mathcal{L}$ , specifying the conditions that must be satisfied for the action to be executable in a state, as well as the change in a state resulting from executing the action. Ground actions are obtained from the action schema by assigning objects to variables in the parameters. The agents are executing the ground actions and therefore it is common to describe the semantics of ground actions, henceforth simply referred to as actions. In this work we restrict ourselves to preconditions in disjunctive normal form. For ground actions these would be disjunctions of conjunctions of ground predicates. All states in which the formula holds, the action is applicable. The add/delete effects are conjunctions of ground predicates, making these predicates true respectively false in the state resulting in successful action execution.
For a given action schema  $o$ , we will denote the grounded instance obtained by replacing the parameter with an object list  $\Theta$  using the function symbol  $\Gamma^{\uparrow}(o, \Theta)$ . We can also define an inverse mapping function  $\lambda^{\uparrow}(o^{\downarrow}, \Theta)$  that retrieves the lifted model given a grounded instance (we can do this by replacing all instances of an object with a variable). This lifting function  $\lambda^{\uparrow}$  is well defined in all cases where we don't have a repeating object in  $\Theta$ . In this particular work, we will only focus on applying such lifting functions in cases where they are well defined. Overloading the notations a bit, we will also apply the functions  $\Gamma^{\uparrow}$  and  $\lambda^{\uparrow}$ , to create grounded and lifted instances of predicates as well. Each planning problem can be represented equivalently in a grounded form as  $\mathcal{M}_{\downarrow} = \langle F_{\downarrow}, A_{\downarrow}, I, G \rangle$ , where  $F_{\downarrow}$  consists of all grounded predicates and  $A_{\downarrow}$  grounded actions. At most this model may have  $2|F_{\downarrow}|$  states. A solution for a planning model is a plan  $\pi = \langle a_1, \dots, a_n \rangle$ , which is a sequence of action whose execution in initial state will lead to a goal state, i.e.,  $\pi(I) \supseteq G$  (where  $\pi(I) = a_n(\dots,a_1(I))$ ).
3.3 Connecting the Symbolic Model to the MDP
For any given deterministic MDP  $\mathcal{M}$  of the form defined in Section 3.1, there must exist a symbolic model that can exactly capture the MDP. In particular, there is a surjective function (many-to-one) mapping the (ground) actions in the symbolic model to MDP actions. Every plan under the symbolic model maps by this mapping to a valid trace of the MDP. The appendix includes a proof that shows by construction how such a model will always exist. However, rather than creating an arbitrary mapping to a symbolic model, we are interested in creating one that leverages the expertise of a human domain expert to creating a potentially more effective representation of the problem. In particular, we start by taking human input to learn how to symbolically represent the states of the MDP. In particular, we expect the human to specify a set of predicates and objects that they might associate with the given problem. We use the symbol  $F_{\downarrow}^{\mathcal{C}}$  to capture the set of all ground predicate possible under this specification. Similar to previous explanatory works [33, 32, 21] that have tried to learn symbolic representations of RL problems, we use this to learn binary classifier that test whether a ground predicate may be true in a given MDP state. We can learn such classifiers by collecting positive and negative examples for each ground concept. Once the classifiers are available, we can construct the symbolic state corresponding to each MDP state, by testing each classifier on any given MDP state. We use the function  $\mathcal{C}: S \to 2^{F_{\downarrow}^{\mathcal{C}}}$  as a way to capture the mapping between the states. For potential actions, we assume that every symbolic ground action corresponds to exactly one action in the MDP. Overloading the notations a bit, we use  $\mathcal{C}^{-1}(a)$  to represent the MDP action corresponding to the symbolic action  $a$ . As we will see later, the agent can also potentially leverage the human's intuitions about how they structure actions to further improve the effectiveness of our method. Finally, we expect the human to provide a specification of the goal states specified in terms of the ground predicates in  $F_{\downarrow}^{\mathcal{C}}$ . We denote this goal specification by  $G^{\mathcal{C}}$ . Additionally, we require that the initial state for the symbolic model corresponds to  $\mathcal{C}(I)$  and for any goal state  $s \in S_G$ ,  $\mathcal{C}(s)$  satisfies  $G^{\mathcal{C}}$  (or, equivalently, there is a symbolic goal action whose precondition meets this requirement).
Algorithm 1 Iteratively refine the model until a goal reaching trace is found
Iterative-Model-Refinement  
Input:  $\mathcal{M}_0^C, \kappa$   
Output: An action sequence  $\langle a_1, \dots, a_k \rangle$  that will lead to the goal  
Procedure:  
 $\mathcal{M}_{curr} \gets \mathcal{M}_0^C$   
execution_statistics  $\leftarrow \{\}$ , solvability_flag  $\leftarrow$  True  
while solvability_flag is True do  
 $\widehat{\mathcal{M}}_{curr} \gets \text{PruneModel}(\mathcal{M}_{curr}, \text{execution_statistics})$ $\widehat{\Pi} \gets \text{DiversePlanner}(\widehat{\mathcal{M}}_{curr}, \kappa)$   
if  $|\widehat{\Pi}| > 0$  then  
for  $\widehat{\pi} \in \widehat{\Pi}$  do  
if  $\widehat{\pi}$  leads to goal in the environment then  
return  $\widehat{\pi}$   
else  
 $\mathcal{M}_{curr}, \text{execution_statistics} \gets \text{UpdateModel}(\mathcal{M}_{curr}, \widehat{\pi}, \text{execution_statistics})$   
end if  
end for  
else  
solvability_flag  $\leftarrow$  False  
end if  
end while  
return No policy with non-zero Value
The basis of our approach is an observation that every deterministic MDP has a precise symbolic representation. By precise representation, we mean that for the specific setting we consider here, there exists a symbolic model that can exactly simulate the MDP: any transition possible under the symbolic model must correspond to non-zero probability transition possible under the MDP and vice versa. However, as discussed, our objective is not to learn such a precise representation but rather only to learn an optimistic approximation. We start from a trivially optimistic representation of the underlying model, which we iteratively refine towards the true representation. At each iteration, the current symbolic representation is used to generate potential plans to the goal. These plans are then tested out in the environment and the observed outcomes of the execution of such action sequences are then used to refine our estimate towards the true representation. At every point of our model refinement process, we ensure that every subsequent model estimate generated is an optimistic one. By maintaining the optimistic nature of the representation, we ensure that no potential valid solution is overlooked at any point in the learning process. So we will start the discussion of our approach by providing a rigorous definition of what we mean by an optimistic representation. In particular, we are interested in creating symbolic representations that allow all valid traces that are possible under the original MDP to be possible under the new representations. Formally, we define this requirement as
Definition 1 For an MDP model  $\mathcal{M}$ , a symbolic model  $\mathcal{M}^{\mathcal{C}}$  defined over a symbol mapping  $\mathcal{C}(\cdot)$  is said to be an optimistic representation, if for every action sequence  $\langle a_1,\dots,a_k\rangle$  such that there exists a valid trace (i.e. it reaches goal), there exists a valid plan in  $\mathcal{M}^{\mathcal{C}}$  of the form  $\pi = \langle a_1',\dots,a_k'\rangle$ , such that  $\mathcal{C}^{-1}(a_i') = a_i$ .