SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the given set of grounded actions  $A_{\downarrow}^{\mathcal{C}}$  and a grounded set of predicates  $F_{\downarrow}^{\mathcal{C}}$ , we can create a symbolic model that is guaranteed to be optimistic for any MDP whose action set is isomorphic to  $A_{\downarrow}^{\mathcal{C}}$  and the state space can be represented using  $F_{\downarrow}^{\mathcal{C}}$ . In particular, the model will have empty preconditions and delete effect and the add effects would correspond to the set of all ground predicates. This means that every action is executable in every state and an execution of any action will satisfy the goal. We will denote this model as  $\mathcal{M}_0^{\mathcal{C}} = \langle F_{\downarrow}^{\mathcal{C}}, A_{\downarrow}^{\mathcal{C}}, I^{\mathcal{C}}, G^{\mathcal{C}} \rangle$ . More formally, every action  $a \in A_{\downarrow}^{\mathcal{C}}$  will be defined as follows:  $a = \langle pre_0^a, add_0^a, del_0^a \rangle$ , where  $pre_0^a = del_0^a = \emptyset$  and  $add_0^a = F_{\downarrow}^{\mathcal{C}}$ . The fact that its an optimistic representation for any MDP possible in this context can be trivially proved (Proof is provided in the appendix).
4.1 Refining the Model
Now, of course, while all valid traces for the original model correspond to a plan in  $\mathcal{M}_0^{\mathcal{C}}$ , the symbolic model may also support plans that may not correspond to any valid trace in the original model. Our basic strategy would be to use this model as a starting point to sample potential plans, simulate/execute them in the environment or simulator and use the outcomes (both successful and failed executions) to refine the current the current estimate. We will continue this process until we find a plan that leads us to the goal. Keeping this general approach in mind, the next step would be to define our model update rule. In particular, let us assume that we receive the following observation from the environment  $\langle s,a,s'\rangle$ , such that  $s' \neq \bot$ . Now we know this corresponds to the symbolic observation
$\langle \mathcal{C}(s),\mathcal{C}(a),\mathcal{C}(s^{\prime})\rangle$  . Given this observation, we know that any changes made in the state must be the result of the action. We will use this information to update action's effects. For add effects, if the estimate previously had hypothesized the action making a predicate true, which doesn't hold in  $\mathcal{C}(s^{\prime})$  then it can be removed from the add effects. Similarly, if there was a predicate that is made false in  $\mathcal{C}(s^{\prime})$  but was not part of the delete effects, it can be added to the set of delete effects. Formally, we can set the new estimate of the action as follows  $a = \langle pre_{i + 1}^{a},add_{i + 1}^{a},del_{i + 1}^{a}\rangle$  , where  $pre_{i + 1}^{a} = \{\phi |\phi \in pre_{i}^{a}$  and  $\phi \subseteq \mathcal{C}(s)\}$  and for effects we have  $add_{i + 1}^{a} = add_{i}^{a}\setminus (F_{\downarrow}^{\mathcal{C}}\setminus \mathcal{C}(s'))$  and  $del_{i + 1}^{a} = del_{i}^{a}\cup (\mathcal{C}(s')\setminus \mathcal{C}(s))$  . If the sampled transition corresponds to a failure  $(\langle s,a,\bot \rangle)$  , we will only update the precondition. Specifically, we will remove any precondition clause that satisfies the state and replace it with a set of preconditions that includes one of the predicates that was false in the model (this follows from the fact that the action failed because some predicate part of the true underlying precondition wasn't true in the given state). More formally, for any  $\phi \in pre_i^a$  , such that  $\phi \subseteq \mathcal{C}(s)$  , we remove  $\phi$  and add  $\Phi = \{\phi \cup f|f\in (F_{\downarrow}^{\mathcal{C}}\setminus \mathcal{C}(s))\}$  . The proof for why this update rules result in optimistic representations are provided in the appendix.
4.2 Overall Algorithm
Algorithm 1 presents the overall iterative algorithm we will be using to identify the action sequence that can lead to a goal state. The algorithm starts with the initial estimate of the model. It iteratively generates plans for the model estimate, which will then be used to progressively refining the model until we get a plan that corresponds to a path to a goal state. These plans are derived using a diverse planner that identifies a set of plans that are diverse in terms of the actions used. This is represented by the procedure DiversePlanner that takes the number of diverse plans to be generated as an argument  $(\kappa)$ . Readers can check [22] for a more detailed discussion of diverse planners. These plans are first tested on the underlying environment/simulator to check whether they lead to the goal from the initial state and if not the experiences sampled from their execution are used to refine the current model. Note that, given the optimistic nature of the model estimate, the planner would generally try to use actions that haven't been previously executed successfully. However, each future use of the action would become progressively harder due to the growing precondition set. With that said, one could further improve the planner behavior by being more careful about the actions being used as part of plans. If an action has been tested quite frequently, it would be better to de-prioritize its usage until no better alternative has been found. Note that this is quite similar to the kind of exploration performed in the context of multi-armed bandits [25]. In fact, one could directly apply methods like UCB [3] to select the action sets to be considered by the planner. This part of the algorithm is captured by the procedure PruneModel. To keep our implementation of the approach simple, we will use a simple queue based system to identify the actions to be included. The exact procedure we use to control the selection of actions is described in the appendix. The variable execution_statistics keeps track of previous action trials and the frequency of success per action. The procedure UpdateModel uses the rules described in Section 4.1 to use the sampled traces to update the given model estimate. One could also further improve the efficiency of the search by always testing all possible actions in every new state that is identified as part of the procedure.
Theorem 1 Algorithm 1 will (a) terminate in a finite number of steps and (b) identify a path to a goal (provided one exists); as long as the diverse planner used is complete (i.e., it will return a non-empty plan set as long as there exists a valid plan).
The proof for the theorem is provided in the appendix.
Leveraging the Exploration Strategy in the Context of RL algorithms: In the context of the overall RL learning process, this exploration method will be used as a way to update the Q values (and depending on the algorithm, structures like replay buffers). Specifically, we will first run this exploration procedure to find a valid trace to the goal. Once such a trace is found, we can update the Q values of all the states that are part of that trace to a more informed value. Once updated, we can employ traditional RL algorithms to identify optimal policies. One could also leverage the proposed method in conjunction with other exploration strategies, during the learning process. It is important to note that any consecutive use of our approach for generating goal directed paths would be much more efficient, as the method will start from a more refined estimate of the model.
Leveraging Lifted Representation The algorithm described above tests each of the available actions to learn a symbolic model corresponding to the observed behavior. However, one of the important points to note here is the fact that this means that the testing and by extension learning of the
model occurs at the level of ground actions. As we had discussed earlier, a very common assumption made throughout symbolic models is that of the existence of a lifted representation of actions. Namely, the fact that the nature of actions could be described independently of the exact objects it may be interacting with. This is a very natural outcome that comes out of relational representations of tasks, where the state is represented in terms of objects and relationship between objects. Consider a simple domain where a robot is tasked with stacking blocks on the table (popularity called blocksworld [35]). It is very easy to see that the outcome of picking up the red block should be quite similar to the case of picking up the green block of the same size. For example, if we observe that the execution of the action 'pick-up red block' results in the agent holding the red block in it's gripper; then it would be quite natural to assume that the execution of 'pick-up green block' should result in the agent holding the green block. We will leverage such symmetry by asking the human to provide some additional information about each action. Specifically, the human can provide us a basic annotation over what actions could share a lifted structure and what objects each actions might interact with. Note that we are not asking the users to specify what the lifted structure may be, but just a grouping of actions and an ordered list of relevant objects. The order may reflect the different roles played by the objects participating in the action. For example, when an object is being placed on top of another, the annotation may list the destination object first and then the object being placed on top of it. The exact ordering wouldn't matter provided they remain consistent through the annotations. Additionally, even if the grouping provided by the human may be a subset of the true possible grouping and the human provides a superset of the objects relevant to any given action, our generalization approach remains valid. The set of objects associated with each action could also be automatically extracted from natural language descriptions of actions, as performed by works like that by [10].
For a given set of actions that are marked as being grounded instances of the same lifted action, we will ensure that learned effects of all actions comply with the most refined action in the set. As discussed earlier, the effects of an action comprises of add and delete effects and for each component we can select the most refined set independent of each other. From the set of effect descriptions, we select the add effect set containing the minimum number of elements and the delete effect set containing the maximum number of elements. For each such set, we can create the lifted description using the  $\lambda^{\dagger}$  function described earlier. Let min_add be the lifted description corresponding to the smallest set of adds and max_del be the largest set of deletes for a given set of actions corresponding to the same lifted action. Then we can simply replace the effect of every action with a grounding of these lifted actions. This will still result in an optimistic model description, as we can show that the min_add and max_del are still optimistic estimates
Proposition 1 Let  $\bar{A} = \langle a_1, \dots, a_m \rangle$  be a set of actions marked as being instances of a single lifted action  $a^\uparrow$ . Then min_add must be a superset of add effects of  $a^\uparrow$  and max_del a superset of deletes of  $a^\uparrow$ , where min_add and max_del are calculated for  $\bar{A}$
The proof for validity of this proposition is discussed in the appendix. This proposition now means that, once  $\max\_del$  and  $\min\_add$  are identified, then for every action  $a$  in the set of possible groundings we can replace add effects and delete effects with the corresponding grounding of the lifted effects, i.e.,  $add^a = \Gamma^\uparrow(\min\_add, \Theta^a)$  and  $del^a = \Gamma^\uparrow(\max\_del, \Theta^a)$ , where  $\Theta^a$  is the object list corresponding to the action  $a$ . One can follow similar lines of reasoning to show that the lifted description of the maximal precondition description is guaranteed to entail the true preconditions.
Lifted Representation as a Basis for Curriculum Learning. While the above discussion focused on leveraging lifted representation to speed up learning within a given planning problem, one could also use lifted representations as basis of transferring model information from one problem instance to another. Within classical planning problems, it is very common to separate the domain information represented in lifted terms from specific problem instances. Each problem instance could differ in terms of the number and identities of objects involved, the initial and goal state. However, actions applicable in every instance share the same lifted definition. Even within RL, benchmarks consisting of various instances of the same problem domain are becoming more popular (Minigrid [8] being a popular example). When such a suite of problem instances are available, one could further speed up learning by using the smaller instance to learn as much of the lifted model as possible. Once such a partial lifted model is learned, it can then be used to refine the optimistic model in the target problem, where the normal learning process then takes over. Note that the access to a smaller problem instance doesn't obviate the need to perform learning in the true underlying model. After all, in the smaller problem there may be lifted actions that may not be executable in any of the reachable states, but needs to be used in the target problem to reach the goal.
$5\times 5$
$6\times 6$
$8\times 8$
$16\times 16$
Cumulative Reward: Gym MiniGrid DoorKey 5x5
Cumulative Reward: Gym MiniGrid DoorKey 6x6
Cumulative Reward: Gym MiniGrid DoorKey 8x8
Figure 1: Four minigrid maps considered and the cumulative rewards per episode comparing our method to a vanilla RL. The values are plotted along with their  $95\%$  confidence interval.
Cumulative Reward: Gym MiniGrid DoorKey 16x16
We perform our evaluation in four different domains. Three of these correspond to traditional planning domains and one a more traditional reinforcement learning benchmark. The planning benchmarks include blockworld, a simple gridworld type domain involving robot picking up objects and a domain where the agent has to control elevator schedules. For the RL domain, we looked at two variants of minigrid problem. One was the version introduced by [26] (henceforth referred to as Minigrid-Parl) and the other being a simplified version of the original minigrid testbed [8]. We chose to use the Minigrid-Parl, since it provided us with annotations that allowed us to use hierarchical RL methods. For the latter variant, we dropped the turn action and introduced directional movement, pick up, drop and toggle actions. This allowed us to use simpler PDDL formalisms to capture the domain. For each planning domain, we selected five different problems (the sizes are approximately listed in the tables in terms of the number of grounded predicates) and two problems for the minigrid domain. We created a simulator wrapper around PDDL models for each of the problems, as it allowed us easy access to the annotation information for lifting. For the minigrid problems, we auto generated PDDL problem files from the simulator code for each specific environment. he code can be found at https://github.com/sarathsreedharan/ModelLearner.
5.1 Reaching Goal States
As a first step, we are interested in testing how our proposed method compares against standard exploration strategies used by RL algorithms. In particular, we compare our method against three baselines: vanilla  $\epsilon$ -greedy exploration (as implemented by the SimpleRL framework [1], as part of the Q learning agent), R-max based exploration strategy (again taken from the SimpleRL framework), which as we discussed is a form of intrinsic reward, and a hierarchical RL method that learns a policy over SMDP using PPO (as implemented by [26]) on instances from Minigrid-PARL.
Problem Instance	ε-greedy	Our method (w/o lifting)	Our method (with lifting)
Name	Size	solved	time	# samples	solved	time	# samples	solved	time	# samples
Blocks	25	3/5	0.89	9164.2	3/5	26.7	19262	5/5	5.59	592
25	11.35	115136.8	399.35	168859.6	31.96	56404.8
25	1.99	20702	46.86	18901.8	9.28	4432.4
36	-	2966996.4	-	96152.8	32.99	191451.2
36	-	3135384	-	142605.4	33.93	138203.4
Elevator	20	0/5	-	2045316.4	2/5	408.79	3394856.8	5/5	36.94	88108
20	-	2087898.2	-	3839855.6	26.73	66507
20	-	2062441.2	401.85	3053364.2	21.25	88835
20	-	2089477.6	-	3099277.6	36.12	83296
20	-	2081322	-	4117586.6	36.57	87747.2
Gripper	25	1/5	5.56	53929.8	2/5	73.52	77450.2	5/5	15.9	16523
25	-	1511944.6	328.74	252954.4	23.17	308598.6
25	-	2663695	-	83551.8	35.93	309964.8
36	-	2395190.2	-	55209.8	43.31	308598.6
36	-	2112899.6	-	37136.8	61.99	624489.6
Minigrid-PARL	94	0/2	-	2743179.4	0/2	-	310032	1/2	86.41	342981.8
593	-	334535.2	-	2458.2	-	8217319.4
Table 1: Our method w/wo lifting generalization vs. Q learning. Times are listed in seconds and only the average time and number of samples are reported (full data is in the Appendix).
Our interest is not only to see how well the current method performs, but also to see how much is contributed by the action-level generalization provided by lifted representations. Our primary metrics of evaluation are going to be, (a) do the methods consistently reach the goals, (b) the number of samples collected from the environment as part of reaching the goal, and (c) the time taken by the method to reach the goal. This third aspect is an important one to consider to make sure that the RL based exploration is given a fair chance when compared against planning based methods. After all, the planning methods reason over environment model, allowing them to perform less interactions with the environment. However, this adds a computational overhead, that might not be required for other method, such as vanilla RL methods. We capture that tradeoff of one computation for another by measuring the time to reaching the goal. Additionally, we set a time limit on the exploration step, as for some of these problems the exploration might not be completed in a reasonable amount of time. For all planning based instances we set the time limit to 10 minutes, while for the minigrid instances we extended the time limit to 30 minutes. Every experiment is run five times, averaging the results to account for possible randomness in the learning process. All seed values were randomly assigned and kept constant through the all five runs. As the underlying diverse planner, we used FI [15], generating ten different plans at every step. Table 1 presents the comparison of our method against Q learning for the planning benchmarks. Both R-max and SMDP time out on all tested instances, so we will skip reporting their values in the table. SMDP took 188416.8 and 106821.4 samples each for the two minigrid problems. We see that apart from Blocksworld and minigrid domain, our vanilla method is able to solve more problems and our method equipped with the application of lifting rule outperforms both by a wide margin. Neither R-Max or SMDP visited any of the goal state in the given time limit.
5.2 Overall Learning Performance
With the initial results collected on how well our method is able reach goal. The next question we wanted to answer was how well an RL algorithm augmented with our new exploration strategy is able to perform. For this question we focused on tabular Q learning and the minigrid environment. Specifically, we compared an instance of Q learning algorithm where Q values were initialized using the plan generated through our exploration process and a vanilla one that started with no such information ( Top row of Figure 1 provides a visualization of the maps considered for these experiments). Columns two and three of Table 2 presents the time and samples taken to get such a plan for each of the problem instance considered. The bottom two rows of Figure 1 present how the total value per episode changes over episodes. As expected, the access to a valid plan (which isn't necessarily optimal) ensures that our method starts from a higher value. For smaller instances, we see that the vanilla method eventually catches up or at least gets closer. However for the largest problem instance, even after 1000 episodes the RL agent is still unable to get a positive reward since it never reaches a goal state.
Problem	with original PruneModel	with bootstrapping	w/o bootstrapping
time	# samples	time	# samples	time	# samples
DoorKey-5 × 5	35.39	23977.2	N/A	N/A	N/A	N/A
DoorKey-6x6	34.45	24418.4	42.14	27666.0	35.91	21531.8
DoorKey-8x8	54.63	260839.8	52.1	143223.4	92.78	367749.2
DoorKey-16x16	1488.08	6705823.2	2353.63	11088198	-	-
Table 2: The results from our method (with lifting) applied to the second Minigrid variant. Includes the performance under our original PruneModel strategy and the results on how bootstrapping helps learning under the new action selection strategy. The entries for DoorKey-16x16 are skipped for w/o bootstrapping, since the method timed out after 90 minutes.
5.3 Curriculum Learning
The final question we were interested in investigating was whether having access to a smaller problem instance had the potential of speeding up our learning process. We again went back to the minigrid problems and tested whether first learning a partial lifted model using the a problem of grid size  $5 \times 5$ , help speed up learning in larger problems. In particular, we stop learning in the smaller instance, as soon as we find a single solvable plan. We then use the lifted model information learned from the smaller instance to create a more informed model of the target problem. Then we follow the same procedure as before, until all the plans returned by the diverse planner is valid. One of the things we noticed was that bootstrapped models did significantly worse off under the original PruneModel method we used. We noticed that using a strategy based on failure count resulted in useful actions getting tested early and getting removed from consideration. So for this set of experiments we considered a different strategy that re-introduces actions more frequently. Last four columns of 2, presents the results for the minigrid problems, where the problem were bootstrapped with a lifted model learned from a  $5 \times 5$  grid. The method without bootstrapping timed out for the largest problem without finding a solution after 90 mins. Except for the smallest problem we see the bootstrapping giving a definite advantage.
6 Conclusions and Discussion