SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our paper, presents a novel exploration paradigm for reinforcement learning algorithms. Our proposed method supports the learning and refinement of optimistic symbolic estimates of the underlying model for the given task. We show how we can start with a trivially optimistic model and then use a diverse planner to drive both the task-level exploration and the refinement of the model. Experiences generated from the execution of identified plans lead to better estimates of the task, which in turn leads to more informed plans. We additionally show how we can leverage lifted representations of the given task, to generalize any learned model information across various instances of the same lifted operators. We also use this mechanism to propose a novel curriculum learning paradigm for model learning. The effectiveness of our proposed method depends on three crucial factors: (a) the possibility of performing systematic refinements of our models while ensuring desirable properties, (b) the availability of fast, diverse planners, and (c) the ability to leverage human intuition about the task. The latter is of crucial importance: even if there were other model classes and planners we could exploit, the ability to tap into human knowledge gives us a significant advantage. Importantly, the same knowledge has been used by many of the other state-of-the-art methods. Further, it only represents a small subset of the information usually provided as part of a complete symbolic planning model. One of the aspects not discussed in the paper was the fact that instead of starting with an empty model, we could have started with a partially complete model. In such cases, the human could just provide whatever they know about the task, and the RL agent can fill in the rest. We expect such settings to provide even more advantages to our method. For future work, a promising direction is to support stochastic transitions. One possible way of using such methods in a stochastic setting would be by considering a separate copy of an action for each possible transition, similar to the methods used by many probabilistic planners [39]. The central challenge here would be to recognize the different transitions associated with the same action and to ensure that the estimate at any given point is still an optimistic estimate of the true model. Last, but by no means least, is the combination with RL methods that use function approximation, especially in settings where the symbolic model might be an abstraction of the true underlying model. Such settings are among the most practical ones from the real-world applications perspective, allowing our method to gradually generalize to fine-grained abstractions and eventually to the real world.
Sarath Sreedharan's research is supported in part by grant NSF 2303019.
[1] David Abel. simple_rl: Reproducible Reinforcement Learning in Python. In ICLR Workshop on Reproducibility in Machine Learning, 2019.
[2] Arthur Aubret, Laëtitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforcement learning. CoRR, abs/1908.06976, 2019.
[3] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., 3:397-422, 2002.
[4] Dimitri Bertsekas. Rollout, policy iteration, and distributed reinforcement learning. Athena Scientific, 2021.
[5] Ronen I. Brafman and Moshe Tennenholtz. R-MAX - A general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res., 3:213-231, 2002.
[6] Ethan Callanan, Rebecca De Venezia, Victoria Armstrong, Alison Paredes, Tathagata Chakraborti, and Christian Muise. MACQ: A holistic view of model acquisition techniques. CoRR, abs/2206.06530, 2022.
[7] Jaime G Carbonell and Yolanda Gil. Learning by experimentation: The operator refinement method. In Machine learning, pages 191-213. Elsevier, 1990.
[8] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018.
[9] Stephen Cresswell, Thomas Leo McCluskey, and Margaret Mary West. Acquiring planning domain models using LOCM. Knowl. Eng. Rev., 28(2):195-213, 2013.
[10] Wenfeng Feng, Hankz Hankui Zhuo, and Subbarao Kambhampati. Extracting action sequences from texts based on deep reinforcement learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4064-4070. ijcai.org, 2018.
[11] Guillem Frances, Miquel Ramirez, and Collaborators. Tarski: An AI planning modeling framework. https://github.com/aig-upf/tarski, 2018.
[12] Clement Gehring, Masataro Asai, Rohan Chitnis, Tom Silver, Leslie Pack Kaelbling, Shirin Sohrabi, and Michael Katz. Reinforcement learning for classical planning: Viewing heuristics as dense reward generators. In Proceedings of the Thirty-Second International Conference on Automated Planning and Scheduling, ICAPS, pages 588–596. AAAI Press, 2022.
[13] Giuseppe De Giacomo, Luca Iocchi, Marco Favorito, and Fabio Patrizi. Foundations for restraining bolts: Reinforcement learning with ltf/ldf restraining specifications. In Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling, ICAPS, pages 128-136. AAAI Press, 2019.
[14] Malte Helmert. Concise finite-domain representations for PDDL planning tasks. Artificial Intelligence, 173:503-535, 2009.
[15] IBM. Forbid-Iterative (FI) planner, 2022.
[16] Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2107-2116. PMLR, 10-15 Jul 2018.
[33] Sarath Sreedharan, Utkarsh Soni, Mudit Verma, Siddharth Srivastava, and Subbarao Kambhampati. Bridging the gap: Providing post-hoc symbolic explanations for sequential decision-making problems with inscrutable representations. In The Tenth International Conference on Learning Representations, ICLR. OpenReview.net, 2022.
[34] Roni Stern and Brendan Juba. Efficient, safe, and probably approximately complete learning of action models. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 4405-4411. ijcai.org, 2017.
[35] Terry Winograd. Procedures as a representation for data in a computer program for understanding natural language. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC, 1971.
[36] Danfei Xu, Roberto Martin-Martin, De-An Huang, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Regression planning networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1317-1327, 2019.
[37] Fangkai Yang, Daoming Lyu, Bo Liu, and Steven Gustafson. PEORL: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI, pages 4860-4866. ijcai.org, 2018.
[38] Qiang Yang, Kangheng Wu, and Yunfei Jiang. Learning action models from plan examples using weighted MAX-SAT. Artificial Intelligence, 171(2-3):107-143, 2007.
[39] Sung Wook Yoon, Alan Fern, and Robert Givan. Ff-replan: A baseline for probabilistic planning. In Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling, ICAPS, page 352. AAAI, 2007.
[40] Sung Wook Yoon, Alan Fern, Robert Givan, and Subbarao Kambhampati. Probabilistic planning via determinization in hindsight. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pages 1010-1016. AAAI Press, 2008.
In this appendix, Section B will cover the formal statements and proof sketches for various theoretical results, Section C will cover the implementation details including hyperparameters. Section E will cover the various assumptions made in the work and finally Section F provides an overview of how our approach is used in the context of Q learning.
First result we are interested in establishing is the fact that for any given MDP of the form described in Section 3.1 in the main paper, there exists a corresponding symbolic model that meets the criteria discussed in Section 3.2 in the paper.
Proposition 2 For an MDP of the form  $\mathcal{M} = \langle S, T, A, I, R \rangle$ , there exists a grounded symbolic model  $\mathcal{M}_{\downarrow} = \langle F_{\downarrow}, A_{\downarrow}, I, G \rangle$ , such that there exists
1. a mapping  $\mathcal{C}$  from the state  $S$  of  $\mathcal{M}$  to the states of  $\mathcal{M}_{\downarrow}$ ,
2. a mapping  $\mathcal{C}^{-1}$  from the actions  $A_{\downarrow}$  of  $\mathcal{M}_{\downarrow}$  to the actions  $A$  of  $\mathcal{M}$ , and
3. a mapping from valid traces in  $\mathcal{M}$  to valid plans in  $\mathcal{M}_{\downarrow}$  and vice-versa.
Proof Sketch: We can build such a model by adding one grounded predicate (each corresponding to a unique lifted predicate of arity 0) for each state other than  $\bot$  into  $F_{\downarrow}$ . Now a state in  $S$  maps (defined by  $\mathcal{C}$ ) exactly to the symbolic state where the corresponding predicate is true and none of the other fluents are true. Now for the action, we will start with an action definition that includes conditional effect, and then convert it to a form assumed by the work. A conditional effect captures cases where an effect of an action only fires if the state meets certain criteria. Now we create one symbolic action for each MDP action. For each possible transition between states (other than  $\bot$ ), we will add a corresponding conditional effect that takes the predicate corresponding to source state as condition and as effect the predicate corresponding to the target state. We will keep as precondition of the actions a disjunctive list of all possible states where it will not fail. For the goal action, we will have corresponding symbolic goal action whose precondition corresponds to potential states in  $S_G$  and the effect is a goal predicate. The initial state consists of only the predicate corresponding to the state  $I$  and the goal corresponds to the goal predicate. Now we can convert the actions with conditional effects to ones with no conditional effect (cf. [30]). Now the action mapping  $\mathcal{C}^{-1}$ , will map each of these new actions to the original MDP action from which it was defined. Now a plan is only valid in this model, if there exists a sequence of transitions from initial state to goal with non-zero probability. Similarly, for every valid trace there must exist a valid plan where each MDP action could be replaced by one of the potential symbolic actions that maps to it.
Next we will talk about the optimism of the initial model estimate
Proposition 3  $\mathcal{M}_0^{\mathcal{C}} = \langle F_{\downarrow}^{\mathcal{C}},A_{\downarrow}^{\mathcal{C}},I^{\mathcal{C}},G^{\mathcal{C}}\rangle$  . More formally, every action  $a\in A_{\downarrow}^{\mathcal{C}}$  will be defined as follows:  $a = \langle pre_0^a,\text{add}_0^a,\text{del}_0^a\rangle$  , where pre<sup>a</sup> = del<sup>a</sup> = ∅ and add<sup>a</sup> = F<sup>1</sup> is optimistic for any MDP model such that there exists a mapping C from MDP state to symbolic states and a function  $\mathcal{C}^{-1}$  mapping symbolic actions to MDP actions.
This can be easily shown by the fact that every possible action sequence is a possible plan here.
Moving onto the update rule.
Proposition 4 Update rule as presented in Section 4.1, will only result in an optimistic representation.
Proof Sketch: The important point to note is that at any point, the update rule is only applied to an optimistic representation. So, in order for it to result in a non-optimistic model, it must have removed a plan corresponding to a valid trace. Given our initial construction of  $\mathcal{M}_0^{\mathcal{C}}$ , we always ensure that in  $\mathcal{M}_0^{\mathcal{C}}$  the execution of an action  $a$  at a state  $\mathcal{C}(s)$  will result in a symbolic state that is a superset of  $\mathcal{C}(s')$ , where  $T(s, a, s') = 1$ . Note that an application of an update rule will only extend the precondition if the corresponding MDP action fails and the preconditions are extended to exclude only the current state (though the list of excluded state, action pairs grows as the number of failed samples grows). Additionally, the effect is changed only to disallow impossible transitions. Since the transitions are deterministic, only one sample is needed to determine that no other transitions are possible from that state and action. This means that the above property (the fact that the resultant
symbolic state will be superset) will be preserved through updates. Which in turn means that any plan that previously corresponded to a valid trace can become invalid.
Now coming to the theorem
Theorem 1 Algorithm 1 (as referenced in the main paper) will (a) terminate in a finite number of steps and (b) identify a path to a goal (provided one exists); as long as the diverse planner used is complete (i.e., it will return a non-empty plan set as long as there exists a valid plan).
Proof Sketch The validity of this theorem follows from the fact that the update rule will remove any plan that doesn't correspond to a valid trace from consideration again. If the planner is complete then it will effectively iterate over all possible plans. Eventually finding one that corresponds to a path that goes to the goal. This is guaranteed to exit in finite steps, as the set of non-redundant plans is guaranteed to be finite when the state space is finite.
Now revisiting Proposition 1
Proposition 1 Let  $\bar{A} = \langle a_1, \dots, a_m \rangle$  be a set of actions marked as being instances of a single lifted action  $a^{\uparrow}$ . Then min_add must be a superset of add effects of  $a^{\uparrow}$  and max_del a superset of deletes of  $a^{\uparrow}$ , where min_add and max_del are calculated for  $\bar{A}$
Proof Sketch The validity is trivial. The update rule makes sure that every effect estimate will be an optimistic estimate of the true ground action effects. In the case of add effects this estimate will be a superset and for delete effects it will be a subset. Thus, the lifted representation of each set must correspond to optimistic estimates of the true lifted representation of the effects.
All experiments were on a laptop running Mac OS v 11.06, with 2 GHz Quad-Core Intel Core i5 and 16 GB 3733 MHz LPDDR4X. We did not use CUDA in any of the experiments. For the planner, we used the FI-diverse-agl planner provided as part of the forbid iterative planner. As discussed we generated 10 plans in every planing query. The search was given a maximum threshold of 1000 iterations, but we never reached that limit given our time limit. We stop an action from being considered if it fails 10 times in a row. We will update this upperbound on number of failures if the planner returns empty plan at any point. Since we found out that the planner was slowed down by the introduction of disjunctive preconditions, we replaced the disjunctions with a set of actions (this is an equivalent compilation popular within planning). To control the growth of the precondition, we introduce an upper bound on its size, set to 10 in our experiment. Note that the true size of the preconditions in all instances we consider here is significantly smaller than our bound. We could make the bound adaptive to a domain, but we do not expect it to make any significant difference. For all the RL baselines we used a discount factor of  $\gamma$ . For Q learning and R max, we used a maximum of 1000000 episodes with 200 steps per episode. For exploration, the  $\epsilon$  and decay rates were set as the same as the one used by SimpleRL experiment scripts. For PPO, we used the same default values used by [26]. The environment names for the two problems we tested in minigrid where where, MazeRooms-8by8-DloorKey-v0 and MazeRooms-2by2-TwoKeys-v0. While creating the PDDL model for minigrid we combined the turn actions with the other actions (move, pickup, drop, etc.), to avoid potential conditional effects.
D.1 Comparison with a Symbolic Baseline
To see how our method compares against other methods for symbolic model acquisition, we look at how many samples are required by a popular model learning method (cf. [19]) to generate a model that can produce a goal reaching plan. Since we don't have access to a plan library, we will generate one through random walks on our simulator. We focused on the Blockworld domain, and for each of the five problems, we look at the number of samples required to generate a model that allows a potential plan to the goal. Since the method generates a pessimistic approximation of the model, any plan generated by the model is guaranteed to be valid and thus the method no longer requires the use of diverse planners to generate potential plans. We placed an upper bound of 600000 on the number of samples originally collected from the simulator (this is nearly six times larger than the number
of samples required by our method). Out of the five problems, we found that only the method was only able to learn a model capable of generating a valid plan in the case of the first problem instance. Even for that problem, we found that the method took on average 548287.8 samples.
D.2 Kitchen Domain
As an additional experiment, we tested our method on the symbolic part of the Kitchen Domain [36]. We tested our method on the domain by creating a symbolic simulator that uses the descriptions provided in the appendix of the paper (specifically the inter-dependencies listed in Figure 5). The purely symbolic domain consisted of one action for each high-level goal possible and the preconditions were built based on the relationships described in the paper. The exact domain consisted of 15 predictions and 13 actions. The goal was the same as the one described in the paper (i.e., both banana and cabbage is cooked, they are placed on plates and the plates are served). The lifted version of our method was able to identify a valid plan in 61.46 sec using 24727.2 time steps (averaged across five runs) and the non lifted version took 393.57 secs and used 140681 samples (again averaged across five runs). Now executing the plans in the true simulator would require an additional component an additional step to drive the simulated robot to achieve each of these subgoal. However, as discussed in the paper, we can do that by using a motion level planner (like an RRT based planner).
E Assumptions Made
Here we explicitly mention all the theoretical assumptions we have made in the paper and how to relax them
Model dynamics:
Deterministic model - our primary formulation and evaluation focus on deterministic domains. However, as discussed in the future works section, we can directly apply our method to stochastic environments by creating symbolic models that correspond to so-called all outcome determinizations of the model [39].
Observability:
We assume that the environment is fully observable. However, previous work have looked at how partial observability can be compiled into classical planning model. For optimistic estimates, we can make further simplification to assume that all unobservable facts are true, thus allowing us to directly apply our methods in such settings.
Finite state and action space:
We assume that the underlying state and action space is finite and thus can be represented exactly using a finite symbolic models. For cases where this is not true, we can still employ our symbolic model to capture an abstraction of the true state and action space.
Symbolic observations:
We assume that noise-free classifiers for each symbolic fluent are given. Previous work [32] have looked at the problem of learning such classifiers, which we can directly use in our scenario. If the classifiers are noisy, this corresponds to a special case of partial observability. As discussed above, we therefore can extend our model to handle noisy classifiers.
F Overall Learning System
Here we provide an overview of the overall learning process. Figure 2 presents a diagrammatic representation of the learning process. As discussed before, we start with a trivially optimistic representation of the model. We then use a diverse planner to potentially generate possible plans from that domain, which are then tested on the simulators. The experiences generated from the simulator are then used to update and refine our optimistic representation. Once a successful plan is identified, this information is used to initialize the RL algorithm. For our experiments, we focused on Q learning; as such, the plans were used to initialize the Q values with more informative estimates. It is worth noticing that the problem of learning a useful refinement of an optimistic model involves solving an additional exploration-exploitation problem. Specifically, this involves identifying what actions
Figure 2: A visualization of the overall learning process
the planner should try to include in the plans. While the use of diverse planners already provides us with mechanisms to promote exploration, given the optimistic nature of the model estimates, there is always a possibility that the planner will try to use actions that have not yet been successfully executed (thereby updating its effect). In our current implementation, we employ a queuing strategy to prevent the planner from retrying the same actions too many times.