SlowGuess/NeurIPS_2023_OCR huggingface dataset:
related problem of figuring out which data sets to request labels for, vanilla active learning approaches are not directly applicable here because the possible set of queries expressible using the literals in a domain is vast: it is the set of all policies expressible using  $\mathcal{C}_N$ . Query-based learning approaches use an estimate of the utility of a query to select "good" queries. This can be a multi-valued measure like information gain [Sollich and Saad, 1994], value [Macke et al., 2021], etc. or a binary-valued attribute like distinguishability [Verma et al., 2021], etc. We use distinguishability as a measure to identify useful queries. According to it, a query  $q$  is distinguishing w.r.t. two models if responses by both models to  $q$  are logically inconsistent. We now discuss methods for generating such queries.
Generating distinguishing queries QACE automates the generation of queries using search. As part of the algorithm, a model  $M^{*}$  is used to generate the three possible models  $M_{T}, M_{F},$  and  $M_{I}$  for a specific predicate  $p$  and location  $l$  combination. Other than the predicate  $p$  at location  $l$ , these models are exactly the same. A forward search is used to generate the policy simulation queries with two possible models  $M_{i}, M_{j}$  chosen randomly from  $M_{T}, M_{F},$  and  $M_{I}$ . The forward search is initiated with an initial state  $\langle s_0^i, s_0^j \rangle$  as the root of the search tree, where  $s_0^i$  and  $s_0^j$  are copies of the same state  $s_0$  from which we are starting the search. The edges of the tree correspond to the capabilities with arguments replaced with objects in the environment. Nodes correspond to the two states resulting from applying the capability in the parent state according to the two possible models. E.g., consider that a transition  $\langle s_0^i, c, s_1^i \rangle$  is possible according to the model  $M_{i}$ , and let  $\langle s_0^j, c, s_1^j \rangle$  be the corresponding transition (by applying the same effect set of  $c$  as  $h_i$ ) according to the model  $M_j$ . Now there will be an edge in the forward search tree with label  $c$  such that parent node is  $\langle s_0^i, s_0^j \rangle$  and child node is  $\langle s_1^i, s_1^j \rangle$ . The search process terminates when a node  $\langle s^i, s^j \rangle$  is reached such that either the states  $s^i$  and  $s^j$  don't match, or the preconditions of the same capability were met in the state according to one of the possible models but not according to the other. Vanilla forward search scales poorly with the number of capabilities and objects in the environment. To address this we reduce the problem to a fully observable non deterministic (FOND) planning problem. This can be solved by any FOND planner. The output of this search is a policy  $\pi$  to reach a state where the two models,  $M_i$  and  $M_j$  predict different outcomes. Additional details about the reduction and an example of the output policy are available in the extended version of the paper [Verma et al., 2023]. The query  $\langle s_I, \pi, G, \alpha, \eta \rangle$  resulting from this search is such that  $s_I$  is set to the initial state  $s_0, \pi$  is the output policy,  $G$  is the goal state where the models' responses doesn't match,  $\alpha$  and  $\eta$  are hyperparameters as mentioned earlier. We next see how to use these queries to prune out the incorrect models.
4.3 Learning Probabilistic Models Using Query Responses
At this point, QACE already has a query such that the response to the query by the two possible models does not match. We next see how to prune out the model inconsistent with the SDMA. QACE poses the query generated earlier to the SDMA and gets its response (line 7 in Alg. 1). If the SDMA can successfully execute the policy, QACE compares the response of the two models with that of the SDMA and prunes out the model whose response does not match with that of the SDMA. If the SDMA cannot execute the policy, i.e., SDMA fails to execute some capability in the policy, then the models cannot be pruned directly. In such a case, a new initial state  $s_0$  must be chosen to generate a new query starting from that initial state. Since generating new queries for the same pair of models can be time consuming, we preempt this issue by creating a pool of states  $\mathbb{S}$  that can execute the capabilities using a directed exploration of the state space with the current partially learned model as discussed below.
Directed exploration A partially learned model is a model where one or more capabilities have been learned (the correct preconditions have been identified for each capability and at least one effect is learned). Once we have such a model, we can do a directed exploration of the state space for these capabilities by only executing a learned capability if the preconditions are satisfied. This helps in reducing the sample complexity since the simulator is only called when we know that the capability will execute successfully, thereby allowing us to explore different parts of the state space efficiently. If a capability's preconditions are not learned, all of its groundings might need to be executed from the state. In the worst case, to escape local minima where no models can be pruned, we would need to perform a randomized search for a state where a capability is executable by the SDMA. In practice, we observed that using directed exploration to generate a pool of states gives at least one grounded capability instance. This helps ensure that during query generation, the approach does not spend a long time searching for a state where a capability is executable.
Learning probabilities of stochastic effects After QACE learns the non-deterministic model, to learn the probabilities of the learned effects it uses the transitions collected as part of responses to queries. This is done using Maximum Likelihood Estimation (MLE) [Fisher, 1922]. For each triplet  $\langle s, c, s' \rangle$  seen in the collected data, let  $count_{c}$  be the number of times a capability  $c$  is observed. Now, for each effect set, the probability of that effect set becoming true on executing that capability  $c$  is given as the number of times that effect is observed on executing  $c$  divided by  $count_{c}$ . As we increase the value of the hyperparameter  $\eta$ , we increase the number of collected triplets, thereby improving the probability values calculated using this approach.
5 Theoretical Analysis and Correctness
We now discuss how the model  $M$  of SDMA  $\mathcal{A}$  learned using QACE fulfills the notions of correctness (Sec. 3) discussed earlier. We first show that the model  $M^{*}$  learned before line 10 of QACE (Alg. 1) is sound and complete according to Def. 2. The proofs for the theorems are available in the extended version of the paper [Verma et al., 2023].
Theorem 1. Let  $\mathcal{A}$  be a black-box SDMA with a ground truth transition model  $\mathcal{T}'$  expressible in terms of predicates  $\mathcal{P}$  and a set of capabilities  $\mathcal{C}$ . Let  $M^{*}$  be the non-deterministic model expressed in terms of predicates  $\mathcal{P}^{*}$  and capabilities  $\mathcal{C}$ , and learned using the query-based autonomous capability estimation algorithm (Alg. 1) just before line 10. Let  $C_N$  be a set of capability names corresponding to capabilities  $\mathcal{C}$ . If  $\mathcal{P}^{*} \subseteq \mathcal{P}$ , then the model  $M^{*}$  is sound w.r.t. the SDMA transition model  $\mathcal{T}'$ . Additionally, if  $\mathcal{P}^{*} = \mathcal{P}$ , then the model  $M^{*}$  is complete w.r.t. the SDMA transition model  $\mathcal{T}'$ .
Next, we show that the final step of learning the probabilities for all the effects in each capability converges to the correct probability distribution under the assumption that all the effects of a capability are identifiable. When a capability  $c$  is executed in the environment, one of its effects  $e_i(c) \in \text{eff}(c)$  will be observed in the environment. To learn the correct probability distribution in  $M$ , we should accurately identify that effect  $e_i(c)$ . Hence, the set of effects is identifiable if at least one state exists in the environment from which each effect can be uniquely identified when the capability is executed. An example of this is available in the extended version of the paper [Verma et al., 2023].
Theorem 2. Let  $\mathcal{A}$  be a black-box SDMA with a ground truth transition model  $\mathcal{T}'$  expressible in terms of predicates  $\mathcal{P}$  and a set of capabilities  $\mathcal{C}$ . Let  $M$  be the probabilistic model expressed in terms of predicates  $\mathcal{P}^*$  and capabilities  $\mathcal{C}$ , and learned using the query-based autonomous capability estimation algorithm (Alg. 1). Let  $\mathcal{P} = \mathcal{P}^*$  and  $M$  be generated using a sound and complete nondeterministic model  $M^*$  in line 11 of Alg. 1, and let all effects of each capability  $c\in \mathcal{C}$  be identifiable. The model  $M$  is correct w.r.t. the model  $\mathcal{T}'$  in the limit as  $\eta$  tends to  $\infty$ , where  $\eta$  is hyperparameter in query  $Q_{\mathrm{PS}}$  used in Alg. 1.
6 Empirical Evaluation
We implemented Alg. 1 in Python to evaluate our approach empirically. We found that our query synthesis and interactive learning process leads to (i) few shot generalization; (ii) convergence to a sound and complete model; and (iii) much greater sample efficiency and accuracy for learning lifted SDM models with complex capabilities as compared to the baseline.
Setup We used a single training problem with few objects  $(\leq 7)$  for all methods in our evaluation and used a test set that was composed of problems containing object counts larger than those in the training set. We ran the experiments on a cluster of Intel Xeon E5-2680 v4 CPUs with CentOS 7.9 running at  $2.4\mathrm{GHz}$  with a memory limit of 8 GB and a time limit of 4 hours. We used PRP [Muisie et al., 2012] as the FOND planner to generate the queries (line 6 in Alg. 1). For QACE, we used  $\alpha = 2d$  where  $d$  is the maximum depth of policies used in queries generated by QACE and  $\eta = 5$ . All of the methods in our empirical evaluation receive the same training and test sets and are evaluated on the same platform. We used Variational Distance (VD) as presented in Eq. 1 to evaluate the quality of the learned SDMA models.
Baseline selection We used the closest SOTA related work, GLIB [Chitnis et al., 2021] as a baseline. GLIB learns a probabilistic model of an intrinsically motivated agent by sampling goals far away from the initial state and making the agent try to reach them. This can be adapted to an assessment setting
Source code available at https://github.com/AAIR-lab/QACE
Figure 3: Screen captures from the Cafe Server Robot simulation. The complete environment is shown in the image on the left. The image grid on the right shows screen captures of multiple steps of the robot delivering a soda-can to a table.
by moving goal-generation based sampling outside the agent, and, to the best of our knowledge, no existing approach addresses the problem of creating intelligent questions for an SDMA. GLIB has two versions, GLIB-G, which learns the model as a set of grounded noisy deictic rules (NDRs) [Pasula et al., 2007], and GLIB-L, which learns the model as a set of lifted NDRs. We used the same hyperparameters as published for the Warehouse Robot and Driving Agent and performed extensive tuning for the others and report results with the best performing settings.
The models learned using GLIB cannot be used to calculate the variational distance presented in Eq. 1 because for each capability GLIB learns a set of NDRs rather than a unique NDR. In order to maintain parity in comparison, we use GLIB's setup to calculate an approximation of the VD. Using it, we sample 3500 random transitions  $\langle s, c, s' \rangle$  from the ground truth transition model  $\mathcal{T}'$  using problems in the test set to compute a dataset of transitions  $\mathcal{D}$ . The sample-based, approximate VD is then given as:  $\frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} \mathbb{1}_{[s' \neq c_M(s)]}$ , where  $c_M(s)$  samples the transition using the capability in the learned model output by each method. In Fig. 5, we compare the approximate variational distance of the three approaches w.r.t.  $\mathcal{D}$  as we increase the learning time. Note that we also evaluated VD for QACE using Eq. 1 and found that  $\delta(\mathcal{T}', M) \approx 0$  for our learned model  $M$  in all SDMA settings.
SDMAs for evaluation To test the efficacy of our approach, we created SDMAs for five different settings including one task and motion planning agent and several SDMAs based on state-of-the-art stochastic planning systems from the literature: Cafe Server Robot is a Fetch robot [Wise et al., 2016] that uses the ATM-MDP task and motion planning system [Shah et al., 2020] to plan and act in a restaurant environment to serve food, clear tables, etc.; Warehouse Robot is a robot that can stack, unstack, and manage the boxes in a warehouse; a Driving Agent that can drive between locations and can repair the vehicle at certain locations; a First Responder Robot that can assist in emergency scenarios by driving to emergency spots, providing first-aid and water to victims, etc.; and an Elevator Control Agent that can control the operation of multiple elevators in a building.
Additional details about each setting are available in the extended version [Verma et al., 2023].
We present an analysis of our approach on all of the SDMA's listed above. We also present a comparative analysis with the baseline on all SDMA's except the Cafe Server Robot, whose task and motion planning system was not compatible with the baseline.
Cafe Server Robot This SDMA setup uses an 8 degrees of freedom Fetch [Wise et al., 2016] robot in a cafe setting on OpenRave simulator [Diankov and Kuffner, 2008]. The low-level environment state consists of continuous x, y, z, roll, pitch, and yaw values of all objects in the environment. The predicate evaluators were provided by ATM-MDP of which we used only a subset to learn a PPDDL model. Each robot capability is refined into motion controls at run-time depending on the configuration of the objects in the environment. The results for variational distance between the learned model and the ground truth model in Fig. 4 show that despite the different vocabulary, QACE learns an accurate transition model for the SDMA.
We now discuss the comparative performance of QACE with the baseline across the four baseline-compatible SDMAs presented above.
Figure 5: A comparison of the approximate variational distance as a factor of the learning time for the three methods: QACE (ours), GLIB-G, and GLIB-L (lower values better).  $\times$  shows that the learning process ended at that time instance for QACE. The results were calculated using 30 runs per method per domain. Solid lines are averages across runs, and shaded portions show the standard deviation.  $\mathcal{T}^{\prime}$  is the ground truth model. Detailed results are available in Verma et al. [2023].
Faster convergence The time taken for QACE to learn the final model is much lower than that of GLIB for three of the four SDMAs. This is because trace collection by QACE is more directed and hence ends up learning the correct model in a shorter time. The only setup where GLIB marginally outperforms QACE is Warehouse Robot, and this happens because this SDMA has just two capabilities, one of which is deterministic. Hence, GLIB can easily learn their configuration from a few observed traces. For SDMAs with complex and much larger number of capabilities – First Responder Robot and Elevator Control Agent – GLIB finds it more challenging to learn the model that is closer to the ground truth transition model. Additionally, QACE takes much fewer samples to learn the model than the baselines. In all settings, QACE is much more sample efficient than the baselines as QACE needed at most  $4\%$  of the samples needed by GLIB-G to reach the variational distance that it plateaued at. In contrast, GLIB-L started timing out only after processing a few samples for complex SDMAs.
Figure 4: Variational Distance between the learned model and the ground truth with increasing time for QACE for Cafe Server Robot.  $\times$  shows that the learning process ended at that time instance.
Few-shot generalization To ensure that learned models are
not overfitted, our test set contains problems with larger quantities of objects than those used during training. As seen in Fig. 5, the baselines have higher variational distance from the ground truth model for complex SDMA setups as compared to QACE. This shows better few-shot generalization of QACE compared to the baselines.
The problem of learning probabilistic relational agent models from a given set of observations has been well studied [Pasula et al., 2007, Mourão et al., 2012, Martínez et al., 2016, Juba and Stern, 2022]. Jiménez et al. [2012] and Arora et al. [2018] present comprehensive reviews of such approaches. We next discuss the closest related research directions.
Passive learning Several methods learn a probabilistic model of the agent and environment from a given set of agent executions. Pasula et al. [2007] learn the models in the form of noisy deictic rules (NDRs) where an action can correspond to multiple NDRs and also model noise. Mourão et al. [2012] learn such operators using action classifiers to predict the effects of an action. Rodrigues et al. [2011] learn non-deterministic models as a collection of rule sets and learn these rule sets incrementally. They take a bound on the number of rules as input. Juba and Stern [2022] provide a theoretical framework to learn safe probabilistic models with a range of probabilities for each probabilistic effect while assuming that each effect is atomic and independent of others. A common issue with such approaches is that they are susceptible to incorrect and sometimes inefficient model
learning as they cannot control the input data used for learning or carry out interventions required for accurate learning.
Sampling of transitions Several approaches learn operator descriptions by exploring the state space in the restricted setting of deterministic models [Ng and Petrick, 2019, Jin et al., 2022]. A few reinforcement learning approaches have been developed for learning the relational probabilistic action model by exploring the state space using pre-determined criteria to generate better samples [Ng and Petrick, 2019]. Konidaris et al. [2018] explore learning PPDDL models for planning, but they aim to learn the high-level symbols needed to describe a set of input low-level options, and these symbols are not interpretable. GLIB [Chitnis et al., 2021] also learns probabilistic relational models using goal sampling as a heuristic for generating relevant data, whereas we use active querying using guided forward search for this. Our empirical analysis shows that our approach of synthesising queries yield greater sample efficiency and correctness profiles than the goal generation used in this approach.
Active learning Several active learning approaches learn automata representing a system's model [Angluin, 1988, Aarts et al., 2012, Pacharoen et al., 2013, Vaandrager, 2017]. These approaches assume access to a teacher (or an oracle) that can determine whether the learned automaton is correct and provide a counterexample if it is incorrect. This is not possible in the black-box SDMA settings that constitute the focus of this work.
In this work, we presented an approach for learning a probabilistic model of an agent using interactive querying. We showed that the approach is few-shot generalizable to larger environments and learns a sound and complete model faster than state-of-the-art approaches in a sample-efficient manner.
QACE describes the capabilities of the robot in terms of predicates that the user understands (this includes novice users as well as more advanced users like engineers). Understanding the limits of the capabilities of the robot can help with the safe usage of the robot, and allow better utilization of the capabilities of the robot. Indirectly, this can reduce costs since the robot manufacturer need not consider all possible environments that the robot may possibly operate in. The use of our system can also be extended to formal verification of SDMAs.
QACE can also be used by standard explanation generators as they need an agent's model. Such models are hard to obtain (as we also illustrate in this paper) and our approach can be used to compile them when they are not available to start with.
Limitations and Future Work In this work, we assume that the agent can be connected to a simulator to answer the queries. In some real-world settings, this assumption may be limiting as users might not have direct access to such a simulator. Formalizing the conditions under which it is safe to ask the queries directly to the agent in the real-world is a promising direction for the future work. Additionally, in this work, we assume the availability of the instruction set of the SDMA as input in the form of capability names. In certain settings, it might be useful to discover the capabilities of an evolving SDMA using methods proposed by Nayyar et al. [2022] and Verma et al. [2022].
We thank Jayesh Nagpal for his help with setting up the Cafe Server Robot SDMA. We also thank anonymous reviewers for their valuable feedback and suggestions. This work was supported by the ONR under grants N00014-21-1-2045 and N00014-23-1-2416.
Fides Aarts, Faranak Heidarian, Harco Kuppens, Petur Olsen, and Frits Vaandrager. Automata learning through counterexample guided abstraction refinement. In Proceedings of the 18th International Symposium on Formal Methods, 2012.
Dana Angluin. Queries and concept learning. Machine Learning, 2(4):319-342, 1988.
Ankuj Arora, Humbert Fiorino, Damien Pellier, Marc Métivier, and Sylvie Pesty. A review of learning planning action models. Knowledge Engineering Review, 33:E20, 2018.
Pulkit Verma, Rushang Karia, and Siddharth Srivastava. Autonomous capability assessment of sequential decision-making systems in stochastic settings (Extended Version). arXiv:2306.04806, 2023.
Melonee Wise, Michael Ferguson, Derek King, Eric Diehr, and David Dymesich. Fetch and freight: Standard platforms for service robot applications. In *IJCAI Workshop on Autonomous Mobile Service Robots*, 2016.
Håkan L. S. Younes and Michael L. Littman. PPDDL 1.0: An extension to PDDL for expressing domains with probabilistic effects. Technical Report CMU-CS-04-167, Carnegie Mellon University, USA, October 2004.
Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online bayesian goal inference for boundedly-rational planning agents. In Advances in Neural Information Processing Systems, 2020.