SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pulkit Verma, Rushang Karia, and Siddharth Srivastava
Autonomous Agents and Intelligent Robots Lab,
School of Computing and Augmented Intelligence,
Arizona State University, AZ, USA
{verma.pulkit, rushang.karia, siddharths}@asu.edu
It is essential for users to understand what their AI systems can and can't do in order to use them safely. However, the problem of enabling users to assess AI systems with sequential decision-making (SDM) capabilities is relatively understudied. This paper presents a new approach for modeling the capabilities of black-box AI systems that can plan and act, along with the possible effects and requirements for executing those capabilities in stochastic settings. We present an active-learning approach that can effectively interact with a black-box SDM system and learn an interpretable probabilistic model describing its capabilities. Theoretical analysis of the approach identifies the conditions under which the learning process is guaranteed to converge to the correct model of the agent; empirical evaluations on different agents and simulated scenarios show that this approach is few-shot generalizable and can effectively describe the capabilities of arbitrary black-box SDM agents in a sample-efficient manner.
AI systems are becoming increasingly complex, and it is becoming difficult even for AI experts to ascertain the limits and capabilities of such systems, as they often use black-box policies for their decision-making process [Popov et al., 2017, Greydanus et al., 2018]. E.g., consider an elderly couple with a household robot that learns and adapts to their specific household. How would they determine what it can do, what effects their commands would have, and under what conditions? Although we are making steady progress on learning for sequential decision-making (SDM), the problem of enabling users to understand the limits and capabilities of their SDM systems is largely unaddressed. Moreover, as the example above illustrates, the absence of reliable approaches for user-driven capability assessment of AI systems limits their inclusivity and real-world deployability.
This paper presents a new approach for Query-based Autonomous Capability Estimation (QACE) of black-box SDM systems in stochastic settings. Our approach uses a restricted form of interaction with the input SDM agent (referred to as SDMA) to learn a probabilistic model of its capabilities. The learned model captures high-level user-interpretable capabilities, such as the conditions under which an autonomous vehicle could back out of a garage, or reach a certain target location, along with the probabilities of possible outcomes of executing each such capability. The resulting learned models directly provide interpretable representations of the scope of SDMA's capabilities. They can also be used to enable and support approaches for explaining SDMA's behavior that require closed-form models (e.g., Sreedharan et al. [2018]). We assume that the input SDMA provides a minimal query-response interface that is already commonly supported by contemporary SDM systems. In particular, SDMA should reveal capability names defining how each of its capabilities
can be invoked, and it should be able to accept user-defined instructions in the form of sequences of such capabilities. These requirements are typically supported by SDM systems by definition.
The main technical problem for QACE is to automatically compute "queries" in the form of instruction sequences and policies, and to learn a probabilistic model for each capability based on SDMA's "responses" in the form of executions. Depending on the scenario, these executions can be in the real world, or in a simulator for safety-critical settings. Since the set of possible queries of this form is exponential in the state space, naive approaches for enumerating and selecting useful queries based on information gain metrics are infeasible.
Main contributions This paper presents the first approach for query-based assessment of SDMAs in stochastic settings with minimal assumptions on SDMA internals. In addition, it is also the first approach for reducing query synthesis for SDMA assessment to full-observable non-deterministic (FOND) planning [Cimatti et al., 1998]. Empirical evaluation shows that these contributions enable our approach to carry out scalable assessment in both embodied and vanilla SDMAs.
We express the learned models using an input concept vocabulary that is known to the target user group. Such vocabularies span multiple tasks and environments. They can be acquired through parallel streams of research on interactive concept acquisition [Kim et al., 2015, Lage and Doshi-Velez, 2020] or explained to users through demonstrations and training [Schulze et al., 2000]. These concepts can be modeled as binary-valued predicates that have their associated evaluation functions [Mao et al., 2022]. We use the syntax and semantics of a well-established relational SDM model representation language, Probabilistic Planning Domain Definition Language (PPDDL) [Younes and Littman, 2004], to express the learned models.
Related work on the problem addresses model learning from passively collected observations of agent behavior [Pasula et al., 2007, Martínez et al., 2016, Juba and Stern, 2022]; and by exploring the state space using simulators [Chitnis et al., 2021, Mao et al., 2022]. However, passive learning approaches can learn incorrect models as they do not have the ability to generate interventional or counterfactual data; exploration techniques can be sample inefficient because they don't take into account uncertainty and incompleteness in the model being learned to guide their exploration (see Sec. 7 for a greater discussion).
In addition to the key contributions mentioned earlier, our results (Sec. 6) show that the approaches for query synthesis in this paper do not place any additional requirements on black-box SDMAs but significantly improve the following factors: (i) convergence rate and sample efficiency for learning relational models of SDMAs with complex capabilities, (ii) few-shot generalizability of learned models to larger environments, and (iii) accuracy of the learned model w.r.t. the ground truth SDMA capabilities. convergence rate to the sound and complete model.
SDMA setup We consider SDMA's that operate in stochastic and fully observable environments. An SDMA can be represented as a 3-tuple  $\langle \mathcal{X},\mathcal{C},\mathcal{T}\rangle$ , where  $\mathcal{X}$  is the environment state space that the SDMA operates in,  $\mathcal{C}$  is the set of SDMA's capabilities (capability names, e.g., "place object x at location y" or "arrange table x") that the SDMA can execute, and  $\mathcal{T}:\mathcal{X}\times \mathcal{C}\to \mu \mathcal{X}$  is the stochastic black-box transition model determining the effects of SDMA's capabilities on the environment. Here,  $\mu \mathcal{X}$  is the space of probability distributions on  $\mathcal{X}$ . Note that the semantics of  $\mathcal{C}$  are not known to the user(s) and  $\mathcal{X}$  may not be user-interpretable. The only information available about the SDMA is the instruction set in the form of capability names, represented as  $\mathcal{C}_N$ . This isn't a restricting assumption as the SDMA's must reveal their instruction sets for usability.
Running Example Consider a cafe server robot that can pick and place items like plates, cans, etc., from various locations in the cafe, like the counter, tables, etc., and also move between these locations. A capability pick-item (?location ?item) would allow a user to instruct the robot to pick up an item like a soda can for any location. However, without knowing its description, the user would not know under what conditions the robot could execute this capability and what the effects will be.
Object-centric concept representation We aim to learn representations that are generalizable, i.e., the transition dynamics learned should be impervious to environment-specific properties such as numbers and configurations of objects. Additionally, the learned capability models should hold in different settings of objects in the environment as long as the SDMA's capabilities does not change. To this
effect, we learn the SDMA's transition model in terms of interpretable concepts that can be represented using first-order logic predicates. This is a common formalism for expressing the symbolic models of SDMAs [Zhi-Xuan et al., 2020, Mao et al., 2022]. We formally represent them using a set of object-centric predicates  $\mathcal{P}$ . The set of predicates used for cafe server robot in Fig. 1 can be (robot-at ?location), (empty-arm), (has-charge), (at ?location ?item), and (holding ?item). Here, ? precedes an argument that can be replaced by an object in the environment. E.g., (robot-at tableRed) means "robot is at the red table." As mentioned
earlier, we assume these predicates along with their Boolean evaluation functions (which evaluate to true if predicate is true in a state) are available as input. Learning such predicates is also an interesting but orthogonal direction of research [Mao et al., 2022, Sreedharan et al., 2022, Das et al., 2023].
Abstraction Using an object-centric predicate representation induces an abstraction of environment states  $\mathcal{X}$  to high-level logical states  $S$  expressible in predicate vocabulary  $\mathcal{P}$ . This abstraction can be formalized using a surjective function  $f: \mathcal{X} \to S$ . E.g., in the cafe server robot, the concrete state  $x$  may refer to roll, pitch, and yaw values. On the other hand, the abstract state  $s$  corresponding to  $x$  will consist of truth values of all the predicates [Srivastava et al., 2014, 2016, Mao et al., 2022].
Probabilistic transition model Abstraction induces an abstract transition model  $\mathcal{T}':\mathcal{S}\times \mathcal{C}\to \mu \mathcal{S}$ , where  $\mu \mathcal{S}$  is the space of probability distributions on  $\mathcal{S}$ . This is done by converting each transition  $\langle x,c,x^{\prime}\rangle \in \mathcal{T}$  to  $\langle s,c,s^{\prime}\rangle \in \mathcal{T}^{\prime}$  using predicate evaluators such that  $f(x) = s$  and  $f(x^{\prime}) = s^{\prime}$ . Now,  $\mathcal{T}'$  can be expressed as model  $M$  that is a set of parameterized action (capability in our case) schema, where each  $c\in \mathcal{C}$  is described as  $c = \langle name(c),pre(c),eff(c)\rangle$ , where  $name(c)\in C_N$  refers to name and arguments (parameters) of  $c$ ;  $pre(c)$  refers to the preconditions of the capability  $c$  represented as a conjunctive formula defined over  $\mathcal{P}$  that must be true in a state to execute  $c$ ; and  $eff(c)$  refers to the set of conjunctive formulas over  $\mathcal{P}$ , each of which becomes true on executing  $c$  with an associated probability. The result of executing
$c$  for a model  $M$  is a state  $c(s) = s'$  such that  $P_M(s'|s, c) > 0$  and one (and only one) of the effects of  $c$  becomes true in  $s'$ . We also use  $\langle s, c, s' \rangle$  triplet to refer to  $c(s) = s'$ . This representation is similar to the Probabilistic Planning Domain Definition Language (PPDDL), which can compactly describe the SDMA's capabilities. E.g., the cafe server robot has three capabilities (shown here as name(args)): pick-item(?location ?item); place-item(?location ?item); and move (?source ?destination). The description of pick-item in PPDDL is shown in Fig. 2.
Variational Distance Given a black-box SDMA  $\mathcal{A}$ , we learn the probabilistic model  $M$  representing its capabilities. To measure how close  $M$  is to the true SDMA transition model  $\mathcal{T}'$ , we use variational distance - a standard measure in probabilistic-model learning literature [Pasula et al., 2007, Martínez et al., 2016, Ng and Petrick, 2019, Chitnis et al., 2021]. It is based on the total variation distance between two probability distributions  $\mathcal{T}'$  and  $M$ , given as:
$$
\delta \left(\mathcal {T} ^ {\prime}, M\right) = \frac {1}{| \mathcal {D} |} \sum_ {\langle s, c, s ^ {\prime} \rangle \in \mathcal {D}} \left| P _ {\mathcal {T} ^ {\prime}} \left(s ^ {\prime} \mid s, c\right) - P _ {M} \left(s ^ {\prime} \mid s, c\right) \right| \tag {1}
$$
where  $\mathcal{D}$  is the set of test samples ( $\langle s, c, s' \rangle$  triplets) that we generate using  $\mathcal{T}'$  to measure the accuracy of our approach. As shown by Pinsker [1964],  $\delta(\mathcal{T}', M) \leq \sqrt{0.5 \times D_{KL}(\mathcal{T}' \| M)}$ , where  $D_{KL}$  is the KL divergence.
3 The Capability Assessment Task
In this work, we aim to learn a probabilistic transition model  $\mathcal{T}'$  of a black-box SDMA as a model  $M$ , given a set of user-interpretable concepts as predicates  $\mathcal{P}$  along with their evaluation functions, and the capability names  $C_N$  corresponding to the SDMA's capabilities. Formally, the assessment task is:
Figure 1: The cafe server robot environment in OpenRave simulator.
Figure 2: PPDDL description for the cafe server robot's pick-item capability.
(:capability pick-item
:parameters (?location ?item)
:precondition (and
(empty-arm) (has-charge)
(robot-at ?location)
(at ?location ?item))
:effect (and (probabilistic
0.7 (and (not (empty-arm))
(not (at ?location ?item))
(holding ?item))
0.2 (and (not (has-charge)))
0.1(and)) #No-change
Definition 1. Given a set of predicates  $\mathcal{P}$  along with their Boolean evaluation functions, capability names  $\mathcal{C}_N$ , and a black-box SDMA  $\mathcal{A}$  in a fully observable, stochastic, and static environment, the capability assessment task  $\langle \mathcal{A}, \mathcal{P}, \mathcal{C}_N, \mathcal{T}' \rangle$  is defined as the task of learning the probabilistic transition model  $\mathcal{T}'$  of the SDMA  $\mathcal{A}$  expressed using  $\mathcal{P}$ .
The solution to this task is a model  $M$  that should ideally be the same as  $\mathcal{T}'$  for correctness. In practice,  $\mathcal{T}'$  need not be in PPDDL, so the correctness should be evaluated along multiple dimensions.
Notions of model correctness As discussed in Sec. 2, variational distance is one way to capture the correctness of the learned model. This is useful when the learned model and the SDMA's model are not in the same representation. The correctness of a model can also be measured using qualitative properties such as soundness and completeness. The learned model  $M$  should be sound and complete w.r.t. the SDMA's high-level model  $\mathcal{T}'$ , i.e., for all combinations of  $c, s$ , and  $s'$ , if a transition  $\langle s, c, s' \rangle$  is possible according to  $\mathcal{T}'$ , then it should also be possible under  $M$ , and vice versa. Here,  $\langle s, c, s' \rangle$  is consistent with  $M$  (or  $\mathcal{T}'$ ) if  $P(s'|s, c) > 0$  according to  $M$  (or  $\mathcal{T}'$ ). We formally define this as:
Definition 2. Let  $\langle \mathcal{A},\mathcal{P},\mathcal{C}_N,\mathcal{T}\rangle$  be a capability assessment task with a learned model  $M$  as its solution.  $M$  is sound iff each transition  $\langle s,c,s^{\prime}\rangle$  consistent with  $M$  is also consistent with  $\mathcal{T}'$ .  $M$  is complete iff every transition that is consistent with  $\mathcal{T}'$  is also consistent with  $M$ .
This also means that if  $\mathcal{T}'$  is also a PPDDL model, then (i) any precondition or effect learned as part of  $M$  is also present in  $\mathcal{T}'$  (soundness), and; (ii) all the preconditions and effects present in  $\mathcal{T}'$  should be present in  $M$  (completeness). Additionally, a probabilistic model is correct if it is sound and complete, and the probabilities for each effect set in each of its capabilities are the same as that of  $\mathcal{T}'$ .
4 Interactive Capability Assessment
To solve the capability assessment task, we must identify the preconditions and effects of each capability in terms of conjunctive formulae expressed over  $\mathcal{P}$ . At a very high-level, we do this by identifying that a probabilistic model can be expressed as a set of capabilities  $c \in C$ , each of which has two places where we can add a predicate  $p$ , namely precondition and effect. We call these locations within each capability. We then enumerate through these  $2 \times |\mathcal{C}|$  locations and figure out the correct form of each predicate at each of those locations. To do this we need to consider three forms: (i) adding it as  $p$ , i.e., the predicate must be true for that capability to execute (when the location is precondition), or it becomes true on executing it (when the location is effect); (ii) adding it as not  $(p)$ , i.e., the predicate must be false for that capability to execute (when the location is precondition), or it becomes false on executing it (when the location is effect); (iii) not adding it at all, i.e., the capability execution does not depend on it (when the location is precondition), or the capability does not modify it (when the location is effect).
Model pruning Let  $\mathcal{M}$  represent the set of all possible transition models expressible in terms of  $\mathcal{P}$  and  $\mathcal{C}$ . We must prune the set of possible models to solve the capability assessment task, ideally bringing it to a singleton. We achieve this by posing queries to the SDMA and using the responses to the queries as data to eliminate the inconsistent models from the set of possible models  $\mathcal{M}$ .
Given a location (precondition or effect in a capability), the set of models corresponding to a predicate will consist of 3 transition models: one each corresponding to the three ways we can add the predicate in that location. We call these three possible models  $M_T$ ,  $M_F$ ,  $M_I$ , corresponding to adding  $p$  (true), not  $p$  (false), and not adding  $p$  (ignored), respectively at that location.
Note that the actual set of possible transition models is infinite due to the probabilities associated with each transition. To simplify this, we first constrain the set of possible models by ignoring the probabilities, and learn a non-deterministic transition model (commonly referred to as a FOND model [Cimatti et al., 1998]) instead of a probabilistic one. We later learn the probabilities using maximum likelihood estimation based on the transitions observed as part of the query responses.
Simulator use Using the standard assumption of a simulator's availability in research on SDM, QACE solves the capability assessment task (Sec. 3) by issuing queries to the SDMA and observing its responses in the form of its execution in the simulator. In non-safety-critical scenarios, this approach can work without a simulator too. The interface required to answer the queries is rudimentary as the SDMA  $\mathcal{A}$  need not have access to its transition model  $\mathcal{T}'$  (or  $\mathcal{T}$ ). Rather, it should be able to interact with the environment (or a simulator) to answer the queries. We next present the types of queries
we use, followed by algorithms for generating them and for inferring the SDMA's model using its responses to the queries.
Policy simulation queries  $(Q_{PS})$  These queries ask the SDMA  $\mathcal{A}$  to execute a given policy multiple times. More precisely, a  $Q_{PS}$  query is a tuple  $\langle s_I,\pi ,G,\alpha ,\eta \rangle$  where  $s_I\in S$  is a state,  $\pi$  is a partial policy that maps each reachable state to a capability,  $G$  is a logical predicate formula that expresses a stopping condition,  $\alpha$  is an execution cutoff bound representing the maximum number of execution steps, and  $\eta$  is an attempt limit. Note that the query (including the policy) is created entirely by our solution approach without any interaction with the SDMA.  $Q_{PS}$  queries ask  $\mathcal{A}$  to execute  $\pi ,\eta$  times. In each iteration, execution continues until either the stopping goal condition  $G$  or the execution bound  $\alpha$  is reached. E.g., "Given that the robot, soda-can, plate1, bowl13 are at table4, what will happen if the robot follows the following policy: if there is an item on the table and arm is empty, pick up the item; if an item is in the hand and location is not dishwasher, move to the dishwasher; if an item is in the hand and location is dishwasher, place the item in the dishwasher?" Such queries will be used to learn both preconditions and effects (Sec. 4.3).
A response to such queries is an execution in the simulator and  $\eta$  traces of these simulator executions. Formally, the response  $\theta_{PS}$  for a query  $q_{PS} \in Q_{PS}$  is a tuple  $\langle b, \zeta \rangle$ , where  $b \in \{\top, \bot\}$  indicates weather if the SDMA reached a goal state  $s_G \models G$ , and  $\zeta$  are the corresponding triplets  $\langle s, c, s' \rangle$  generated during the  $\eta$  policy executions. If the SDMA reaches  $s_G$  even once during the  $\eta$  simulations,  $b$  is  $\top$ , representing that the goal can be reached using this policy. Next, we discuss how these responses are used to prune the set of possible models and learn the correct transition model of the SDMA.
4.1 Query-based Autonomous Capability Estimation (QACE) Algorithm
We now discuss how we solve the capability assessment task using the Query-based Autonomous Capability Estimation algorithm (Alg. 1), which works in two phases. In the first phase, QACE learns all capabilities' preconditions and non-deterministic effects using the policy simulation queries (Sec. 4.2). In the second phase, QACE converts the nondeterministic effects of capabilities into probabilistic effects (Sec. 4.3). We now explain the learning portion (lines 3-11) in detail.
QACE first initializes a model  $M^{*}$  over the predicates in  $\mathcal{P}$  with capabilities having names  $c_{N} \in \mathcal{C}_{N}$ . All the preconditions and effects for all capabilities are empty in this initial model. QACE uses  $M^{*}$  to maintain the current partially learned model. QACE iterates over all combinations of  $L$  and  $\mathcal{P}$  (line 4). For each pair, QACE creates 3 can
didate models  $M_T$ ,  $M_F$ , and  $M_I$  as mentioned earlier. It then takes 2 of these (line 5) and generates a query  $q$  (line 6) such that responses to the query  $q$  from the two models are logically inconsistent (see Sec. 4.2). The query  $q$  is then posed to the SDMA  $\mathcal{A}$  whose response is stored as  $\theta_A$  (line 7). QACE finally prunes at least one of the two models by comparing their responses (which are logically inconsistent) with the response  $\theta_A$  of the SDMA on that query (line 8). QACE also updates the effects of all models in the set of possible models to speed up the learning process (line 9). Finally, it learns the probabilities of the observed stochastic effects using maximum likelihood estimation (line 10). An important feature of the algorithm (similar to PLEX [Mehta et al., 2011] and AIA [Verma et al., 2021]) is that it keeps track of all the locations where it hasn't identified the correct way of adding a predicate. The next section presents our approach for generating the queries in line 6.
4.2 Algorithms for Query Synthesis
One of the main challenges in interactive model learning is to generate the queries we discussed above and to learn the agent's model using them. Although active learning [Settles, 2012] addresses the
Algorithm 1: QACE Algorithm
Input: predicates  $\mathcal{P}$ ; capability names  $\mathcal{C}_N$ ; state  $s$ ; SDMA  $\mathcal{A}$ ; hyperparameters  $\alpha, \eta$ ; FOND Planner  $\rho$
Output:  $M$
1  $L\gets \{pre,eff\} \times \mathcal{C}_N$
2  $M^{*}\gets$  initializeModel  $(\mathcal{P},\mathcal{C}_N)$
3 for each  $\langle l,p\rangle \in \langle L,\mathcal{P}\rangle$  do
4 Generate  $M_T, M_F, M_I$  by setting  $p$  at  $l$  in  $M^*$
for each pair  $M_{i},M_{j}$  in  $\{M_T,M_F,M_I\}$  do
6  $q\gets$  generateQuery  $(M_i,M_j,\alpha ,\eta ,s,\rho)$  7  $\theta_{\mathcal{A}},\mathbb{S}\gets$  getResponse(q,A,s)
8  $M^{*}\gets$  pruneModels  $(\theta_{\mathcal{A}},M_i,M_j)$
9  $M^{*}\gets$  learn possible stochastic effects of capability with  $c_{N}$  in  $l$  using  $\zeta$  (in  $\theta_{\mathcal{A}}$ )
10  $M \gets$  learnProbabilitiesOfStochasticEffects( $\zeta$ ,  $M^{*}$ )
11 return  $M$