Synthesizing Explainable Behavior for Human-AI Collaboration Subbarao Kambhampati Arizona State

As AI technologies enter our everyday lives at an ever increasing pace, there is a greater need for AI systems to work synergistically with humans. This requires AI systems to exhibit behavior that is explainable to humans. Synthesizing such behavior requires AI systems to reason not only with their own models of the task at hand, but also about the mental models of the human collaborators. Using several case-studies from our ongoing research, I will discuss how such multi-model planning forms the basis for explainable behavior. ACM Reference Format: Subbarao Kambhampati. 2019. Synthesizing Explainable Behavior for HumanAI Collaboration. In Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13–17, 2019, IFAAMAS, 2 pages. When two humans collaborate to solve a task, both of them will develop approximate models of the goals and capabilities of each other (the so called “theory of mind”), and use them to support fluid team performance. AI agent interacting with humans–be they embodied or virtual–will also need to take this implicit mental modeling into account. In order for the AI agent to show behavior that makes sense to the human, AI agents thus need to go beyond planning with their own models of the world, and take into account the mental model of the human in the loop. The mental model here is not just the goals and capabilities of the humans in the loop, but includes the human’s model of the AI agent’s goals/capabilities. LetMR andMH correspond to the actual goal/capability models of the AI agent and human. To support collaboration, the AI agent needs an approximation of MH , we will call it M̃H r , to take into account the goals and capabilities of the human. The AI agent also needs to recognize that the human will have a model of it’s goals/capabilitiesMR h , and needs an approximation of this, denoted M̃R h . Synthesizing explainable behavior then becomes a challenge of supporting planning in the context of thesemultiple models. (A note on the model representation. In much of our work, we have used relational precondition-effect models. We believe however that our frameworks can be readily adapted to other model representations; e.g. [14].) Proactive help: Left to itself, the AI agent will useMR to synthesize its behavior. When the agent has access to M̃H r , we show how it can use that model to plan behaviors that proactively help the human user–either by helping them complete their goals (c.f. [1]) or avoiding resource contention with them (c.f. [8]). Explicability: When the agent has access to M̃R h , it can use that model to ensure that its behavior is explainable. We start by looking at generation of explicable behavior, which requires the AI agent to Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), N. Agmon, M. E. Taylor, E. Elkind, M. Veloso (eds.), May 13–17, 2019, Montreal, Canada. © 2019 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved. not only consider the constraints of its modelMR , but also ensure that its behavior is in line with what is expected by the human. We can formalize this as finding a plan π that trades off the optimality with respect to MR and “distance” from the plan π ′ that would be expected according to M̃R h . This optmization can be done either in a model-based fashion, where the distances between π and π ′ are explicitly estimated (c.f. [10]) or in a model-free fashion, where the distance is indirectly estimated with the help of a learned “labeling” function that evaluates how far π is from the expected plan/behavior (c.f. [18]). Our notion of explicability here has interesting relations to other notions of interpretable robot behavior considered in AI and robotics communities; we provide a critical comparison of this landscape in [3]. Explanation: In some cases, M̃R h might be so different from M R that it will be too costly or infeasible for the AI agent to conform to those expectations. In such cases, the agent needs to provide an explanation to the human (with the aim of making its behavior more explicable). We view explanation as a process of “model reconciliation,” specifically the process of helping the human bringMR h closer toMR . While a trivial way to accomplish this is to send the whole ofMR as the explanation, in most realistic tasks, this will be both costly for the AI agent to communicate, and more importantly, for the human agent to comprehend. Instead, the explanation should focus on minimal changes E toMR h , such that the robot behavior π is explicable with respect to MR h + E , thus in essence making the behavior interpretable to human in light of the explanation. In [7] we show that computing such explanations can be cast as a meta search in the space of models spanningMR and M̃R h (which is the AI agent’s approximation ofMR h ). We also provide methods to make this search more efficient, and discuss a spectrum of explanations with differing properties that can all be computed in this framework. Balancing Explicability & Explanation: While the foregoing presented explicable behavior and presenting explanation as two different ways of exhibiting explainable behavior, it is possible to balance the trade-offs between them. In particular, given a scenario where π∗ would have been the plan that is optimal with respect to MR , the AI agent can choose to go with a costlier plan π̃ (where π̃ is still not explicable with respect toMR h ), and provide an explanation E ′ such that π̃ is explicable with respect to MR h + E ′. In [4], we show how we can synthesize behaviors that have this trade-off. Model Acquisition: While we focused on the question of reasoning with multiple models to synthesize explainable behavior, a closely related question is that of acquiring the models. In some cases, such as search and rescue scenarios, the human and AI agent may well start with the same shared model of the task. Here the AI agent can assume that as the default mental model. In other cases, the AI agent may have an incomplete model of the human; in [12], we provide an approach to handle the incomplete model, viewing it as a union of complete models. More generally, the AI agent may Keynote Talk AAMAS 2019, May 13-17, 2019, Montréal, Canada