Explainable , Normative , and Justified Agency
(2018)

In this paper, we pose a new challenge for AI researchers – to develop intelligent systems that support justified agency. We illustrate this ability with examples and relate it to two more basic topics that are receiving increased attention – agents that explain their decisions and ones that follow societal norms. In each case, we describe the target abilities, consider design alternatives, note some open questions, and review prior research. After this, we return to justified agency, offering a hypothesis about its relation to explanatory and normative behavior. We conclude by proposing testbeds and experiments to evaluate this empirical claim and encouraging other researchers to contribute to this crucial area. 1 Background and Motivation Autonomous artifacts, from self-driving cars to drones to household robots, are becoming more widely adopted, and this trend seems likely to accelerate in the future. The increasing reliance on these devices has raised concerns about our ability to understand their behavior and our capacity to ensure their safety. Before intelligent agents can gain widespred acceptance in society, they must be able to communicate their decision making to humans in ways that convince us they actually share our aims. This challenge involves two distinct but complementary issues. The first is the need for agents to explain the reasons they carried out particular courses of action in terms that we can understand. Langley et al. (2017) have referred to this as explainable agency. The second is the need for assurance that, when agents pursue explicit goals, they will also follow the many implicit rules of society. We will refer to this ability as normative agency. Both of these functions are necessary underpinnings of trustable autonomous systems, and two capabilities are closely intertwined. Consider a scenario in which a person drives a friend with a ruptured appendix to the hospital. The driver exceeds the speed limit, weaves in and out of traffic, runs through red lights, and even drives on a sidewalk, although he is still careful to avoid hitting other cars or losing control on turns. Afterwards, the driver explains that he took such drastic actions because he thought the passenger’s life was in danger, Copyright c © 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. so reaching the hospital in short order had higher priority than being polite to others along the way or obeying traffic laws. Humans can defend their behavior in this manner even when they violate serious societal norms, and we want intelligent agents of the future to exhibit the same ability to justify their decisions and actions on demand. We discuss this challenge in the sections that follow, arguing that such justified agency is an important topic that deserves substantial attention from AI researchers. We first examine the two related topics of explainable agency and normative agency. In each case, we describe the desired abilities, offer illustrative examples, and touch on relevant research. We also consider the space of agent designs, including some open issues that require additional work. After this, we turn to justified agency, arguing that it combines the first two abilities and proposing a hypothesis about what else is required to support it. We close by considering some testbeds and experiments that would let the research community evaluate progress in this critical arena. 2 Explainable Agency People can usually explain their reasons for making decisions and taking actions. When someone purchases a microwave oven, rearranges furniture in a room, or plans a vacation, he can state the choices considered, why he selected one alternative over the others, and even how his decision might have differed in other circumstances. Retrospective reports are seldom perfect, but they often offer important windows into the decision process. As intelligent agents become both more autonomous and more prevalent, it is essential that they support similar abilities. We will say that: • An intelligent system exhibits explainable agency if it can provide, on request, the reasons for its activities. Consider some examples from the realm of autonomous vehicles. If we ask a self-driving car why it followed a given route, it should be able to state that the path had few lights and stop signs while still being reasonably short. More importantly, when we ask the vehicle why it swerved into another car, it should explain that it was the only way to avoid hitting an unexpected jaywalker. Like humans, such agents should have reasons for their actions and they should be able to communicate them to others when asked. Tackling this problem requires that we make design decisions about representations and processes needed to support explainable agency, some of which we will borrow from Langley et al.’s (2017) analysis. One issue concerns what will count as legitimate explanations. Should plausible post hoc rationalizations be acceptable if an agent’s decisionmaking procedures are not interpretable or should we require genuine insights into why the agent took its actions? We argue that only the latter should be viewed as reasonable accounts, which implies that the agents should make decisions in ways that are transparent and easily communicated. There are many well-established methods in the AI arsenal that meet this criterion, but opaque policies induced from large training sets will not suffice. Another design choice involves whether to explain activities in terms of individual actions or higher-level structures. Research in machine learning has emphasized reactive control (e.g., Erwig et al., 2018), which supports the first alternative, whereas AI planning systems make choices about entire sequences of actions. We predict that humans will find plan-oriented explanations more familiar and easier to understand, and thus offer a natural approach to adopt. Even when a plan goes awry during execution, an agent can still give the reasons it decided to change course and how its new plan responded to the surprise. Some frameworks, like hierarchical task networks, also specify plans at multiple levels of abstraction, which would let a system offer more or less detailed explanations, down to individual actions if desired. Any intelligent agent must use information to make decisions about which activities to pursue, and a third issue concerns how to encode this content. One response, widely adopted in AI planning research, relies on symbolic goals, often stated as logical expressions. Another option, popular in game-playing systems, instead uses numeric evaluation functions that comprise sets of weighted features. Each approach has advantages for explainable agency: goals provide explicit end points for chains of actions, while functions show how such plans handle trade offs. Although these typically appear in isolation, they can also be combined. For instance, Langley et al. (2016) describe an agent architecture that associates functions with goals, using their weighted sum to guide planning. Such hybrid frameworks offer one promising approach to building explainable agents. Within this design space, we still need research on a number of open issues about explainable agency: These include extending intelligent systems to: • Generate explanatory content. When deciding on courses of action, an agent must consider different alternatives, evaluate them, and select one of the options to pursue. This should take place during generation of plans and during their execution, producing traces that can be used in later explanations of the agent’s activities. • Store generated content. As it makes these decisions, an agent must cache information about the choices that it considered and the reasons that it favored one over the others, in an episodic memory or similar repository. This requires not just retaining the content about decisions, but also indexing it in ways that support later access. • Retrieve stored content. After it has completed an activity, an agent must be able to retrieve decision traces that are relevant to different types of questions. This requires transforming the queries into cues and using them to access content in episodic memory about alternatives considered, their evaluations, and the final choices made. • Communicate retrieved content. Once it has retrieved episodic traces in response to a question, an agent must identify those aspects most relevant to the query, translate them into an understandable form, and share the answer. This should include no more detail than needed to convey the reasons for making the decision under examination. Research on analogical planning (e.g., Jones and Langley, 2005; Veloso et al., 1995) has addressed issues of storage, indexing, and retrieval, but not for the purpose of self report. Leake (1992) presented a computational theory of explanation, but it focused on accounts of other agents’ behaviors. The AI literature also includes other research relevant to this topic. Early explanation systems recorded inference chains and recounted them when asked to justify their conclusions (Swartout, Paris, and Moore, 1991), with some systems supporting higher-level accounts with meta-level rules (Clancey, 1983), but these did not deal with physical activities. More relevant work comes from Johnson (1994) and van Lent et al. (2004), who developed agents for military mssions that recorded their decisions, offered reasons on request, and anwered counterfactual queries. However, they dealt with knowledge-guided execution rather than agentgenerated plans and, despite linking actions to objectives, did not state why some activities were preferable to others. In more recent work, Briggs and Scheutz (2015) have reported an interactive robot that gives reasons why it cannot carry out some task, drawing on five explanation types, including lack of knowledge and physical ability. Th