MA-RADAR – A Mixed-Reality Interface for Collaborative Decision Making

There has been a lot of recent interest in the planning community towards adapting automated planning techniques for the role of decision support for human decision makers in the loop. A unique challenge in such settings is the presence of multiple humans collaborating during the planning process which not only requires algorithmic advances to handle issues such as diverging mental models and the establishment of common ground, but also the development of user interfaces that can facilitate the distributed decision making process among the human planners. We posit that recent advances in augmented reality technology is uniquely positioned to serve this need. For example, a mixed-reality workspace can be ideal for curating information towards the particular needs (e.g. explanations) of the individual decision makers. In this paper, we report on ongoing work along these directions and showcase MA-RADAR, the multi-agent version of the decision support system RADAR (Sengupta et al. 2017). In (Sengupta et al. 2017), we explored the evolving roles of an automated planner in the scope of decision support for a single human in the loop. Specifically, we outlined how well-established principles of naturalistic decision-making and the automation hierarchy studied in existing literature on human-computer interaction (Parasuraman, Sheridan, and Wickens 2000; Klein 2008) can be adopted for the design of automated decision support using planners as well. In this regard, we demonstrated how the traditional role of an automated planner changes from one of plan generation to more nuanced roles of plan validation, recognition, recommendation, critique, explanations, and so on. However, most of these techniques, as well as the GUI itself, were specifically designed to deal with a single human decision maker in the loop. As we illustrate in the paper, these become ineffective in a distributed setting. What if there are multiple humans in the loop? A common feature of most collaborative planning settings is the presence of multiple human decision makers who are actively involved in the construction of the plan on a shared graphical user interface (GUI) in “control room” styled environments (Chakraborti et al. 2017b; Karafantis 2013; Murphy 2015). For the design of decision support technologies, this raises several unique challenges such as (1) dealing ∗Authors marked with ∗ contributed equally. with diverse points of view, preferences, and goals; (2) diverging beliefs and mental models; (3) resolution of competing truths and establishment of common ground; and so on. Some of these issues have been highlighted recently in (Kim and Shah 2017). From the perspective of the GUI itself, the presence of multiple decision makers poses new challenges on how information is presented to the end users, not only in the way it is displayed, but also the approach to generate that information which drives the decision support infrastructure in the back-end. Recent advances in augmented-reality technologies in opening up newer channels of communications with AI agents (Williams et al. 2018) can begin to address some of these challenges. What can augmented reality bring to the table? We argue that augmented reality (AR) brings in capabilities that are uniquely suited for this purpose. This is because AR can, in effect, provide different versions of the same interface to the commanders based on their specific needs, while still preserving the convenience and efficiency of collaboration across a shared GUI. In this work, we thus will build on our previous decision support system – c.f. RADAR (Sengupta et al. 2017) – and highlight challenges, especially as it relates to the design of the interface for the decision support system, when the collaborative decision making setting is extended to deal with multiple human planners simultaneously. We will, in particular, show how – Augmented reality provides an effective medium of augmenting the shared GUI with private information (as studied in planning literature (Brafman and Domshlak 2008)) – thus the same plan will appear differently on the shared GUI than in the mixed-reality view where the private actions will be coupled with the public plan; and Augmented reality can reduce irrelevant information on the screen by porting them into the mixed-reality view. Such situations can occur, for example, when one user asks for an explanation, which the others may not require and thus should not appear on the shared GUI and potentially cause cognitive overload. Finally, we will end with a discussion on the current work in progress and considerations of the trade-offs in AR versus distributed graphical interfaces. We note, as far as the vitamin versus aspirin question is concerned, AR firmly Figure 1: Multiple commanders involved in the collaborative decision making process on the MA-RADAR interface. The shared interface (GUI) provides an overview of the public plan and resources, constraints, etc. pertaining to the planning problem. The mixed-reality view for each commander augments private information (such as private action in the plan) or personalized explanations of the plan for the commander. Refer to Figures 2 and 3 for the augmented view for each of these use cases. lies with the former group – after all, the humans could be equipped with separate personal screens on top of a shared GUI. However, we argue, and hopefully this is apparent in the demonstrations as well, that AR provides an attractive solution towards providing personalized planning interfaces to the human decision makers while still leveraging the paradigm of a shared collaborative interface of a control room accepted as the de-facto standard in these settings.