Real World Visual Processing Speech Recogniser Parser State Manager Planner / Execution Monitor Output Generation Talking-Head Controller Robot Motion Planner State Model

A robot coexisting with humans must not only be able to perform physical tasks, but must also be able to interact with humans in a socially appropriate manner. In many social settings, this involves the use of social signals like gaze, facial expression, and language. In this paper, we discuss the problem of planning social and task-based actions for a robot that must interact with multiple human agents in a dynamic domain. We show how social states are inferred from low-level sensors, using vision and speech as input modalities, and use a general purpose knowledge-level planner to model task, dialogue, and social actions, as an alternative to current mainstream methods of interaction management. The resulting system has been evaluated in a real-world study with human subjects, in a simple bartending scenario. Introduction and Motivation As robots become integrated into daily life, they must increasingly deal with situations in which socially appropriate interaction is vital. In such settings, it is not enough for a robot simply to achieve task-based goals; instead, it must also be able to satisfy the social goals and obligations that arise through interactions with people in real-world settings. However, the problem of building a robot to meet the goals of social interaction presents several challenges, especially for the reasoning and action selection components of such a system. Not only does the robot require the ability to recognise and understand appropriate multimodal social signals (e.g., gaze, facial expression, and language), but it must also generate realistic responses using similar modalities. To address this challenge, and help focus our work, we are investigating the sub-problem of task-based social interaction using a bartending scenario as our target domain. In particular, we are developing a robot bartender (Figure 1) that is capable of dealing with multiple customers in a drink-ordering scenario. Interactions in this scenario incorporate both task-based aspects (e.g., ordering and paying for drinks) and social aspects (e.g., managing multiple interactions). Moreover, the primary interaction modality in this setting is speech; humans communicate with the robot via speech and the robot must respond in a similar manner. Our approach to high-level reasoning and action selection uses AI planning, specifically, knowledge-level planFigure 1: The robot bartender ning (Petrick and Bacchus 2002; 2004) techniques. The ability to reason and plan is essential for a cognitive agent acting in a dynamic and incompletely known world such as the bartending scenario. General-purpose automated planners are good at building goal-directed plans of action under many challenging conditions, especially in task-based contexts. Moreover, recent work (Steedman and Petrick 2007; Brenner and Kruijff-Korbayová 2008; Benotti 2008; Koller and Petrick 2011) has investigated the use of automated planning for natural language generation and dialogue— research fields that have a long tradition of using planning, but where such techniques are no longer the focus of mainstream study. The use of planning for natural language processing is particularly important since plan generation in the bartending domain will require a mix of traditional task-based actions (e.g., handing the customer a drink) and speech acts (e.g., asking a customer for a drink order). While planning offers a tool for reasoning and action selection, it is only one component in a larger robot system that must operate in a real-world environment. This introduces some difficulties that must be overcome when the planner interacts with other parts of the system. For instance, the problem of integrating low-level sensor data with symbolic planners introduces representational difficulties that must be addressed: high-level planners typically use representations based on discrete models of objects, properties, and actions, described in logical languages, while many lowlevel sensors tend to generate continuous streams of lowlevel, noisy data. Moreover, some aspects of the traditional planning problem, like the initial state, cannot be defined A customer approaches the bar and looks at the bartender ROBOT: [Looks at Customer 1] How can I help you? CUSTOMER 1: A pint of cider, please. Another customer approaches the bar and looks at the bartender ROBOT: [Looks at Customer 2] One moment, please. ROBOT: [Serves Customer 1] ROBOT: [Looks at Customer 2] Thanks for waiting. How can I help you? CUSTOMER 2: I’d like a pint of beer. ROBOT: [Serves Customer 2] Figure 2: An example interaction in the bartending scenario. offline (e.g., the number of customers in the bar). Instead, they must be provided to the planner based on observations of the scene sensed by low-level input modalities such as vision and speech. Furthermore, in an inherently dynamic domain like a busy bar, these sensors may not be able to fully observe the world, or may provide noisy information. Thus, the planner cannot be viewed as simply a black box but must be appropriately situated in the wider cognitive system. We focus on three main areas of work in this paper: • First, we describe the structure of the robot system and highlight the role of the planner within the architecture. In particular, we focus on how states are derived from sensor observations and how generated plans are executed. • Second, we show how plans are built by viewing taskbased social interaction as an instance of planning with incomplete information and sensing, using the PKS planner (Petrick and Bacchus 2002; 2004) as an alternative to more mainstream methods of interaction management. • Finally, we present a planning domain for a simple bartending scenario, and show how it generates interesting interaction behaviours. This domain has been evaluated in a real-world study, and provides the basis for future work. This work forms part of a larger project called JAMES, Joint Action for Multimodal Embodied Social Systems, exploring social interaction with robot systems.1 Overview of the Scenario and System The target application for this work is a bartending scenario supporting interactions similar to the one in Figure 2. In this scenario, two agents enter the bar area and attempt to attract the robot’s attention to order a drink. Even this simple interaction presents certain challenges to the robot system tasked with the role of bartender: the vision system must track the locations and body postures of the agents; the speechrecognition system must detect and deal with speech in an open setting; the reasoning components must determine that both customers require attention and should ensure that they are served in the order that they arrived; while the output components must select and execute concrete actions for each output channel that correctly realises high-level plans. To address these challenges, the system architecture builds on a standard three-layer structure: the low level deals 1See http://james-project.eu/ for more information. Real World Visual Processing Speech Recogniser Parser State Manager Planner / Execution Monitor