Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

1 INTRODUCTION . Humans exhibit altruistic behaviors at an early age ( Warneken & Tomasello , 2006 ) . Without much prior experience , children can robustly recognize goals of other people by simply watching them act in an environment , and are able to come up with plans to help them , even in novel scenarios . In contrast , the most advanced AI systems to date still struggle with such basic social skills . In order to achieve the level of social intelligence required to effectively help humans , an AI agent should acquire two key abilities : i ) social perception , i.e. , the ability to understand human behavior , and ii ) collaborative planning , i.e. , the ability to reason about the physical environment and plan its actions to coordinate with humans . In this paper , we are interested in developing AI agents with these two abilities . Towards this goal , we introduce a new AI challenge , Watch-And-Help ( WAH ) , which focuses on social perception and human-AI collaboration . In this challenge , an AI agent needs to collaborate with a human-like agent to enable it to achieve the goal faster . In particular , we present a 2-stage framework as shown in Figure 1 . In the first , Watch stage , an AI agent ( Bob ) watches a human-like agent ( Alice ) performing a task once and infers Alice ’ s goal from her actions . In the second , Help stage , Bob helps Alice achieve the same goal in a different environment as quickly as possible ( i.e. , with the minimum number of environment steps ) . This 2-stage framework poses unique challenges for human-AI collaboration . Unlike prior work which provides a common goal a priori or considers a small goal space ( Goodrich & Schultz , 2007 ; Carroll et al. , 2019 ) , our AI agent has to reason about what the human-like agent is trying to achieve by watching a single demonstration . Furthermore , the AI agent has to generalize its acquired knowl- 1Code and documentation for the VirtualHome-Social environment are available at https : // virtual-home.org . Code and data for the WAH challenge are available at https : //github.com/ xavierpuigf/watch_and_help . A supplementary video can be viewed at https : //youtu.be/ lrB4K2i8xPI . edge about the human-like agent ’ s goal to a new environment in the Help stage . Prior work does not investigate such generalization . To enable multi-agent interactions in realistic environments , we extend an open source virtual platform , VirtualHome ( Puig et al. , 2018 ) , and build a multi-agent virtual environment , VirtualHomeSocial . VirtualHome-Social simulates realistic and rich home environments where agents can interact with different objects ( e.g , by opening a container or grabbing an object ) and with other agents ( e.g. , following , helping , avoiding collisions ) to perform complex tasks . VirtualHome-Social also provides i ) built-in agents that emulate human behaviors , allowing training and testing of AI agents alongside virtual humans , and ii ) an interface for human players , allowing evaluation with real humans and collecting/displaying human activities in realistic environments ( a functionality key to machine social intelligence tasks but not offered by existing multi-agent platforms ) . We plan to open source our environment . We design an evaluation protocol and provide a benchmark for the challenge , including a goal inference model for the Watch stage , and multiple planning and deep reinforcement learning ( DRL ) baselines for the Help stage . Experimental results indicate that to achieve success in the proposed challenge , AI agents must acquire strong social perception and generalizable helping strategies . These fundamental aspects of machine social intelligence have been shown to be key to humanAI collaboration in prior work ( Grosz & Kraus , 1996 ; Albrecht & Stone , 2018 ) . In this work , we demonstrate how we can systematically evaluate them in more realistic settings at scale . The main contributions of our work are : i ) a new social intelligence challenge , Watch-And-Help , for evaluating AI agents ’ social perception and their ability to collaborate with other agents , ii ) a multiagent platform allowing AI agents to perform complex household tasks by interacting with objects and with built-in agents or real humans , and iii ) a benchmark consisting of multiple planning and learning based approaches which highlights important aspects of machine social intelligence . 2 RELATED WORK . Human activity understanding . An important part of the challenge is to understand human activities . Prior work on activity recognition has been mostly focused on recognizing short actions ( Sigurdsson et al. , 2018 ; Caba Heilbron et al. , 2015 ; Fouhey et al. , 2018 ) , predicting pedestrian trajectories ( Kitani et al. , 2012 ; Alahi et al. , 2016 ) , recognizing group activities ( Shu et al. , 2015 ; Choi & Savarese , 2013 ; Ibrahim et al. , 2016 ) , and recognizing plans ( Kautz , 1991 ; Ramırez & Geffner , 2009 ) . We are interested in the kinds of activity understanding that require inferring other people ’ s mental states ( e.g. , intentions , desires , beliefs ) from observing their behaviors . Therefore , the Watch stage of our challenge focuses on the understanding of humans ’ goals in a long sequence of actions instead . This is closely related to work on computational Theory of Mind that aims at inferring humans ’ goals by observing their actions ( Baker et al. , 2017 ; Ullman et al. , 2009 ; Rabinowitz et al. , 2018 ; Shum et al. , 2019 ) . However , in prior work , activities were simulated in toy environments ( e.g. , 2D grid worlds ) . In contrast , this work provides a testbed for conducting Theory-of-Mind type of activity understanding in simulated real-world environments . Human-robot interaction . The helping aspect of the WAH challenge has been extensively studied in human-robot interaction ( HRI ) . However , prior work in HRI has been mainly restricted in lab environments ( Goodrich & Schultz , 2007 ; Dautenhahn , 2007 ; Nikolaidis et al. , 2015 ; Rozo et al. , 2016 ) , and the goals in the collaborative tasks were either shared by both agents or were defined in a small space . The setup in WAH is much more challenging – the goal is sampled from a large space , needs to be inferred from a single demonstration , and must be performed in realistic and diverse household environments through a long sequence of actions . Multi-agent virtual environments . There has been a large body of platforms for various multiagent tasks ( Jaderberg et al. , 2019 ; Samvelyan et al. , 2019 ; OpenAI , 2018 ; Lowe et al. , 2017 ; Resnick et al. , 2018 ; Shu & Tian , 2018 ; Carroll et al. , 2019 ; Suarez et al. , 2019 ; Baker et al. , 2019 ; Bard et al. , 2020 ) . However , these multi-agent platforms can only simulate simple or game-like environments and do not support for human-AI collaborations on real-life activities . Existing platforms for realistic virtual environments mainly focus on single agent settings for tasks such as navigation ( Savva et al. , 2019 ; Xia et al. , 2018 ; Brodeur et al. , 2017 ; Zhu et al. , 2017 ; Xia et al. , 2018 ) , embodied question answering ( Gordon et al. , 2017 ; Wijmans et al. , 2019 ; Das et al. , 2018 ) , or single agent task completion ( Puig et al. , 2018 ; Shridhar et al. , 2019 ; Misra et al. , 2018 ; Gao et al. , 2019 ) . In contrast , the proposed VirtualHome-Social environment allows AI agents to engage in multi-agent household activities by i ) simulating realistic and interactive home environments , ii ) incorporating humanoid agents with human-like behaviors into the system , iii ) providing a wide range of commands and animations for navigation and object manipulation , and iv ) allowing human participation . Because of these features , VirtualHome-Social can serve as a testbed for complex social perception and humanAI collaboration tasks , which is complementary to existing virtual environments . 3 THE WATCH-AND-HELP CHALLENGE . The Watch-And-Help challenge aims to study AI agents ’ ability to help humans in household activities . To do that , we design a set of tasks defined by predicates describing the final state of the environment . For each task , we first provide Bob a video that shows Alice successfully performing the activity ( Watch stage ) , and then place both agents in a new environment where Bob has to help Alice achieve the same goal with the minimum number of time steps ( Help stage ) . Figure 2 provides an overview of the system setup for the Watch-And-Help challenge . For this challenge , we build a multi-agent platform , VirtualHome-Social ( Section 4 ) , that i ) supports concurrent actions from multiple agents and ii ) provides observations for the agents . Alice represents a built-in agent in the system ; she plans her actions based on her own goal and a partial observation of the environment . Bob serves as an external AI agent , who does not know Alice ’ s ground-truth goal and only has access to a single demonstration of Alice performing the same task in the past . During the Help stage , Bob receives his observation from the system at each step and sends an action command back to control the avatar in the environment . Alice , on her part , updates her plan at each step based on her latest observation to reflect any world state change caused by Bob . We also allow a human to control Alice in our system . We discuss how the system and the built-in agent work in Section 4 . Problem Setup . Formally , each task in the challenge is defined by Alice ’ s goal g ( i.e. , a set of goal predicates ) , a demonstration of Alice taking actions to achieve that goalD = { stAlice , atAlice } Tt=1 ( i.e. , a sequence of states stAlice and actions a t Alice ) , and a new environment where Bob collaborates with Alice and help achieve the same goal as quickly as possible . During training , the ground-truth goal of Alice is shown to Bob as supervision ; during testing , Bob no longer has access to the ground-truth goal and thus has to infer it from the given demonstration . Goal Definitions . We define the goal of a task as a set of predicates and their counts , which describes the target state . Each goal has 2 - 8 predicates . For instance , “ ON ( plate , dinnertable ) :2 ; ON ( wineglass , dinnertable ) :1 ” means “ putting two plates and one wine glass onto the dinner table. ” The objects in a predicate refer to object classes rather than instances , meaning that any object of a specified class is acceptable . This goal definition reflects different preferences of agents ( when setting up a dinner table , some prefer to put water glasses , others may prefer to put wine glasses ) , increasing the diversity in tasks . We design five predicate sets representing five types of household activities : 1 ) setting up a dinner table , 2 ) putting groceries / leftovers to the fridge , 3 ) preparing a simple meal , 4 ) washing dishes , and 5 ) reading a book while having snacks or drinks . In total , there are 30 different types of predicates . In each task , the predicates of a goal are sampled from one of the five predicate sets ( as a single household activity ) . More details about the predicate sets and goal definitions are listed in Appendix B.1 .

Summary: The paper targets to demonstrate social perception and human-AI collaboration in common household activities. It shows the development of a multi-agent virtual environment that is used to test an AI agent’s ability to reason about other agents’ mental states and help them in unfamiliar scenarios. This is performed by presenting an experimental study over specifically selected scenarios which involve aspects of social intelligence.