Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Authors: Xavier Puig
Tianmin Shu
Shuang Li
Zilin Wang
Yuan-Hong Liao
Joshua B. Tenenbaum
Sanja Fidler
Antonio Torralba
Venue: ICLR 2021 Spotlight (2021)
Keywords: social perception
human-AI collaboration
theory of mind
multi-agent platform
virtual environment

Abstract: In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration). For this challenge, we build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines. We evaluate the performance of AI agents with the human-like agent as well as and with real humans using objective metrics and subjective user ratings. Experimental results demonstrate that our challenge and virtual environment enable a systematic evaluation on the important aspects of machine social intelligence at scale.

Published as a conference paper at ICLR 2021
WATCH-AND-HELL A CHALLENGE FOR SOCIAL PER- CEPTION AND HUMAN-A AI COLLABORATION
Xavier Puig' Tianmin Shu'
Shuang Li
Zilin Wang² Antonio Torralba*
Yuan-Hong Liao
Joshua B. Tenenbaum'
Sanja Fidler3.4.5 3,4,5
Massachusetts Institute of Technology 3 University of Toronto "NVIDIA
EETH Zurich Vector Institute
ABSTRACT
In this paper, we introduce Watch-And-Help (WAH), challenge for testing so- cial intelligence in agents. In WAH, an Al agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the Al agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordi- nate with the human-like like agent to solve the task in unseen environment as fast as possible (human-A) AI collaboration). For this challenge, we build VirtualHome- Social, multi-agent household environment, and provide a benchmark including both planning and learning based baselines. We evaluate the performance of Al agents with the human-like agent as well as with real humans using objective metrics and subjective user ratings. Experimental results demonstrate that the proposed challenge and virtual environment enable a systematic evaluation the important aspects of machine social intelligence at scale.'
INTRODUCTION
Humans exhibit altruistic behaviors an early age (Warneken & Tomasello, 2006). Without much prior experience, children can robustly recognize goals of other people by simply watching them act in environment, and are able come up with plans to help them, even in novel scenarios. In contrast, the most advanced AI systems to date still struggle with such basic social skills.
In order to achieve the level of social intelligence required to effectively help humans, Al agent should acquire two key abilities: i) social perception, i.e., the ability to understand human behavior, and ii) collaborative planning, i.e., the ability to reason about the physical environment and plan its actions to coordinate with humans. In this paper, we are interested in developing AI agents with these two abilities.
Towards this goal, we introduce a new AI challenge, Watch-And-Help (WAH), which focuses on social perception and human-AI collaboration. In this challenge, AI agent needs to collaborate with a human-like agent to enable it to achieve the goal faster. In particular, we present 2-stage framework as shown in Figure 1. In the first, Watch stage, Al agent (Bob) watches human like agent (A (Alice) performing task once and infers Alice's 's goal from her actions. In the second, Help stage, Bob helps Alice achieve the same goal different environment as quickly as possible (i.e., with the minimum number of environment steps).
This 2-stage framework poses unique challenges for human-Al collaboration. Unlike prior work which provides common goal priori or considers a small goal space (Goodrich & Schultz, 2007; Carroll et al., 2019), our Al agent has to reason about what the human-like agent is trying to achieve by watching single demonstration. Furthermore, the Al agent has to generalize its acquired knowl-
'Code and documentation for the VitualHome-Socins environment are available at https:// virtual-bome. org. Code and data for the WAH challenge are available at https://githbt con/ xavi erpui htw./////aaaaaaaoeo help. A supplementary videa can be viewed at https: / //youtuubbtt bbe TB4421 1 8xPI.

---

Published as a conference paper at ICLR 2021
Alice's is task: set up a dinner table
Bob's task: guess Alice's goal and help her
WATCH stage: Bob watches Alice's behaviors and infers her goal
may want uo dinner table
HELP stage: Bob works with Alice to achieve her goal
Overview of the Watch-And-Hae Help challenge. The challenge has two stages: in the Watch stage, Bob will watch a single demonstration of Alice performing a task and infer her goal; ii) then in the Help stage, based on the inferred goal, Bob will work with Alice help finish the same task as fast as possible in a different environment.
Figure 1:
edge about the human-like agent's goal to new environment in the Help stage. Prior work does not investigate such generalization.
To enable multi-agent interactions in realistic environments, we extend an open source virtual plat- form, VirtualHome (Puig al., 2018), and build multi-agent virtual environment, VirtualHome- Social. VirtualHomeSSocha simulates realistic and rich home environments where agents can inter- act with different objects (e.g, by opening container or grabbing an object) and with other agents (e.g., following, helping, avoiding collisions) to perform complex tasks. VirtualHome-Sociaa also provides i) built-in agents that emulate human behaviors, allowing training and testing of Al agents alongside virtual humans, and ii) interface for human players, allowing evaluation with real hu- mans and collecting/displaying human activities in realistic environments (a functionality key to machine social intelligence tasks but not offered by existing multi-agent platforms). We plan to open source our environment.
We design evaluation protocol and provide benchmark for the challenge, including goal in- ference model for the Watch stage, and multiple planning and deep reinforcement learning (DRL) baselines for the Help stage. Experimental results indicate that achieve success in the proposed challenge, Al agents must acquire strong social perception and generalizable helping strategies. These fundamental aspects of machine social intelligence have been shown be key to human- AI collaboration in prior work (Grosz & Kraus, 1996; Albrecht & Stone, 2018). In this work, we demonstrate how we can systematically evaluate them in more realistic settings scale.
The main contributions of our work are: i) a new social intelligence challenge, Watch-And-Help. for evaluating Al agents' social perception and their ability to collaborate with other agents, ii) a multi- agent platform allowing Al agents to perform complex household tasks by interacting with objects and with built-in agents or real humans, and iii) a benchmark consisting multiple planning and learning based approaches which highlights important aspects of machine social intelligence.
RELATED WORK
Human activity understanding. An important part of the challenge is to understand human activi- ties. Prior work on activity recognition has been mostly focused on recognizing short actions (Sig- urdsson et al., 2018; Caba Heilbron et al., 2015; Fouhey et al., 2018), predicting pedestrian trajec- tories (Kitani et al., 2012; Alahi et al., 2016), recognizing group activities (Shu et al., 2015; Choi & Savarese, 2013; Ibrahim al., 2016), and recognizing plans (Kautz, 1991; Ramirez & Geffner, 2009). We are interested the kinds of activity understanding that require inferring other people's mental states (e.g., intentions, desires, beliefs) from observing their behaviors. Therefore, the Watch stage of our challenge focuses on the understanding of humans' goals in a long sequence of actions instead. This is closely related to work computational Theory of Mind that aims at inferring
2

---

Published as a conference paper at ICLR 2021
WATCH stage
HELP stage
VirtualHome-Soona
Ground-truth Goal
oxiplate, table): OX glass, table) ON fork, table):
Built-in Aliie
Alice's action
or Human
Alice's observation
Task Demonstration
Bob's Model
Bob's action
27
Environment (Unseon demanstration)
Bob's observation
Figure 2: The system setup for the WAH challenge. An Al agent (Bob) watches a demonstration of a lumen-like agent (Alice) performing task, and infers the goal set of predicates) that Alice was trying to achieve. Afterwards, the Al agent is asked to work together with Alice to achieve the same goal in a new environment as fast as possible. To do that, Bob needs to plan its actions based on i) its understanding of Alice's goal, and ii) a partial observation of the environment. It also needs to adapt to Alice's s plan. We simulate environment dynamics and provide observations for both agents in our HirtualHom-SSochte which is able to plan its actions based on the ground-truth goal, and can react to any world state change caused by Bob through re-planning every step based on its latest observation. Our system also offers an interface for real humans to control Alice and work with an Al agent in the challenge.
multi-agent platform. The platform includes built-in agent as Alice
humans" goals by observing their actions (Baker et al., 2017; Ullman et al., 2009; Rabinowitz et al., 2018; Shum et al., 2019). However, in prior work, activities were simulated in toy environments (e.g., 2D grid worlds). In contrast, this work provides testbed for conducting Mncory-of-Mind type of activity understanding in simulated real-world environments.
Human-robot interaction. The helping aspect of the WAH challenge has been extensively studied in human-robot interaction (HRI). However, prior work in HRI has been mainly restricted in lab environments (Goodrich & Schultz, 2007; Dautenhahn, 2007; Nikolaidis et al., 2015; Rozo al,, 2016), and the goals in the collaborative tasks were either shared by both agents or were defined in a small space. The setup in WAH is much more challenging the goal sampled from large space, needs to be inferred from a single demonstration, and must be performed in realistic and diverse household environments through a long sequence of actions.
Multi-agent virtual environments. There has been large body of platforms for various multi- agent tasks (Jaderberg al., 2019; Samvelyan et al., 2019; OpenAI, 2018: Lowe et al., 2017; Resnick et al., 2018; Shu & Tian, 2018; Carroll et al., 2019; Suarez et al., 2019; Baker et al., 2019; Bard et al., 2020). However, these multi-agent platforms can only simulate simple or game-like environments and do not support for human-Al collaborations
real-life activities. Existing platforms for real- istic virtual environments mainly focus on single agent settings for tasks such as navigation (Savva et al., 2019; Xia et al., 2018; Brodeur et al., 2017; Zhu et al., 2017; Xia et al., 2018). embodied question answering (Gordon et al., 2017; Wijmans et al., 2019; Das et al., 2018), or single agent task completion (Puig al., 2018; Shridhar et al., 2019; Misra et al., 2018; Gao et al., 2019). In contrast, the proposed VirtualHome- Social environment allows Al agents to engage in multi-agent household activities by i) simulating realistic and interactive home environments, ii) incorporating humanoid agents with human-like hehaviors into the system, iii) providing a wide range of commands and an- imations for navigation and object manipulation, and iv) allowing human participation. Because of these features, VirtualHomeSSciall can serve as a testbed for complex social perception and human- AI collaboration tasks, which is complementary to existing virtual environments.
THE WATCH-AND-HOR CHALLENGE
The Watch-And-Help challenge aims to study AI agents' ability to help humans in household ac- tivities. To do that, we design a set of tasks defined by predicates describing the final state of the environment. For each task, we first provide Bob video that shows Alice successfully performing the activity (Watch stage), and then place both agents in a new environment where Bob has to help Alice achieve the same goal with the minimum number of time steps (Help stage).
3

---

Published as a conference paper at ICLR 2021
Figure provides overview of the system setup for the Watch-And-Hon challenge. For this chal- lenge, we build a multi-agent platform, VirtualHome-Sonn -Sciall (Section 4), that i) supports concurrent actions from multiple agents and ii) provides observations for the agents. Alice represents built-in agent in the system; she plans her actions based on her own goal and a partial observation of the environment. Bob serves only has access to a single demonstration of Alice performing the same task in the past. During the Help stage, Bob receives his observation from the system each step and sends an action command back to control the avatar in the environment. Alice, on her part, updates her plan at each step based on her latest observation to reflect any world state change caused by Bob. We also allow a human to control Alice in our system. We discuss how the system and the built-in agent work in Section 4.
external Al agent, who does not know Alice's s ground-truth goal and
Formally, each task in the challenge is defined by Alice's goal g (i.e., a set of goal predicates), a demonstration of Alice taking actions to achieve that goal D = { Alice Alice ) (i.e., a sequence of states Alice and actions " 'Aice) and a new environment where Bob collaborates with Alice and help achieve the same goal as quickly possible. During training, the ground-trute goal of Alice is shown to Bob as supervision; during testing, Bob no longer has access to the ground-truth goal and thus has to infer it from the given demonstration.
Problem Setup.
a
Goal Definitions. We define the goal of task as a set of predicates and their counts, which describes the target state. Each goal has 2 predicates. For instance, "ON (plate, dinnert able) :2; ON (wi negl 1 ass, dinnertable) : 11" means "putting two plates and one wine glass onto the dinner table." The objects in predicate refer to object classes rather than instances, meaning that any object of a specified class is acceptable. This goal definition reflects different preferences of agents (when setting up dinner table, some prefer to put water glasses, others may prefer to put wine glasses), increasing the diversity in tasks. We design five predicate sets representing five types of household activities: 1) setting up dinner table, 2) putting groceries / leftovers to the fridge. 3) preparing simple meal, 4) washing dishes, and 5) reading book while having snacks or drinks. each task, the predicates of a goal are sampled
In total, there are 30 different types of predicates. from one of the five predicate sets (as single household activity). More details about the predicate sets and goal definitions are listed in Appendix B.I.
4 VIRTUALHOME-SOCIAL
Building machine social intelligence for real-life activities poses additional challenges compared to typical multi-agent settings, such as far more unconstrained goal and action spaces, and the need to display human actions realistically for social perception.
With that in mind, we create VirtualHome-Socis ing real humans) can execute actions concurrently and observe each other's 's behaviors. Furthermore, we embed planning- based agents in the environment as virtual humans that AI agents can reason about and interact with.
new environment where multiple agents (includ-
In the rest of this section, we describe the observations, actions, and the built-in human-like agent provided in VirtualHome-Son Sociia... Appendix A includes more information.
Observation space. The environment supports symbolic and visual observations, allowing agents to learn helping behaviors under different conditions. The symbolic observations consist on a scene graph, with nodes representing objects and edges describing spatial relationships between them.
Action space. Agents can navigate in the environment and interact with objects in it. To interact with objects, agents need to specify an action and the index of the intended object (e.g., "grab (3)" stands for grabbing the object with id 3). An agent can only interact with objects that are within its field of sight, and therefore its action space changes at every step.
Human-like agents. To enable a training and testing environment for human-AI interactions, it is critical to incorporate built-in agents that emulate humans when engaging in multi-agent activities. Carroll et al. (2019) has attempted to train policies imitating human demonstrations. But those policies would not reliably perform complex tasks in partially observable environments. Therefore, baased based agent with bounded rationality. provided part of the platform. This agent operates on the symbolic representation of its partial observation of the environment. As shown in Figure 3, it relies two key components: 1) a belief of object locations in the environment
devise a planning-
4

---

Published as a conference paper at ICLR 2021
Goal
Observation
Helping PolccyPlamne
Belief
Goal Inference
Predicate as subgeal e4. IN cupeake, (ridge)
High-level
Low-level PolicyPlanner
t
PolcyPPanner
Bob
Planner Bub
Action
Environment
The overall design of the baseline models. goal inference model infers the goal from a demonstration D and feeds it to helping pol- icy (for learning- based baselines) or to a planner generate Bob's 's action. We adopt a hierarchical approach for all baselines.
Figure 4:
Figure 3: Overview of the human-like agent.
(Figure 13 in Appendix A.3), and 2) a hierarchical planner, which uses Monte Carlo Tree Search (MCTS) (Browne al., 2012) and regression planning (RP) (Korf, 1987) to find plan for given goal based on its belief. At every step, the human-like agent updates its belief based on the latest observation, finds
plan, and executes the first action of the plan concurrently with other agents. The proposed design allows agents to robustly perform tasks in partially observable environments while producing human-like behaviors". We provide more details of this agent in Appendix A.3.
BENCHMARK
5.1 EVALUATION PROTOCOL
Training and Testing Setup. We create a training set with 1011 tasks and 2 testing sets (test-1, test-2). Each test set has 100 tasks. We make sure that i) the helping environment in each task is different from the environment in the pairing demonstration (we sample different apartment and randomize the initial state), and ii) goals (predicate combinations) in the test set are unseen during training. To evaluate generalization, we also hold out 2 apartments for the Help stage in the test sets. For the training set and test-l set, all predicates in each goal are from the same predicate set, whereas goal in test-2 consists of predicates sampled from two different predicates sets representing multi- activity scenarios (c.g., putting groceries to the fridge and washing dishes). Note that during testing, the ground-truie truth goals are not shown to the evaluated Bob agent. More details be found in Appendix B. An episode is terminated once all predicates in Alice's goal are satisfied (i.e., a success) or the time limit (250 steps) is reached (i.e., failure).
Evaluation Metrics. We evaluate the performance of an Al agent by three types of metrics: i) success rate, ii) speedup, and iii) a cumulative reward. For speedup, we compare the episode length when Alice and Bob are working together (LHep) with the episode length when Alice is working alone (Lause), i.e., / LBbb 1. To account for both the success rate and the speedup, we define the cumulative reward of an episode with T steps as R = =E 1(s' = sy) - 0.004, where is the state step t, 8g is the goal state. R. ranges from -1 (failure) to 1 (achieving the goal in zero steps).
5.2 BASELINES
To address this challenge, propose set of baselines that consist of two components shown in Figure 4: a goal inference model and a goal-conditioned helping planner / policy. In this paper, we assume that the AI agent has access the ground-truth states objects within its field of view (but one could also use raw pixels as input). We describe our approach for the two components below.
Goal inference. We train goal inference model based on the symbolic representation of states in the demonstration. At each step, we first encode the state using Transformer ( (Vaswani et al., 2017) over visible objects and feed the encoded state into a long short-term memory (LSTTM) (Hochreiter & Schmidhuber, 1997). We use average pooling to aggregate the latent states from the L.TM over time and build a classifier for each predicate to infer its count. Effectively, we build 30 classifiers, corresponding to the 30 predicates in our taxonomy and the fact that each can appear multiple times.
We conducted user study rating how realistic were the trajectories of the agents and those created by hu- mans, and found significant difference between the groups. More details can be found Appendix D.4.
5

---

Published as a conference paper at ICLR 2021
Helping policy/planner. Due to the nature of the tasks in our challenge e.g.. partial observability, a large action space, sparse rewards, strict preconditions for actions - it is difficult to search for a helping plan learn helping policy directly over the agent's 's actions. To mitigate these difficulties, we propose a hierarchical architecture with two modules for both planning and RI.-based approaches as shown in Figure 4. At every step. given the goal inferred from the demonstration, g. and the current observation of Bob, a high-level policy or planner will output a predicate as the best subgoal to pursue for the current step; the subgoal subsequently fed to a low-level policy or planner which will yield Bob's 's action aea at this step. In our baselines, we use either learned policy or planner for each module. We use the symbolic representation of visible objects as Bob's observation Boa for all models. We summarize the overall design of the baseline models as follows (please refer to Appendix C for the details of models and training procedures):
HP: A hierarchical planner, where the high-level planner and the low-level planner are implemented by MCTS and regression planning (RP) respectively. This the same planner as the one for Alice, except that i) it has its own partial observation and thus different belief from Alice, and ii) when given the ground-truth goal, the high-level planner uses Alice's s plan to avoid overlapping with her.
Hybrid: A hybrid model of RL and planning, where an RL policy serves as the high-level policy and an RP is deployed to generated plans for each subgoal sampled from the RL-based high-level policy. This is to train agent equipped with basic skills for achieving subgoals to help Alice through RL.
HRL: A hierarchical RL baseline where high-level and low-level policies are all learned.
Random: A naive agent that takes random action at each step.
To show the upper bound performance in the challenge, we also provide two oracles:
Oracle An HP-based Bob agent with full knowledge of the environment and the true goal of Alice.
Oracle B, Alice has full knowledge of the environment too.
5.3 RESULTS
We evaluate the Watch stage by measuring the recognition performance of the predicates. The pro- posed model achieves a precision and recall of 0.85 and 0.96 over the test-l set. To evaluate the importance of seeing the full demonstration, we test a model that takes as input the graph represen- tation of the last observation, leading to a precision and recall of 0.79 and 0.75. When using actions taken by Alice as the input, the performance increases to precision and recall of 0.99 and 0.99. The chance precision and recall is 0.08 and 0.09.
We report the performance of our proposed baselines (average and standard error across all episodes) in the Help stage in Figure In addition to the full challenge setup, we also report the performance of the helping agents using true goals (indicated by the subscript TG) and using random goals (by RG), and the performance of Alice working alone. Results show that planning-base most effective in helping Alice. Specifically, HPTG achieves the best performance among non-oracle baselines by using the true goals and reasoning about Alice's ' future plan, avoiding redundant actions and collisions with her (Figure 6 illustrates example of collaboration). Using the inferred goals, both HP and Hybrid can offer effective help. However, with a random goal inference RI)RR). a capable Bob agent becomes counter productive- frequently undoing what Alice has achieved due to their conflicting goals (conflicts appear in 40% of the overall episodes, 65% for Put Groceries and Set Meal). This calls for an AI agent with the ability to adjust its goal inference dynamically by observing Alice's behavior in the new environment (e.g., Alice correcting a mistake made by Bob signals incorrect goal inference). HRL works no better than Random, even though shares the same global policy with Hybrid. While the high level policy selects reasonable predicates to perform the task, the low level policy does not manage to achieve the desired goal. In most of the cases, this is due to the agent picking the right object, but failing to put it to the target location afterwards. This suggests that it is crucial for Bob to develop robust abilitics to achieve the subgoals. There is no significant difference between Random and Alice baselines (t(99) = -1.38, p 0.17).
approaches are the
We also evaluate the baselines in the test-2 set, containing tasks with multiple activities. The goal inference model achieves a precision and recall of 0.68 and 0.64. The performance gap from test-1 indicates that the model fails to generalize to generalize to activit scenarios, overfitting to predicate combinations seen during training. For the Help stage, we evaluate the performance of

---

Published as a conference paper at ICLR 2021
Oraclehe b
Mis
HPo
Hybndiu
2.0
Random Action
HP
Hylaii
ARL
S1.6
Orack
Reward
2066
4.44
Aybndic
Alice
0.0
NRL
Random Action
-02
0.75
0.80
Success Rate
0.85
0.00
0.55
1.00
Set table
Groceries
Set Meal
Wast Dishes
Read Book
Overali
Figure 5: a) Success rate (x axis) and speedup (y axis) of all baselines and oracles. The performance of an effective Bob agent should fall into the upper-right side of the Alice-alone baseline in this plot. b) Cumulative reward the overall test set and in each household activity category (corresponding to the five predicate sets introduced in Section 3).
Alice
helps
Alice's plan:
Alice's plan:
capcupa,,,, table)
muppakk,, cupcake, table)
08] cupenke, table)
OR pudding table)
Bob's plan:
OR ORI poundeake, table)
paoding,, table) oe poundeake table)
Figure 6: Example helping plan. The arrows indicate moving directions and the circles with black borders indicate moments when agents interacted with objects. When working alone (left), Alice had to search different rooms; but with Bob's s help (right), Alice could finish the task much faster.
Bob and Alice both to grab the fork
b Bob avoids conflict
C Bob's actions change Alice's bellef
Bich
gradita
calerel revoaling Brig whee bodto.
Abos to catooea and fndd winoo
lowards the fork.
Pork
Bob boki sbowlhere
in losking wine.
d Bob blocks Alice
e Bob's actions cause false bellef for Alice
Bob commetts lc gi abete
Whe NEW CONTAN DUNCK opplo APSITION..
Robitbocko Thn room
Buo bbokk Bho Irogo
The cacinet.
And puls EOn the MERR
Figure 7:
Example helping behaviors. We show more examples in the supplementary video.
Alice alone, well as the best performing baseline, HP. Alice achieves a success rate of 95.40 + 0.01, while the HP baseline achieves a success rate of 88.60 + 0.02 and speedup of 0.21 + 0.04. Compared to its performance in the test-1 set, the HP baseline suffers significant performance degradation in the test-2 set, which is a result of the lower goal recognition accuracy in the Watch stage.
To better understand the important factors for the effectiveness helping, we analyze the helping behaviors exhibited in our experiments and how they affect Alice from the following aspects.
Predicting Alice's Future Action. When coordinating with Alice, Bob should be able to predict Alice's future actions to efficiently distribute the work and avoid conflicts (Figure 7ab).
Helping Alice's Belief's Update. In addition to directly achieving predicates Alice's goal, Bob also help by influencing Alice's belief update. A typical behavior is that when Bob opens containers, Alice can update her belief accordingly and find the goal object more quickly (Figure 7c). This is the main reason why Bob with random actions sometimes help speed up the task too.
Multi-level Actions. The current baselines do not consider plans over low-level actions (c.g., pathfinding). This strategy significantly decreases the search space, but will also result in inefficient

---

Published as a conference paper at ICLR 2021
Human Oriving Alloc
Agend Drivin Alce
NPas
0.8
APa
Hybrid
Human Rating
Reward
9.6
0.2
fruman Oriving Mis
PyjAt Driving Mice
03: 03 ES 1.0
0.7 65
00
13
Helghdnese
Emist
Figure 8: a) Success rate (x axis) and speedup (y axis). b) Cumulative reward with real humans or with the human-like agent. с) Subjective ratings from Exp. 2. Here, Alice refers to humans or the human-like agent acting alone, whereas HP, Hybrid, and HPRG indicate different AI agents helping either humans or the human-like agent. All results are based on the same 30 tasks in the test set.
pathfinding and inability to predict other agents' future paths. Consequently, Bob agent sometimes unintentionally blocks Alice (Figure 7d). A better Al agent should consider actions on both levels.
False Belief. Actions taken by an agent may cause another agent to have false beliefs (Figure 7e).
6 HUMAN EXPERIMENTS
Our ultimate goal is to build AI agents that can work with real humans. Thus, we further conduct the following two human experiments, where Alice is controlled by a real human.
recruited 6 subjects to perform tasks alone by controlling Alice. Subjects were given the same observation and action space what the human-like agent had access to. They could click one of the visible objects (including all rooms) and select corresponding action (e.g.. "walking towards", "open") from a menu to perform. They could also choose to move forward or turn left/right by pressing arrow keys. We evaluated 30 tasks in the test set. Each task was performed by 2 subjects, and we used the average steps they took as the single- aagget performance for that task, which is then used for computing the speedup when Al agents help humans. The performance of a single agent when being controlled by human or by a human-like agent in these 30 tasks shown in Fig. 8ab with the label of Alice. Human players are slightly more efficient than the human-like agent but the difference is not significant, reported by the t-test over the number of steps they took (t() = -1.63, p = .11).
Experiment 1: Human performing tasks alone. In this experiment,
Experiment 2: Collaboration with real humans. This experiment evaluates how helpful AI agents are when working with real humans. We recruited 12 subjects and conducted 90 trials of human-Al collaboration using the same 30 tasks in Exp. 1. In each trial, subject one three bascline agents, HP, Hybrid, and HPRG. to perform a task. After each trial, subjects asked rate the AI agent they just worked with on scale of 1 to 7 based on three criteria commonly used in prior work (Hoffman, 2019): i) how much the agent knew about the true goal (1 some knowledge,
randomly paired with
perfect knowledge), ii) how helpful you found the agent was (1 hurting, 4 neutral, 7- very heipful), and iii) whether you would trust the agent to do its job (I no trust, 4- neutral, 7 full trust). For fair comparison, we made sure that the random goal predictions for HPRG were the same as the ones used in the evaluation with the human- like agent.
no knowledge,
As shown Figure the ranking of the three baseline AI agents remains the same when the human- like agent is replaced by real humans, and the perceived performance (subjective ratings) is consis- tent with the objective scores. We found no significant difference the objective metrics between helping humans and helping the human-like agent; the only exception is that, when paired with real humans, HPRG had a higher success rate (and consequently higher average cumulative reward). This is because humans recognized that the AI agent might have conflicting subgoals and would finish other subgoals first instead competing over the conflicting ones with the Al agent forever, whereas the human-like agent was unable to do Appendix D.3 shows example. This adaption gave humans a better chance to complete the full goal within the time limit. We provide more details of the procedures, results, and analyses of the human experiments in Appendix D.
8

---

Published as a conference paper at ICLR 2021
7 CONCLUSION
In this work, we proposed am Al challenge to demonstrate social perception and human-Al collab- oration in common household activities. We developed multi-agent virtual environment to test Al agent's 's ability to reason about other agents' mental states and help them in unfamiliar scenarios. Our experimental results demonstrate that the proposed challenge can systematically evaluate key aspects of social intelligence at scale. We also show that our human-like agent behaves similarly to real humans in the proposed tasks and the objects metrics consistent with subject ratings.
Our platform opens exciting directions of future work, such as online goal inference and direct communication between agents. We hope that the proposed challenge and virtual environment can promote future research on building more sophisticated machine social intelligence.
ACKNOWLEDGMENTS
The information provided in this document derived from an effort sponsored by the Defense Advanced Research Projects Agency (DARPA), and awarded to Raytheon BBN Technologies under Contract Number HR001 120C0022.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971, 2016.
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-99 95, 2018.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:190.007528, 2019.
Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, (44): 1-10, 2017.
Nolan Bard, Jakob Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 1032163,,,, 2020.
Simon Brodcur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron C. Courville. Home: a household multimodal environment. CoRR, abs/1711.11001,, 2017. URL http: / wawiiviiiivv///ai//eeaa/aaaaavva org/ abs/ abs/1711 /111/111111 11017.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and in games, 4(1):1-43, 2012.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, pp. 5175-5186, 2019.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Wongun Choi and Silvio Savarese. Understanding collective activitiesof people from videos. JEEE fransactions on pattern analysis and machine intelligence, 1626):242 2013.

---

Published as a conference paper at ICLR 2021
Abhishek Das. Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embod- ied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2054-2063, 2018.
Kerstin Dautenhahn. Socially intelligent robots: dimensions of human-robot interaction. Philo- sophical transactions of the royal society B: Biological sciences, 362(1480)6679-74,, 2007.
David Fouhey, Wei-cheng Kuo, Alexei Efros, and Jitendra Malik. From lifestyle vlogs to everyday interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4991-5000, 2018.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Damian Mrowca, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, James J. DiCarlo, Josh McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Threedworld: A platform for interactive multi-modal physical simulation, 2020.
Xiaofeng Gao, Ran Gong, Tianmin Shu, Xie, Shu Wang, and Song-Chun Zhu, Vrkitchen: an interactive 3d virtual environment for task-oriented learning. arXiv, abs/1903.05757, 2019.
Michael A Goodrich and Alan C Schultz. Human-robot interaction: survey. Foundations and trends in human human-computer interaction, 1(3):203- 2007.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: visual question answering in interactive enviromments. CoRR, abs/1712.03316, 2017. URL http: / www.v.wrg...a...aaa........oooct abs / 1112..1111111 03316. 6.
Barbara Grosz and Sarit Kraus. Collaborative plans for complex group action. Artificial Intelligence, 1996.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Guy Hoffman. Evaluating fluency in human- collaboration. IEEE Transactions Human- Machine Systems, 49(3):209-216 2019.
Mostafa S Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierar- chical deep temporal model for group activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1971-19888 2016.
Max Jaderberg, Wojeiech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukeuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443) 865, 2019.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artifi- cial intelligence experimentation. In ICCAI, pp. 4246-4247, 2016.
Henry A Kautz. A formal theory of plan recognition and its implementation. Reasoning about plans, pp. 69-125, 1991.
Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In European Conference on Computer Vision, pp. 201-214. Springer, 2012.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Richard E Korf. Planning as search: A quantitative approach. Artificial intelligence, 33(1):65-88, 1987.
Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, and Sanja Fidler. Synthesizing environment-awwan activities via activity sketches. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6291-6299, 2019.
10

---

Published as a conference paper at ICLR 2021
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compenee processing systems, pp. 6379-6399 6390, 2017.
environments. In Advances in neural information
Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. CoRR, abs/1 809.00088, 2018. URL http: / /awxiiv.vv.// httaaaiiioo.ogabaaeee/o wogg/bbs/1/ 1809.00786.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Stefanos Nikolaidis, Ramya Ramakrishnan, Keren Gu, and Julie Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In 2015 10th ACM/IEEE International Conference on Human-Role Interaction (HRI), pp. 189-196. IEEE, 2015.
OpenAI. Openai five. https: / /blog.looo// openai www.////aaa-iiiiiiaeeeee fiee, 2018.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8502, 2018.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. arXiv preprint arXiv: (5f2:00880077444 2018.
Miquel Ramurez and Hector Geffner. Plan recognition as planning. In Proceedings of the 2 international joint conference on Artifical intelligence. Morgan Kaufmann Publishers Inc, pp- 1778-1783, 2009.
Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A multi-agent playground. arXiv preprint arXiv: 00..0,1144 2018.
Leonel Rozo, Sylvain Calinon, Darwin G Caldwell, Pablo Jimenez, and Carme Torras. Learn- ing physical collaborative robot behaviors from human demonstrations. IEEE Transactions on Robotics, 3333)511,- 2016.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Phil ip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188 2019.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9339- 9347, 2019.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. arXiv preprint arXiv:1912.01734, 2019.
Tianmin Shu and Yuandong Tian. M 3 rl: Mind-aware multi-agent management reinforcement learning. arXiv preprint 1rxiv: 111.00000007 2018.
Tianmin Shu, Dan Xie, Brandon Rothrock, Sinisa Todorovic, and Song Chun Zhu. Joint inference of groups, events and human roles in aerial videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4576-4584, 2015.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hicrarchical and interpretable skill acquisition in multi-task reinforcement learning, arXiv preprint arXiv:1712.072%. 2017.
Michael Shum, Max Kleiman- Weiner, Michael Littman, and Joshua B Tenenbaum. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6163-6170, 2019.

---

Published as a conference paper at ICLR 2021
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteck Alahari. Charades-ego: large-scale dataset of paired third and first person videos. arXiv preprint 000Y::,804.09666, 2018.
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: massively mul- tiagent game environment for training and evaluating intelligent agents. arXiv preprint 17X7049903.00784. 2019.
Tijmen Tieleman and Geoffrey Hinto. Lecture 6.5 rmsprop:: Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Leurning, 2012.
Tomer Ullman, Chris Baker, Owen Macindoe, Owain Evans, Noah Goodman, and Joshua B Tenen- baum. Help or hinder: Bayesian models social goal inference. Advances in neural informa- fion processing systems, pp. 1874-18822 2009.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Felix Warneken and Michael Tomasello. Altruistic helping in human infants and young chim- panzees. Science, 31 311(6665))300--1303 2006.
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photoreal- istic environments with point cloud perception. CoRR, abs/1904.033611, 2019. URL http: / //arxiv. abs abbs/1904 03461.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint arXiv: (0.Xiv:88010000 180100209, 2018.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1168-9079, 2018.
Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In Proceedings of the IEEE International Conference on Computer Vision, pp. 483-492. 2017.
12

---

Published as a conference paper at ICLR 2021
A VIRTUAALHOME-SCCIAL
A. COMPARISON WITH EXISTING PLATFORMS
There have been many virtual environments designed for single-agent and multi-agent tasks. Table summarizes the key features the proposed VirtualHome-Soce in comparison with existing virtual platforms. The key features of our environment include i) multiple camera views, ii) both high-level and low-level actions, iii) humanoid avatars with realistic motion simulations, iv) built-in human-like agents emulating human behaviors in household activities, and v) multi-agent capacities.
Critically, VirtualHome-Sciial enables collecting and displaying human activities realistic envi- ronments, which is key function necessarily for social perception and human-AI collaboration. In contrast, existing multi-agent platforms do no offer such functionality.
Table 1: We compare VirtualHome- with existing embodied single-agent and multi-agent
platforms on the following aspects: 1) action space (high-level actions and/or low-level actions),
2) views (3rd person and/or egocentric views), realistic environments, 4) humanoid agents, 5)
human-like built-in agents that other agents can interact with, and 6) multi-agent capabilities.
Platform
Action
Views
Realistic
Humanoid
Human-like Agent
Multi-agent
Overcooked (Carroll et al., 20,99 2019)
High/Low Low
3rd Person
No
No
Yes
Yes
Malmo (Johnson 2016)
High/Low
3rd Person/Ege
No
No
Yes
ThrleDWorld (Gan et al., 2020)
High/t ow
3rd Person/Ego
Yes
No
No
Yes
VRKitchen (Gao ol., 2019)
High/Low
Person/Ego
Yes
No
AI2-THOR (Kolve et al., 2017)
High/Low
Ego
Yes
No
No
Yes
House (Wu el al., 2018)
Low
Ego
Yes
No
No
HoME (Bmdeur et 2017)
Low
Egn
Yes
No
No
No
Gibson (Xia et al., 2018)
Low
Ego
Yes
No
No
No
Habitat
2019)
Low
Ego
No
No
VirtulHome-Socii
High/Low
3rd Person/Ego
Yes
Yes
Yes
Yes
A.2 ENVIRONMENT DESCRIPTION
The environment is composed different apartments with objects that can be placed to generate diverse scenes for the Watch and Help stages. Each object contains class name, set of states, 3D coordinates and an index for identification, which needed for action commands that involve object interaction. The object indices are unique and consistent in the scene so that agent can track the identities individual objects throughout an episode.
A.2.1 APARTMENTS
Training Apartments
Testing Apartments
Figure 9: Apartments used in Home-Socalll The last two apartments are uniquely used as helping environments during the testing phase.
We provide 7 distinctive apartments in total shown in Figure 9. For the purpose of testing agents' generalization abilities, in the Watch-And-Hi -nd- challenge, the last two apartments are held out for the helping environments in the testing set exclusively.
13

---

Published as a conference paper at ICLR 2021
Figure 10: Avatars available in VirualHHom-SSciial.
a Egocentric
Third Person
Symbolic
Multi-Modal
cake
character
sofa
table
milk
bounds: (bx, by, bx stalver: drowd
center: (t.02)
character
Current Action Space Object Interaction
Navigation
cabinet. 11 open/close/pur
walk to kitchen/ba /btphhatthho/..ooommmmmo
cabinet.2: 2: open/cl open/close/
Low-Level
oven. apen/close//ee /ptt/////// pat//turttonnnnnntoeeeeo
grab/put
right/left
pan.5: grab/put.
move forward
toaster.6: open/close/ http://coooo/////tteeeeeeee pptta//rttttoooononoom/
Figure 11: a) Virtual SociaSocal provides egocentric views, third-person views and scene graphs with symbolic state representations objects and agents. It also offers multi-modal inputs (RGB, segmentation, depth, 3D boxes and skeletons). b) Illustration of the action space at one step.
A.2.2 AVATARS
VirtualHome-Socia provides pool of diverse humanoid avatars (see Figure 10). This allows to randomly sample different avatars for both agents in the Watch-And-Hod challenge. We hope this can help reduce the biases in the environment. The supplementary video shows an example of this, where the clothing color indicates the role of each agent. For the public release of the platform, intend further increase diversity the avatar pool.
A.2.3 OBSERVATION
The environment supports symbolic and visual observations (Figure 11a), allowing agents to learn helping behaviors under different conditions. The visual observations provide RGB, depth, semantic and instance segmentation, albedo and luminance, normal maps, 3D skeletons and bounding boxes. Building upon Liao al. (2019), we represent the symbolic observations as state graph with each node representing the class label and physical state of an object, and each edge representing the spatial relation of two objects. The environment also provides multiple views and supports both full observability and partial observability settings.
We show examples the observations in the supplementary video. In addition to the world states, our system also allows users to include direct messages from other agents as part of the observation for an agent.
A.2.4 ACTION SPACE
As shown in Figure Il, agents in VirtualHome- SSciall can perform both high-level actions, such as navigating towards known location, interacting with observed object, and low-level actions,
14

---

Published as a conference paper at ICLR 2021
Goal ONicclaee dinnertablej 2
Sampled State
Hierarchal Planner
Observed
pans 12
MCTS
Belief
din satioa
High Plan
CN112.
ON/S2
Sampled
RP
detee 12
Symbolic Observation
Low-level Plan
Environment
walk <12>
1. walk <t
walk
2. grab <12>
5. put <31s
walk <108>
Figure 12: Schematic of the human-like agent. Based the state graph sampled from the belief, the hierarchical planner searches for high-level plan over subgoals using MCTS; then RP searches for a low-level plan over actions for each subgoal. The first action of each plan is sent back the environment for execution.
Prior
Observation
Posterior
wine glass
apple
redio.1
glass
apple:
Location distribution of wine glase
glass
appla.3
Drawer
Fridge
Counter
Gabinat
Drawer
Fridge Counter Cabinet
bock.3
The agent's ' belief is represented as the location distribution of objects, and is updated at each step based on the previous belief and the latest observation. In the example, the open cabinet reveals that the wine glass can not be in there, and that there is an apple inside, updating the belief' accordingly.
Figure 13:
such turning or moving forward for small step. For actions involving interactions with entities (objects or other agents), agent needs to specify the indices of the intended entities (e.g., "grab (3)" stands for grabbing the object with id 3). An agent can only interact with objects that are within its field of sight, and therefore its action space changes at every step. When executing navigation actions, an agent can only move meter towards the target location within one step. On average, an agent's action space includes 167 different actions per step.
A.3 HUMAN-LIKE AGENT
We discuss how the human-liks agent works more details here. The agent pipeline can be seen in Figure 12. The agent has access to partial observation of the environment, limited to the objects that are in the same room and not in some closed container. The agent equipped with a belief module (Figure 13), that gives information about the unseen objects. under the assumption that the existence of objects in environment is known, but not their location. For each object in environment, the belief contains distribution of the possible locations where it could be. We adopt uniform distributions as the initial belief when the agent has not observed anything.
At each time, the agent obtains partial observation, and updates its belief distribution accordingly. Then, the belief module samples a possible world state from the current distribution. To ensure that the belief state is consistent between steps, only resample object locations that violate the current belief (e.g. an object was believed to be in the fridge but the agent sees that the fridge is in fact empty).
Based on the sampled state, a hierarchical planner will search for the optimal plan for reaching the goal, based the goal definition. Specifically, we use MCTS to search for sequence of subgoals
15

---

Published as a conference paper at ICLR 2021
(i.e., predicates), and then each subgoal is fed to regression planner (RP) that will search for an action sequence to achieve the subgoal. For the high-level planner, the subgoal space is obtained by the intersection between what predicates remained to be achieved and what predicates could be achieved based on the sampled state. Note here each subgoal would specify an object instance instead of only the object class defined in the goal that the low-level planner will be informed which object instances it needs to interact with. For instance, in the example illustrated in Figure 12, there are two plates (whose indices are 12, 52) and the dinner table's index is 31 according to the sampled state. There are two unsatisfied goal predicates (i.e., two ON (plate, dinnertable)) ) ON
then possible subgoal space for the high-level planner would be {CN ( (12, 31), ON (52, 31) }. For RP, it starts from the state defined by the subgoal and searches for the low-level plan backward until it finds an action that is part of the current action space of the agent.
To mimic human behaviors in home setting, we also expect the human-like agent to close contain- ers unless it needs look inside or put objects into them. For that, we augment the MCTS-based high-level planner with heuristics for the closing behavior - the agent will close an container when finds no relevant goal objects inside or has already grabbed/put in the all target objects out that container. We find that this augmentation makes the overall agent behaviors closer what real human would do in a household environment.
Thanks to the hierarchical design, the planner for the human-like agent can be run in real-time (on average, replanning at each step only takes 0.05 second). This also gives the agent a bounded rationality, in that the plan is not the most optimal but is reasonably efficient. The optimality of the planner can be further tuned by the hyper-parameters of MCTS, such as the number of simulation, the maximum number steps in the rollouts, and the exploration coefficients.
A.4 SPECIFICATIONS
The environment can run single or multiple processes, A single process runs at 10 actions per second. We train our models using 10 processes in parallel.
B MORE DETAILS ON THE CHALLENGE SETUP
B.1 PREDICATE SETS FOR GOAL DEFINITIONS
Table 2: Predicate sets used for defining the goal of Alice in five types of activities.
Set up a dinner table
ON ( dinnertable), (fork,, dinnertable).
ON (ww.ergllas,, cim.eraaae..immmtttb.llleeee ON (wineglass, dinnertable)
Put groceries
NN wuwpcaaeeffiiceeee. IN iipanccc,,, fridge), IN (poundcake, fridge),
IN (p
IN [i dinnertalll
Prepare a meal
ON offofffepppg.daannetttaalllyyeor
ON [tgaacckkehhinnercaeeeee/e/lllle ON (tpp:p/ig..ccaaaccaeaeeeaaaaaaaa ON otfpueeehhhaatttteeeeelllliaaaaa
Wash dishes
IN HKD_atcceEEhhaahhheeee....o.dddd
IN Masesssiiaesssssahhaaaaaeeeeeeen
Read book
HOLD BDDDD ON (cupcake,
(cupck,,, coffectablc),
ON (pudding,, opp coffetttalllooooooboot.oocccc.ll ON ottppp/icyy.offffereeeeleelllle.
Table 2 summarizes the five predicate used for defining goals. Note that Virualloome-ocial supports more predicates for potential future extensions on the goal definitions.
B.2 TRAINING AND TESTING SETUP
During training, randomly sample one of the 1011 I training tasks for setting up training episode. For evaluating an Al agent on the testing set, we run each testing task for five times using different random seeds and report the average performance.
16

---

Published as a conference paper at ICLR 2021
1.0
Objects
0.6
0.4
0.2
Locations
Initial location distributions of all objects in the environment. Rows objects and columns are locations. The color indicates the frequency.
Figure 14:
For training goal inference, we also provide an additional training set of 5303 demonstrations (with- pairing helping environments) synthesized in the 5 training apartments. Note that these demon- strations are exclusively used for training goal inference models and would not be used for helping tasks.
B.3 DISTRIBUTION OF INITIAL OBIECT LOCATIONS
Figure shows the initial location distribution of all objects in the helping environments sampled for the challenge, and Figure 15 shows the initial location distributions for only the objects involved the goal predicates.
C IMPLEMENTATION DETAILS BASELINES
C.I GOAL INFERENCE MODULE
Figure shows the architecture the goal inference model described the paper, where 128 indicates the dimension of vectors. In this network, the LSTM has 128 hidden units and the MLP units are comprised two 128-dim fully connected layers. For both node embeddings and the latent states from the LSTM, average pooling.
17

---

Published as a conference paper at ICLR 2021
apple
0.16
0.13
0.09
0.1
0.16
0.01
0.05
0.16
0.09 0.04
coffeepot
0.36
0.09
0.36
0.18
0.35
cupcake
0.19
0.09
0.16
0.07
0.09
0.09
0.09
0.16
0.07
cutleryfork
0.17
0.03
0.19
0.14
0.17
0.06
0.06
0.08
0.08
0.30
juice
0.21
0.17
0.08
0.21
0.23
0.1
Objects
0.25
pancake
0.38
0.25
0.31
0.06
0.15
0.23
0.16
0.19
0.05
0.01
0.11
0.11
0.20
poundcake
0.42
0.19
0.13
0.23
0.03
pudding
0.16
0.16
0.14
0.13
0.06
0.14
0.14
0.07
waterglass
0.18
0.03
0.19
0.16
0.04
0.12
0.12
0.08
0.04
0.04
0.21
0.25
0.14
0.14
0.25
0.05
wineglass
0.17
0.05
0.19
0.14
0.17
0.09
0.06 0.05
0.05
0.00
dichwasher
bench
cabinet
ON coffeetable
ON Mithhnncbiretts
ON kitchnncontter
ON kitchemaallc
ON pillow
ON plate
ON tostand
IN
Locations
Figure 15: Initial location distributions of the goal objects. Rows are objects and columns are locations. The color indicates the frequency.
State Encoder
Erfyy ancodar
Visible Objects
namer tpaster
Predicto
states
MLP
coorde.cck 1y) bx
Predicata MLP
Predam Cuns
Transformer
LSTM
AvePPoo
LSTM
Figure 16: Network architecture of the goal inference model, which encodes the symbolic state sequence in demonstrations and infers the count for each predicate.
18

---

Published as a conference paper at ICLR 2021
State Encoder
Ooject
Visible Objecte
Diiy
Loghs
Youeler
Object Selection Pollc
atdic Cy. by
MLP
Softmax
AvgPool
Transformer
Clcsenoss
Prediction
(Pudilary Tack)
Tobi
Type Polley
Open
"Open 2>
Goal Encoder
Close
Predicate
Doe erooder
LSTM
MLP
Value
Crab
Wal
Put
ONIPlste, Oinerrbbio
Dyype
F-din
Figure 17: Network architecture of the low-level policy in the HRL baseline. Note that the object selection policy also considers "Null" dummy object node for actions that do not involve object, which is not visualized here.
C.2 HIERARCHICAL PLANNER
The hierarchical planner (HP) baseline is similar the planner designed for the human- like agent (Section A.3) but has its own observation and belief. When given the ground-trun -ruth goal of Alice, the MCTS-based high-level planner will remove the subgoal that Alice is going to pursue from own subgoal space.
C.3 GENERAL TRAINING PROCEDURE FOR RL-BASED APPROACHES
We train the high-level RL policy by giving ground-trutt truth goals and by using RP as the low-level planner to reach the subgoals sampled from the high-level policy. Whenever goal predicate is satisfied (either by Alice or by Bob), Bob will get reward of +2; it will also get a -0.1 penalty after each time step. We adopt the multi-task RL approach introduced in Shu et al. (2017) train the low- level policy a single-agent setting, where we randomly sample one the predicates in the goal in each training episode and set it to be the objective for Bob. This is to ensure that Bob can learn to achieve subgoals through the low-level policy by himself. The HRL baseline implemented by combining the high-level and low-level policies that are trained separately.
C.4 LOW-LEVEL POLICY
Figure 17 illustrates the network architecture for the low-level policy. We use the symbolic obser- vation (only the visible object nodes) input, and encode them in the same way Figure 16 does. We encode two object classes the given subgoal sg (i.e., predicate) through word2vec encod- ing yielding 128-dim vectors. We then concatenate these vectors and feed them to fully connected layer to get a 128-dim goal encoding. Based on the goal encoding, we further get two attention vectors, Johjear and Oyyee- Each element of the attention vectors ranges from each object node, we use the element-wiss product of oobecct and its node embedding to get its re- shaped representation. Similarly, we can get the reshaped context representation by an element-wise product of the context embedding and type. This is inspired by a common goal-conditioned policy network architecture (Chaplot al., 2018; Shu et al., 2017), which helps extract state information relevant to the goal. From each reshaped node representation, we can get scalar for each object representing the log-likelihood of selecting that object to interact with for the current action. After a softmax over all the object logits, get the object selection policy Tobjecr(k)o .sg), where k is the index of the object instance selected from all visible objects (which also includes "Null" for actions that do not involve object). For encoding the history, feed the reshaped context representation to an LSTM with 128 hidden units. Based on the latent state from the LSTM, we get i) the action type policy (apeealo sg), which selects an action type (i.e., "open," "close," "grab," "put," "walk,"
1. For
19

---

Published as a conference paper at ICLR 2021
State Encoder
Ertby encoder
Visible Objects
name: toaster
state:
coords. OX cy. oZ bx. by, bz
AvaPool d
ON(Plate, Dinnertable)
Transformer
Objjett Subgoa Policy Location Subgoai Policy
MLP
Goal Encoder
Predicate Combination
Goel enooder
LSTM
ONialatee Dinnerabble)
SumPool
sigmaid
Og
ONiPlate, Dinnertable)
ON(Wineglase, Dinnertable)
Figure 18: Network architecture the high-level policy for the Hybrid and the HRL baselines.
or "follow"), and ii) the value function V sg). The sampled & and jointly define the action for the Al agent. Note that some sampled combinations may not be valid actions, which will not be executed by the VirtualHome-Sove
environment.
In addition to the policy and value output, we also build a binary classifier for each visible node to predict whether it is close enough for the agent interact with according to the symbolic graphs. This closeness prediction serves an auxiliary prediction which helps the network learn a better state representation and consequently greatly improves the sample efficiency.
In each training episode, we randomly sample a predicate from the complete goal definition as the final goal of the agent. The agent gets reward of 0.05 for being close to the target object and/or location, and reward of 10.0 when it grabs the correct object or puts it to the correct location. Note that when training the low-level policy, we set up single-agent environment to ensure that the AI agent can learn to achieve predicate by itself.
We adopt a 2-phase currieulum learning similar to Shu al. (2017): In the first phase, we train policy for grabbing the target object indicated in the goal. During this phase, a training episode terminates whenever the agent grabs the correct type of object. the second phase, we train another policy which learns to reuse the learned grabbing policy (which is deployed whenever the "grab" action type is sampled) to get the goal object and then put the grabbed object to target location specified in the goal.
We use off-policy advantage actor-critic (A2C) (Mnih et al., 2016) for policy optimization. The network is updated by RMSprop (Ticleman & Hinto, 2012) with learning rate of 0.001 and a batch size of 32, The first phase is trained with 100,000 episodes and the second phase trained with 26,000 episodes.
C.5 HIGE-LEVEL POLICY
As Figure 18 depicts, the high-level policy (used by Hybrid and HRL baselines) has similar architecture design as the low-level policy. Compared with the low-level policy, it does not need to define object selection policy; instead, based on the latent state from the LSTM, it outputs the policy for selecting the first and the second object class in predicate to form subgoal". It also augments the goal encoder in the low-level policy with a sum pooling (i.e., Bag of Words) to aggregate the encoding of all predicates in a goal, where predicates are duplicated w.r.t their counts in the goal definition (e.g.. in Figure 18, ON (plate, dinnertable) appears twice, which means there are
Note that this is different from the subgoals generated from the high-level planner (Section A.3), which would specify object instances.
20

---

Published as a conference paper at ICLR 2021
Oracle
2.0
Oracle
1.5
Speed Up
1.0
TG
0.5
Hybrid
Hybridns
Alice
HRL
HPRG
HRLTs
Random Action
0.75
0.80
0.85
0.90
0.95
1.00
Success Rate
Figure 19: Success rate (x axis) and speedup (yaxis) of all the baselines and oracles
should be plates the dinnertable). Similar to the low-level policy, we get attention vector dg from the goal encoding to reshape the state representation. In total, the network has three outputs: the object subgoal policy for sampling the object class name in the subgoal, the location subgoal policy for sampling the target location class name in the subgoal, and a value function.
The high-level policy is trained with regression planner deployed to find low-level plan for reaching that subgoal. Note that the regression planner searches for plan based on state sampled from the agent's 's belief maintained by belief module discussed in Section A.3. It will also randomly select object instances from the sampled state that fit the defined object classes in the subgoals sampled from the high-level policy.
Similar to the low-level policy, we use off-policy A2C for policy optimization, and the network is updated by RMSprop with a learning rate of 0.001 and a batch size of 16. We first train the high- level policy in single-agent setting where the Al agent is trained to perform task by itself; we then finetune the high-level policy
full training setting where the human-like agent is also present and works alongside with the AI agent. During training, we always provide the ground-truth goal of Alice to the AI agent.
D ADDITIONAL DETAILS OF HUMAN EXPERIMENTS
D.1 HUMAN SUBJECTS
Both the collection of human plans as well as the evaluations in our user studies were conducted by recruited participants, who gave informed consent.
D.2 PROCEDURE FOR COLLECTING HUMAN PLANS
To collect the tasks for both experiments, we built a web interface on top of SituualHom-SSocal,, allowing humans to control the characters in the environment. Specifically, the subjects in our human experiments were always asked to control Alice. At every step. humans were given a set visible objects, and the corresponding actions that they could perform with those objects (in addition to
21

---

Published as a conference paper at ICLR 2021
the low-level actions), matching the observation and action space of the numan-like agent. When working with an Al agent, both the human player and the Al agent took actions concurrently.
In both experiments, human players were given short tutorial and had chance to get familiar with the controls. They were shown the exact goals to be achieved, and were instructed to finish the task as fast as possible. For each task, set the same time limit, i.e., 250 steps. A task is terminated when it exceeds the time limit or when all the goals specified have been reached.
The 30 tasks used in the human experiments were randomly sampled from the test set and were evenly distributed across 5 task categories (i.e., 6 tasks for each category).
In Experiment 2, each subject was asked to perform 7 8 trials. We made sure that each subject got to play with all three baseline Al agents in at least trials.
D.3 EXAMPLE OF HUMAN ADAPTING TO AI AGENTS WITH CONFLICTING GOALS
The main reason why real humans work better than the human-like agent when paired with an Al agent that has conflicting goal (in particular, the HPRG baseline), is that they can recognize the conflicting goal, and avoid competing over the same objects forever. Figure 20 depicts an example of this adaptive behavior from real human player in Experiment 2, which results in the completion of the task within the time limit. Note that in our experiments, a task is considered successful and terminated once all the predicates in goal have been achieved.
This also calls for an AI agent with the ability to adjust its goal inference dynamically by observing Alice's behavior in the new environment (e.g., Alice correcting a mistake made by Bob signals incorrect goal inference).
D.4 SUBJECTIVE EVALUATION OF SINGLE AGENT PLANS
To evaluate whether people think the human-like agent behaves similarly humans given the same goals, we recruited another 8 subjects. We showed each subject 15 videos, each of which is a video replay of a human or the human-like agent performing one of the 30 tasks (we randomly selected one human video and one built-in agent video for each task). For each video, subjects were given the goal and asked to rate how much they agreed with the statement, "the character the video behaves similarly to a human given the same goal in this apartment," on Likert scale of 5 (1 is "strongly disagree," " is "neutral," and 5 is "strongly agree") The average ratings for the characters controlled by the human-like agent and by the real humans are 3.38 (+0.93) and 3.72 (+0.92) respectively. We found no significant difference between the ratings for the human-like agent's 's plans and the ratings for the real humans' plans in tasks, reported by paired, two-tailed t-test (t(29) -1.35, p = 19). This demonstrates that the proposed human-like agent can produce plans that are similar real humans' plans in our challenge.
Based on the free responses collected from the subjects who rated these videos, human plans look slightly more efficient sometimes since they do not look for objects in unlikely places and avoid moving back and forth between rooms frequently. The human-like agent behaves similarly in most of the time but would occasionally search through the rooms in a counter-intuitive order due to its bounded rationality and the fact that plans are sampled stochastically.
D.5 ADDITIONAL QUANTITATIVE ANALYSES OF HUMAN EXPERIMENT RESULTS
To evaluate whether the performance of a baseline AI agent helping the human- -ika agent reflects the performance of it helping real humans, we conduct paired, two-tailed t-test for the three baselines in Experiment 2 based on their cumulative rewards. For RR. there is significant difference between helping the human- -like agent and helping real humans (t(29) -2.36, p .03) as discussed in Section 6 and Appendix D.3. However, there is no significant difference for HP (1(29) = -1.78, p .1) and Hybrid ((9) -0.5, p- 62)). This validates that, in general collaboration with
"Since focus on the agents' plans in this work, users were instructed to focus on the actions taken by the agents, rather than the graphical display of their hody motion.
22

---

Published as a conference paper at ICLR 2021
Ground-truth goal:
A random goal sampled by Bob (HPra): I8 (Vineggas,, dishwasher): 1 pouuddaaae,, dinnertable): IN pancake, fridge): 2 (K(wine, dinnertable):
Ou(plate, dinnertable): 1 oR dinneriable): OR OW(wnneglase, dinnertable) ON( fork, sinnertable):
The human-like agent and HPRG
Bob immediatoly grabs wine glass
Alice grabs wine glass
goos
puts
So she the dishwasher
il back
Bob Fest Fhe glass
So he puls the glass back to the distwwalhee again
Alice finds out
table
real human player HPRG
getting glass, Alice on other subgoals
puts wine gloes to the bable
immodiately grabsit
dishwasher
and puts
After finishing the suogoals, goes wine glass
table, completing
An example of how real human differs from the human-like agent when working with an Al agent (i.e., HPRG) with conficcing goal. this example, Bob incorrectly thinks that Alice wants to put the wine glass to the dishwasher whereas Alice actually wants put it to the dinner table. When controlled by human-like agent, Alice enters into a loop with Bob trying to change the location of the same object. The real human player, the other hand, avoids this conflict by first focusing on other objects in the goal, and going back to the conflicting object after all the other goal objects have been placed the dinner table. Consequently, the real human completes the full task successfully within the time limit.
Figure 20:
the human-like agent is comparable collaboration with real humans. Given these analyses, the training and evaluation procedure presented in this paper is both scalable and comprehensive.
ST..., i) training AI agents with the human-like agent, and then ii) evaluating them both with the human-like agent (in larger test set), and with real humans (in smaller but representative test set).
23


Reviews:
[{"review_id": "YdHb_nkFEa", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\t1. Summary of Paper:\n\t\t1. This paper contributes a new challenge task (called Watch and Help) and a multi-agent virtual environment evaluation platform (a multi-agent extension of VirtualHome called VirtualHome-Social) for goal recognition and collaborative planning. The Watch and Help task requires an evaluated agent (say Bob) to watch a single demonstration of another agent (say Alice) performing a task in isolation. These tasks are drawn from a set of 5 potential home-related task sets (e.g. reading, putting groceries away, etc.). After watching Alice perform a given task in isolation, Bob is required to help Alice perform that same task in a different environment. Bob's help or the success of the collaboration is measured by the success rate, speed up (vs. performing the task alone), and cumulative reward (+1 if goal is successful and -1 if goal is not after T timesteps) from performing the task. The article presents a \"human-like\" planning agent that uses hierarchical planning (MCTS search for sub-goal selection and regression planning for low-level action selection) over beliefs and goal states to act as the fixed demonstrator and collaborator to train the agents being evaluated. The paper also provides several benchmark agents and compares their collaborative performance with both the hierarchical planning Alice agent and a human-controlled Alice avatar.\n\t2. Strengths\n\t\t1. Clarity: \n\t\t\ta. The paper is remarkably clear while explaining its contributions, the challenge task, agent behaviours during experiments, experimental comparisons, etc.\n\t\t\tb. The paper uses visualisation figures and images from the environment to great effect explaining sequences of behaviour that would otherwise be difficult to show without the reader having to look at supplementary videos (which are nonetheless provided).\n\t\t\tc. The related work is clearly specified showing how relevant works differ from the current approach. One potential weakness in comparisons to comparable virtual environments is given later.\n\t\t\td. The different variant agents being compared in the task benchmark are clearly explained and form an appropriate experimental design.\n\t\t2. Novelty/Impact\n\t\t\ta. The work aims to showcase a new challenge task, evaluation platform, and benchmark agent performance for goal recognition followed by collaborative planning. The work as described compares favourably to similar work in evaluation platforms and benchmarks referenced in the related work section and appendix. The differences are made clear, though the use of some featured distinctions are not demonstrated in the paper (e.g. visual observations are possible but not used in the benchmarking).\n\t\t3. Experimental Rigour\n\t\t\ta. This work is not primarily about demonstrating the benefits of a particular approach over others in a particular application. It demonstrates benchmarks for agent performance in a newly introduce problem setting. From that perspective, the paper has strong experimental rigour. The experimental design is appropriate, comparing multiple baselines and oracles with several sets of experimental variants from both automated planning and reinforcement learning communities.\n\t\t\tb. The comparison of experimental variants is conducted with both a computationally-controlled agent and a human-controlled avatar to evaluate collaboration performance in increasingly realistic settings.\n\t\t\tc. The claim that the computationally-controlled Alice agent is human-like is repeated throughout the paper. This is not justified in the actual text of the main paper, but is supported to a moderate degree (human-like in strategy/planning if not movement/behaviour) through experiments with human subjects that are described in the appendix.\n\t\t\td. Effort paid to ensure diverse avatars in experimentation.\n\t\t4. Reproducibility\n\t\t\ta. The work is almost entirely reproducible, with details of all agent architectures used for experiments provided with hyperparameters and architecture design. The authors describe that the environment will be released as open-source, which will then make the article wholly reproducible. This reviewer appreciated the level of additional detail provided in the appendix to improve this area of evaluation.\n\t3. Weaknesses\n\t\t1. This paper uses the term social intelligence to motivate the context for this challenge task. Social intelligence is a much broader term than what is actually being evaluated here and would require evaluating capabilities beyond goal recognition and coordinated planning/task execution. It is suggested to replace this claim with \"goal recognition and collaborative planning\".\n\t\t2. From the motivation provided, i.e. evaluating social perception and collaboration, why is the task specifically about watching then helping and not both together or just helping or just watching or some combination of these activities fluidly occurring throughout the interaction?\n\t\t3. Further, the work itself does not explicitly motivate why this, specific challenge task for goal recognition followed by collaborative planning is necessary for moving the state of the art in human-AI collaboration forward. However, it is a small leap to see the impact of this platform/task in evaluating applications like service robotics, social robotics, collaborative human-agent task performance, video games, etc. This reviewer can understand the impact of the work, but it would be clearer to explicitly discuss this.\n\t\t4. It would be clearer to specify that this task is limited to situations where there is explicitly only one goal throughout the entire demonstration + execution episode. This is important since it precludes using this challenge task for research into agents that need to use goal recognition after the initial demonstration, potentially continuously over the course of execution. This second kind of continuous goal monitoring is more similar to real-world applications of watching and helping or assistive agents or social robotics, since the human collaborator can (and often will) change their mind.\n\t\t5. Similarly, it should be noted that there is an explicit limitation of this challenge task and the evaluation metrics to scenarios where the entire success or failure of the approach is purely based on the final team accomplishment. This is similar to situations like team sports, where all that matters is the final game score. Many real-world scenarios for human-AI collaboration, differ by also requiring individual collaborators to do well or for the primary human user to do better with collaboration (than without). For example, in a computer game where Bob represents a team-mate to Alice who is a human player, Bob can choose to steam-roll Alice and win the game by itself. However, this leads to lower subjective user experience for the human team-mate. In this case, the score might be greater than what Alice could accomplish on their own and the game might be won faster than Alice could on their own, but the experience would be different based on whether they are truly collaborating or one is over-shadowing the other.\n\t\t6. A final assumption, is that there is no difference in expertise between Alice and Bob. The human is expected to be able to competently finish the task and for Bob to linearly cut down the time taken to perform this task. There are many real-world tasks in human-AI collaboration where this assumption does not hold and there could be non-linear interactions between success-rate and speed-up due to different levels of expertise between Alice and Bob. \n\t\t7. The fixed Alice agent is called human-like through out the article and this was not properly justified anywhere in the main text of the paper. However, the appendix actually describes results that compare the performance of the computationally-controlled and human-controlled variants of Alice to human observers. This potentially justifies this weakness. For clarity, it would be valuable to refer to the presence of this validation experiment in the main paper.\n\t\t8. Why aren't there benchmark results (more than one) for the goal recognition component similar to the planning task experimentation? If both parts of the task are important, it would be valuable to provide additional experiments to show comparisons between goal recognition approaches as well, even if that is in the appendix for space reasons.\n\t\t9. There could be more analysis of the benchmark agent performance, 1) Why does the purely random agent work relatively well across tasks? 2) Why doesn't HRL work better? Is this due to less hyperparameter tuning compared to other approaches or due to some intrinsic aspect of the task itself? 3) Perhaps I missed this, but why not try a flat model-based or model-free RL without a hierarchy?\n\t\t10. There are several comments about other environments in the related work section and appendix being toy environments. However, the tasks in the environment demonstrated in this paper only use a small set of predicates as goals. Similarly, it CAN generate visual observations but that isn't used by any of the baselines in the paper. Several comparisons to related virtual environments are made in appendix, but some of the features aren't used here either (humanoid agent - this challenge task works equally well with non-humanoid avatars/behaviours and realism - visual realism is present but it isn't clear if behavioural or physical realism is present due to seeming use of animations instead of physical simulation).\n\t\t11. None of the tasks described allow the use of communication between agents or evaluate that. Other multi-agent environments like Particle Environments (below) allow for that. Communication is a natural part of collaboration and should have been mentioned if only to distinguish future work or work out of current scope.\n\t\t\ta. @article{mordatch2017emergence,  title={Emergence of Grounded Compositional Language in Multi-Agent Populations},  author={Mordatch, Igor and Abbeel, Pieter},  journal={arXiv preprint arXiv:1703.04908},  year={2017}}\n\t\t12. \"planning and learning based baselines\", \"and multiple planning and deep reinforcement learning (DRL) baselines\", etc. - There is potential for confusion with the use of terms \"planning\" and \"learning\" methods to do what both fields (automated/symbolic planning and reinforcement learning) would potentially consider as planning tasks. It would be clearer to indicate this distinction in terminology.\n\t\t13. The human-likeness evaluation experiment asked subjects to evaluate performance one agent video at a time. A more rigorous evaluation might compare two agents side by side and ask the human to guess the human performance. This could also be in addition to the current evaluation. The current evaluation is a ceiling on performance while the comparative evaluation is a potential floor.\n\t4. Recommendation: \n\t\t1. I recommend accepting this paper, which was clear, novel, empirically strong, and supremely reproducible. The strengths conveyed above outweighed the weaknesses.\n\t5. Minor Comments/Suggestions: \n\t\t1. Some minor typos in the manuscript:\n\t\t\ta. Using the inferred goals, both HP and Hybrid can offer effective. - page 6\n\t\t\tb. IN(pundcake, fridge) - appendix table 2\n\t\t\tc. This closeness perdition - appendix page 19", "extra_scores": {}, "extra_text": {}}, {"review_id": "V-7t2LdhPf2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an AI challenge where an AI agent needs to collaborate with a human-like agent to enable it to achieve the goal quicker. Two stages are defined: a watch stage and a help stage. In the first one, an AI agent  watches a human-like\nagent performing a task once and infers human-like agent’s goal from its actions. In the help stage, the AI agent helps the human-like agent to achieve the same goal in different environments as fast as possible.\n\n\nThis work tackles problems related to human-AI collaboration, a topic that poses several great challenges for the AI community, and other scientific communities too. The main theme of the paper and research presented fits well ICLR.\n\nFrom my point of view, the authors do an excellent job presenting the challenge, the multi-agent virtual platform (open source planned for the future if I understood correctly?) and all their functionalities (create agents that emulate human behaviours, the interface that supports evaluation, etc.) They also present a benchmark.\n\nHowever, I am not familiar with having a paper that has as main contribution a challenge. I think there are other contributions in the paper, but they are all presented in terms of such a challenge and the virtual environment. I found myself unsure if the authors evaluate a particular method (that the scientific community can use) or an instantiation of a method that is only valid in the virtual environment/challenge they present. To summarise, either the focus is on the main new ideas presented in the paper and the challenge/virtual environment to demonstrate those, or we make this virtual environment open source (as the plan is) and present the paper not as a regular paper, but another type of paper that describes the virtual environment (including the previous version) as a testing platform for other researchers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RB0NzFbpLFd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quality\n- I enjoyed reading the paper. In general I think it was of good quality, but mentioned in my summary there was a lot going on from lots of perspectives. T\n- This not only includes the scope but the experiments (agent and humans), large number of complex agent technologies, many types of agents. \n- Obviously the page count would not suffice and hence there is what I would consider significant core content in the appendices.\n- While the conclusion is short and to the point, the paper would have benefited from a critical evaluation/discussion of the approach taken (limitations etc). \n- In the conclusion the authors state that the experiment demonstrated the ability to evaluate aspects of social intelligence at scale. What do you mean by scale? What if I had 10, 100, 1000 agents?\n\n### Clarity\n- From a readability point of view the paper is largely OK. \n- The abstract and introduction which motivate the problem are very good.\n- I would have liked to have seen further \n- Lots of moving parts/components/subsystems made it sometimes difficult to get your head around the whole system being described. \n- It wasn't clear to me as to the significance of the domain chosen. What is the long term goal with this work? Is it social intelligence for human robotics?\n- I think the authors could do a better job of explaining/interpreting results.\n\n### Originality\n- There were certainly many original aspects to the work. But I believe it was more combining existing approaches and techniques to solve the problem rather than coming up with new learning techniques from scratch.\n\n### Significance\n- The work is significant to researchers who are interested in modelling social intelligence, robotics and multi-agent systems. \n- So perhaps with that in mind this paper may be more suited to a conference like AAMAS (Autonomous Agents and Multi-Agent Systems) rather than ICLR which is more machine learning focused. I'm not making a judgement here, just something to consider with this work going forward.\n\n### Other questions and comments for the authors:\n\n- Does Bob learn the sequence of actions that Alice take to achieve the task or just the end state of Alice's task? If it is the latter, does Bob work as well if he doesn't participate in the watch stage and is just told what the goal is in the help stage?\n\n- If Bob is helping Alice, does he automatically assume that Alice is trying to achieve the task that Bob watched Alice previously doing? Or does Bob have to observe Alice a second time and then infer that she is trying to achieve a goal which Bob knows how to help with because he previously observed her doing it in the Watch phase?\n\n- In the watch phase, does Bob ever get confused? For example, assuming that Alice has started task A, when in fact she started task B and Bob starts helping with task A?\n\n- You don't mention agent communication which is often an important part of modelling agent social interaction. What if Alice communicated to Bob her goal?\n\n- I really like Figure 6a.This is an excellent chart except for the y-axis removal of sections between 0.6 and 1.6 and 1.8 and 2.0. I think this is misleading. Perhaps a log scale would be appropriate. \n\n- That aside I think this type of chart tells the reader more about the performance of the different agents compared to 6b which plots the reward. With 6b I need some additional interpretation which is a bit lacking in the text.\n\n- What are the error bars in 6b? Standard deviation, standard error, confidence intervals?\n\n- While the addition of human experiments is good and comprehensive, perhaps that could be the topic of another paper and in its place have section on discussing the pros/cons of the results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CN3JCf2NvJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper targets to demonstrate social perception and human-AI collaboration in common household activities. It shows the development of a multi-agent virtual environment that is used to test an AI agent’s ability to reason about other agents’ mental states and help them in unfamiliar scenarios. This is performed by presenting an experimental study over specifically selected scenarios which involve aspects of social intelligence.\nThe set-up includes a watch step, based on goal inference modelling and of a help stage, based on multiple planning and reinforcement learning. This is an interesting set up which can model some aspects of human behavior. The implementation description, which involves specific activities and their analysis and planning, illustrates the presented concepts and methodology.\nHowever, the presentation, in the form of large descriptions, in the main part of the paper, the Appendices and the Supplementary material do not match the requirements of a conference paper; it is rather the presentation of a platform, that would fit a demo of a prototype or a submission to a journal. There is no discussion, justification, or evaluation of the specific algorithmic choices in modules, e.g., in the goal inference module. In fact the paper should be reduced in length, focusing on the AI related issues and selections adopted and how they were implemented, with the detailed platform description be included in the supplementary, if possible, or in a url where this information could be reached.\nSome vague statements should be better defined, e.g., react 'as quickly as possible'.     ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:
This paper provides an interesting and unique challenge problem on human-AI collaboration, with sample baselines. I think this is an extremely important topic and the community should embrace such challenge problems.

Discussion:
Reviewers agreed this paper should be accepted, particularly after seeing that ICLR has accepted such challenge papers in the past.

Recommendation:
I'd really like to see this get a spotlight as it would be great to highlight this innovative challenge to the community. 
Decision: Accept (Spotlight)