Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper targets to demonstrate social perception and human-AI collaboration in common household activities. It shows the development of a multi-agent virtual environment that is used to test an AI agent’s ability to reason about other agents’ mental states and help them in unfamiliar scenarios. This is performed by presenting an experimental study over specifically selected scenarios which involve aspects of social intelligence.
The set-up includes a watch step, based on goal inference modelling and of a help stage, based on multiple planning and reinforcement learning. This is an interesting set up which can model some aspects of human behavior. The implementation description, which involves specific activities and their analysis and planning, illustrates the presented concepts and methodology.
However, the presentation, in the form of large descriptions, in the main part of the paper, the Appendices and the Supplementary material do not match the requirements of a conference paper; it is rather the presentation of a platform, that would fit a demo of a prototype or a submission to a journal. There is no discussion, justification, or evaluation of the specific algorithmic choices in modules, e.g., in the goal inference module. In fact the paper should be reduced in length, focusing on the AI related issues and selections adopted and how they were implemented, with the detailed platform description be included in the supplementary, if possible, or in a url where this information could be reached.
Some vague statements should be better defined, e.g., react 'as quickly as possible'.     
review_title: Interesting set up and prototype; should be revised to fit the scope of a conference paper presentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces an AI challenge where an AI agent needs to collaborate with a human-like agent to enable it to achieve the goal quicker. Two stages are defined: a watch stage and a help stage. In the first one, an AI agent  watches a human-like
agent performing a task once and infers human-like agent’s goal from its actions. In the help stage, the AI agent helps the human-like agent to achieve the same goal in different environments as fast as possible.


This work tackles problems related to human-AI collaboration, a topic that poses several great challenges for the AI community, and other scientific communities too. The main theme of the paper and research presented fits well ICLR.

From my point of view, the authors do an excellent job presenting the challenge, the multi-agent virtual platform (open source planned for the future if I understood correctly?) and all their functionalities (create agents that emulate human behaviours, the interface that supports evaluation, etc.) They also present a benchmark.

However, I am not familiar with having a paper that has as main contribution a challenge. I think there are other contributions in the paper, but they are all presented in terms of such a challenge and the virtual environment. I found myself unsure if the authors evaluate a particular method (that the scientific community can use) or an instantiation of a method that is only valid in the virtual environment/challenge they present. To summarise, either the focus is on the main new ideas presented in the paper and the challenge/virtual environment to demonstrate those, or we make this virtual environment open source (as the plan is) and present the paper not as a regular paper, but another type of paper that describes the virtual environment (including the previous version) as a testing platform for other researchers.
review_title: Interesting AI challenge, but enough contribution for a regular paper for ICLR?

final_score: rating: 7: Good paper, accept
review_content: 	1. Summary of Paper:
		1. This paper contributes a new challenge task (called Watch and Help) and a multi-agent virtual environment evaluation platform (a multi-agent extension of VirtualHome called VirtualHome-Social) for goal recognition and collaborative planning. The Watch and Help task requires an evaluated agent (say Bob) to watch a single demonstration of another agent (say Alice) performing a task in isolation. These tasks are drawn from a set of 5 potential home-related task sets (e.g. reading, putting groceries away, etc.). After watching Alice perform a given task in isolation, Bob is required to help Alice perform that same task in a different environment. Bob's help or the success of the collaboration is measured by the success rate, speed up (vs. performing the task alone), and cumulative reward (+1 if goal is successful and -1 if goal is not after T timesteps) from performing the task. The article presents a "human-like" planning agent that uses hierarchical planning (MCTS search for sub-goal selection and regression planning for low-level action selection) over beliefs and goal states to act as the fixed demonstrator and collaborator to train the agents being evaluated. The paper also provides several benchmark agents and compares their collaborative performance with both the hierarchical planning Alice agent and a human-controlled Alice avatar.
	2. Strengths
		1. Clarity: 
			a. The paper is remarkably clear while explaining its contributions, the challenge task, agent behaviours during experiments, experimental comparisons, etc.
			b. The paper uses visualisation figures and images from the environment to great effect explaining sequences of behaviour that would otherwise be difficult to show without the reader having to look at supplementary videos (which are nonetheless provided).
			c. The related work is clearly specified showing how relevant works differ from the current approach. One potential weakness in comparisons to comparable virtual environments is given later.
			d. The different variant agents being compared in the task benchmark are clearly explained and form an appropriate experimental design.
		2. Novelty/Impact
			a. The work aims to showcase a new challenge task, evaluation platform, and benchmark agent performance for goal recognition followed by collaborative planning. The work as described compares favourably to similar work in evaluation platforms and benchmarks referenced in the related work section and appendix. The differences are made clear, though the use of some featured distinctions are not demonstrated in the paper (e.g. visual observations are possible but not used in the benchmarking).
		3. Experimental Rigour
			a. This work is not primarily about demonstrating the benefits of a particular approach over others in a particular application. It demonstrates benchmarks for agent performance in a newly introduce problem setting. From that perspective, the paper has strong experimental rigour. The experimental design is appropriate, comparing multiple baselines and oracles with several sets of experimental variants from both automated planning and reinforcement learning communities.
			b. The comparison of experimental variants is conducted with both a computationally-controlled agent and a human-controlled avatar to evaluate collaboration performance in increasingly realistic settings.
			c. The claim that the computationally-controlled Alice agent is human-like is repeated throughout the paper. This is not justified in the actual text of the main paper, but is supported to a moderate degree (human-like in strategy/planning if not movement/behaviour) through experiments with human subjects that are described in the appendix.
			d. Effort paid to ensure diverse avatars in experimentation.
		4. Reproducibility
			a. The work is almost entirely reproducible, with details of all agent architectures used for experiments provided with hyperparameters and architecture design. The authors describe that the environment will be released as open-source, which will then make the article wholly reproducible. This reviewer appreciated the level of additional detail provided in the appendix to improve this area of evaluation.
	3. Weaknesses
		1. This paper uses the term social intelligence to motivate the context for this challenge task. Social intelligence is a much broader term than what is actually being evaluated here and would require evaluating capabilities beyond goal recognition and coordinated planning/task execution. It is suggested to replace this claim with "goal recognition and collaborative planning".
		2. From the motivation provided, i.e. evaluating social perception and collaboration, why is the task specifically about watching then helping and not both together or just helping or just watching or some combination of these activities fluidly occurring throughout the interaction?
		3. Further, the work itself does not explicitly motivate why this, specific challenge task for goal recognition followed by collaborative planning is necessary for moving the state of the art in human-AI collaboration forward. However, it is a small leap to see the impact of this platform/task in evaluating applications like service robotics, social robotics, collaborative human-agent task performance, video games, etc. This reviewer can understand the impact of the work, but it would be clearer to explicitly discuss this.
		4. It would be clearer to specify that this task is limited to situations where there is explicitly only one goal throughout the entire demonstration + execution episode. This is important since it precludes using this challenge task for research into agents that need to use goal recognition after the initial demonstration, potentially continuously over the course of execution. This second kind of continuous goal monitoring is more similar to real-world applications of watching and helping or assistive agents or social robotics, since the human collaborator can (and often will) change their mind.
		5. Similarly, it should be noted that there is an explicit limitation of this challenge task and the evaluation metrics to scenarios where the entire success or failure of the approach is purely based on the final team accomplishment. This is similar to situations like team sports, where all that matters is the final game score. Many real-world scenarios for human-AI collaboration, differ by also requiring individual collaborators to do well or for the primary human user to do better with collaboration (than without). For example, in a computer game where Bob represents a team-mate to Alice who is a human player, Bob can choose to steam-roll Alice and win the game by itself. However, this leads to lower subjective user experience for the human team-mate. In this case, the score might be greater than what Alice could accomplish on their own and the game might be won faster than Alice could on their own, but the experience would be different based on whether they are truly collaborating or one is over-shadowing the other.
		6. A final assumption, is that there is no difference in expertise between Alice and Bob. The human is expected to be able to competently finish the task and for Bob to linearly cut down the time taken to perform this task. There are many real-world tasks in human-AI collaboration where this assumption does not hold and there could be non-linear interactions between success-rate and speed-up due to different levels of expertise between Alice and Bob. 
		7. The fixed Alice agent is called human-like through out the article and this was not properly justified anywhere in the main text of the paper. However, the appendix actually describes results that compare the performance of the computationally-controlled and human-controlled variants of Alice to human observers. This potentially justifies this weakness. For clarity, it would be valuable to refer to the presence of this validation experiment in the main paper.
		8. Why aren't there benchmark results (more than one) for the goal recognition component similar to the planning task experimentation? If both parts of the task are important, it would be valuable to provide additional experiments to show comparisons between goal recognition approaches as well, even if that is in the appendix for space reasons.
		9. There could be more analysis of the benchmark agent performance, 1) Why does the purely random agent work relatively well across tasks? 2) Why doesn't HRL work better? Is this due to less hyperparameter tuning compared to other approaches or due to some intrinsic aspect of the task itself? 3) Perhaps I missed this, but why not try a flat model-based or model-free RL without a hierarchy?
		10. There are several comments about other environments in the related work section and appendix being toy environments. However, the tasks in the environment demonstrated in this paper only use a small set of predicates as goals. Similarly, it CAN generate visual observations but that isn't used by any of the baselines in the paper. Several comparisons to related virtual environments are made in appendix, but some of the features aren't used here either (humanoid agent - this challenge task works equally well with non-humanoid avatars/behaviours and realism - visual realism is present but it isn't clear if behavioural or physical realism is present due to seeming use of animations instead of physical simulation).
		11. None of the tasks described allow the use of communication between agents or evaluate that. Other multi-agent environments like Particle Environments (below) allow for that. Communication is a natural part of collaboration and should have been mentioned if only to distinguish future work or work out of current scope.
			a. @article{mordatch2017emergence,  title={Emergence of Grounded Compositional Language in Multi-Agent Populations},  author={Mordatch, Igor and Abbeel, Pieter},  journal={arXiv preprint arXiv:1703.04908},  year={2017}}
		12. "planning and learning based baselines", "and multiple planning and deep reinforcement learning (DRL) baselines", etc. - There is potential for confusion with the use of terms "planning" and "learning" methods to do what both fields (automated/symbolic planning and reinforcement learning) would potentially consider as planning tasks. It would be clearer to indicate this distinction in terminology.
		13. The human-likeness evaluation experiment asked subjects to evaluate performance one agent video at a time. A more rigorous evaluation might compare two agents side by side and ask the human to guess the human performance. This could also be in addition to the current evaluation. The current evaluation is a ceiling on performance while the comparative evaluation is a potential floor.
	4. Recommendation: 
		1. I recommend accepting this paper, which was clear, novel, empirically strong, and supremely reproducible. The strengths conveyed above outweighed the weaknesses.
	5. Minor Comments/Suggestions: 
		1. Some minor typos in the manuscript:
			a. Using the inferred goals, both HP and Hybrid can offer effective. - page 6
			b. IN(pundcake, fridge) - appendix table 2
			c. This closeness perdition - appendix page 19
review_title: Clear, novel, empirically rigorous, and highly reproducible work. Recommend accept.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Quality
- I enjoyed reading the paper. In general I think it was of good quality, but mentioned in my summary there was a lot going on from lots of perspectives. T
- This not only includes the scope but the experiments (agent and humans), large number of complex agent technologies, many types of agents. 
- Obviously the page count would not suffice and hence there is what I would consider significant core content in the appendices.
- While the conclusion is short and to the point, the paper would have benefited from a critical evaluation/discussion of the approach taken (limitations etc). 
- In the conclusion the authors state that the experiment demonstrated the ability to evaluate aspects of social intelligence at scale. What do you mean by scale? What if I had 10, 100, 1000 agents?

### Clarity
- From a readability point of view the paper is largely OK. 
- The abstract and introduction which motivate the problem are very good.
- I would have liked to have seen further 
- Lots of moving parts/components/subsystems made it sometimes difficult to get your head around the whole system being described. 
- It wasn't clear to me as to the significance of the domain chosen. What is the long term goal with this work? Is it social intelligence for human robotics?
- I think the authors could do a better job of explaining/interpreting results.

### Originality
- There were certainly many original aspects to the work. But I believe it was more combining existing approaches and techniques to solve the problem rather than coming up with new learning techniques from scratch.

### Significance
- The work is significant to researchers who are interested in modelling social intelligence, robotics and multi-agent systems. 
- So perhaps with that in mind this paper may be more suited to a conference like AAMAS (Autonomous Agents and Multi-Agent Systems) rather than ICLR which is more machine learning focused. I'm not making a judgement here, just something to consider with this work going forward.

### Other questions and comments for the authors:

- Does Bob learn the sequence of actions that Alice take to achieve the task or just the end state of Alice's task? If it is the latter, does Bob work as well if he doesn't participate in the watch stage and is just told what the goal is in the help stage?

- If Bob is helping Alice, does he automatically assume that Alice is trying to achieve the task that Bob watched Alice previously doing? Or does Bob have to observe Alice a second time and then infer that she is trying to achieve a goal which Bob knows how to help with because he previously observed her doing it in the Watch phase?

- In the watch phase, does Bob ever get confused? For example, assuming that Alice has started task A, when in fact she started task B and Bob starts helping with task A?

- You don't mention agent communication which is often an important part of modelling agent social interaction. What if Alice communicated to Bob her goal?

- I really like Figure 6a.This is an excellent chart except for the y-axis removal of sections between 0.6 and 1.6 and 1.8 and 2.0. I think this is misleading. Perhaps a log scale would be appropriate. 

- That aside I think this type of chart tells the reader more about the performance of the different agents compared to 6b which plots the reward. With 6b I need some additional interpretation which is a bit lacking in the text.

- What are the error bars in 6b? Standard deviation, standard error, confidence intervals?

- While the addition of human experiments is good and comprehensive, perhaps that could be the topic of another paper and in its place have section on discussing the pros/cons of the results.
review_title: Interesting paper with a lot of good ideas, somewhat let down by its ambitious scope and large amount of content. A smaller scope and tighter focus would improve this paper significantly.

Meta-review: Summary:
This paper provides an interesting and unique challenge problem on human-AI collaboration, with sample baselines. I think this is an extremely important topic and the community should embrace such challenge problems.

Discussion:
Reviewers agreed this paper should be accepted, particularly after seeing that ICLR has accepted such challenge papers in the past.

Recommendation:
I'd really like to see this get a spotlight as it would be great to highlight this innovative challenge to the community. 
Decision: Accept (Spotlight)