Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method for learning two models for human-robot collaborative tasks: one that reproduces human-like behaviors, and one that optimizes the robot's behavior for efficient collaboration. In learning the former, the human's strategy is represented as a latent representation and used to produce trajectories that maximize informativeness. The results indicate that the proposed method produces trajectories that reflect a variety of strategies demonstrated by a human (even when those strategies are unbalanced in the dataset). Furthermore, the proposed method resulted in more successful handovers during a real robot task.

strengths_and_weaknesses: Strengths:
* The paper is well written and easy to follow.
* The motivation is convincing, and the related works section clearly explains why the proposed method provides a benefit over current SOTA.
* The evaluation is very thorough as a result of comparing the proposed method to four different baseline approaches.
* The results themselves are compelling, and test each part of the system (latent space learning, policy learning, and execution).

Weaknesses:
* More discussion on the effect of the hyperparameters in eq 4 would be useful.
* During training, what defines the ground truth for z in eq 2?
* Axis labels in Figures 3 and 4 are way too small
summary_of_recommendation: Overall, this paper is clear and presents thorough, compelling results. 
issues: * More discussion on the effect of the hyper parameters in eq 4 would be useful.
* During training, what defines the ground truth for z in eq 2?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a technique to generate robot behavior that can collaborate successfully with humans by learning from human-human interaction. They model this as a two-agent imitation learning problem where behaviors for both the human and the robot are learned concurrently. Their main contribution is to extend the MA-GAIL [26] framework by introducing a latent representation of the human's strategy, also learned from the data, which enables the robot to adapt to the diverse set of behaviors exhibited by the human during training. They test this model in three collaborative tasks and show that it outperforms several baselines.
strengths_and_weaknesses: Strengths
 - The paper is well-written.
 - The technique is motivated by human behavior and grounded in past work.
 - Tested in multiple domains.
 - Experiments included their model interacting with humans which is rare.

Weaknesses: 
- The experiment descriptions are missing some details like the user recruitment procedure including for the training data. Also, were the different strategies that the humans used pre-decided? Was any form of communication permitted between participants?
- The superior performance of the baselines in table 2 is explained in the paper with overfitting. Can you provide some more detail why that should be the case since Interpolation is not necessarily playing trajectories from the training data? If I understand correctly,  the interpolated trajectories should also contain multi-modal behavior that MA-GAIL should find difficult to adapt to. Also, would be helpful if it is explained how the strategy sampled here was learned.  

That makes sense, seems to indicate that the proposed method underperformed the baselines. It would be useful if the authors presented some reasons for this.  
summary_of_recommendation: The paper is well-written, they describe the problem clearly. Their solution seems logical given the explanation of past work. Their experiments provide a thorough test of their method. 
issues: - More details relating to the experiment design as well as the participants used for the study as described in the weaknesses.

- Space permitting, it would be good if authors can mention more detail about their experimental setup for reproducibility

- Is there a plan to release the software and data? 

- Analyze and explain the results from table 2.

- The description of Fig 4b is difficult to understand.

- In 4.3, which learned strategy space is being used for Interpolation?

- line 24, not 30% in the HR-S

Typos/ minor issues:
- (line 116) an -> a
- (line 231) reply -> replay, if -> of 
- (line 244) table 2-> 1
- (line 249) table 2-> 1
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a learning approach to train control policies that allow robotic agents to assist human partners in collaborative tasks. 
The main insight is to incorporate the diverse range of behaviors/strategies that humans may follow during collaborative tasks into robot control policies for a more effective joint task execution.
Similar to prior work, the proposed approach first trains a collaborative policy using human-human demonstrations that results in a shared policy network that outputs actions for both agents (human and robot). 
This policy learning is then extended, again similar to prior work, by integrating a latent representation of human strategies, i.e., by incentivizing the human policy to condition on the strategy to replicate demonstrated human actions. 
To improve capturing diverse human behaviors, this latent representation and the human policy are also co-optimized as an autoencoder using demonstrations. 
The results highlight comparable or better performance in terms of strategy coverage and responsiveness on the three tasks being evaluated.
strengths_and_weaknesses: The main strength of this work is its learning architecture that effectively integrates human strategies. 
This integration allows modeling human behaviors, which in turn improves the robustness of trained robot policies that rely on them.

One critical weakness of this work is that the diversity of human strategies is not analyzed and discussed within the main text. It's not clear for continuous problems (such as handover or sequential task) how to identify human strategies and then analyze them. The appendix includes further info but I believe as a core aspect of this work, the main manuscript should provide a better explanation on those details. 

Second weakness is the experimental setup used for HRI scenarios. The user interface is based on a teleoperation interface (using mobile phones), which I assume would not provide representative human movements, and thus spoils their strategy selection and execution preferences. 
A more representative teleoperation interface would make the experiments and results more convincing.
summary_of_recommendation: The paper proposes a novel extension of prior work which improves human-robot collaboration performance for three different tasks. This is achieved by incentivizing the encoding of diverse human strategies while learning robot policies. The experimental setup in terms of how the data was acquired remains to be a critical issue in order to justify their representativeness of real human movement and strategy selection behaviors.

issues: Please also see above (Strengths & Weaknesses).\
Some further improvement points:\
There is a strong emphasis on capturing diverse human behaviors, but even though this is supported by the interpolation evaluation on the 2D-Fetch-Quest task, looking at the other tasks' results (e.g., Fig4), I couldn't identify a significant (qualitative) difference between the proposed Co-GAIL approach and DYIAN. 
Also, I'd recommend to not only focus on the positive results of the proposed approach but also where it underperforms, e.g., success rate of interpolation evaluations for the HRI experiments (Table 2).
I'd suggest using a better/fairer phrasing and/or quantitative comparison, especially to justify diversity claims.

Real human user interacting with robot policy: Even though it's a proof-of-concept, it's not clear what the protocol for this experiment was:
- who were the subjects? how much they had experience with robotics and/or this particular setup? are they PhD students, friends, random people? etc.
As a qualitative analysis, it might be fine to report on, but a quantitative analysis would be unreliable without a proper user study protocol.

small issues:
- typo: caption of Fig.1: 'Pick and object ...', line 212: manipulate -> manipulating, line 231: reply -> replay, line 271: '... instead (of) covering ...'
- misreference: line 152: Figure 2b -> 2a, line 160: Figure 2c -> 2b, line 245: Table 2 -> 1, line 277: Table 1 -> 2 and Table 2 -> 1
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors made a good faith attempt to improve the quality of their evidence, and to improve the means by which they analyzed the evidence. As a result of this, I am more confident that Co-GAIL is better than their baselines, although I am still somewhat confused by the evidence. In particular, a T3 Dunnett post-hoc test found that Co-GAIL was only significantly superior to MA-InfoGAIL in one of three test environments, and was not significantly superior in another two. However, a GLM predicting success rate against the model, task, and user found that the weight on Co-GAIL was significantly higher than the weights on the other method terms. The only differences between these comparisons is that the GLM aggregates across tasks, and has a per-user term (which seems justified with n=20 trials per user). I am not an expert statistician, but these results taken together seem to say that Co-GAIL is a lot better on some tasks, and not obviously worse on the rest. The authors also gave a strong explanation for why my interpretation of Table 2 was incorrect, and the explanation in their paper was likely to be correct. This is enough for me to shift my judgement from weak reject to weak accept. The original review follows unchanged.

===

Doing imitation learning in an environment that requires interaction with a human is not handled well by most imitation learning methods, as they tend to assume any human involvement is in the form of a fixed human policy that does not react to the robot. This is either false (if the robot's state is not well captured in the state) or difficult to model and collect data on (when it is).

In order to address this issue, there exist a collection of methods that attempt to estimate both a human and a robot policy from a collection of demonstrations where humans act as both the human in the environment and the robot helping them accomplish a task. The most successful of these methods are MA-GAIL, which adapts the GAIL framework to a multi-agent setting, MA-InfoGAIL, a modification of the above that considers a latent strategy variable, and DYIAN, which tries to maximize a diversity criteria. MA-InfoGAIL and DYIAN have policies that are conditioned on a strategy latent variable that attempts to capture the fact that humans will act according to different strategies even in the same state.

The novel contribution of this paper is to introduce Co-GAIL, a modification of MA-InfoGAIL that includes an additional loss designed to prevent mode collapse in the estimation of the latent strategy variable. This new method (and several baselines) are evaluated on a 2D-gridworld and two 3D physics based object manipulation tasks. The authors find that, according to some evaluations using historical (fixed) human data and a small pilot study of humans reacting live to the robot's actions, co-GAIL outperforms all of the baselines overall. They also prevent evidence that mode collapse is a problem in prior methods.
strengths_and_weaknesses: Strengths:
The paper has a very strong introduction, and a very clear explanation of their method. I was not familiar with the MA-GAIL line of work, but was able to follow their paper.

The novel contribution is a natural solution to the problem at hand.

The paper provides evidence that the problem it is trying to fix actually exists, which is something many neglect to do.

The paper compares its method to a fairly comprehensive set of baselines.

Weaknesses:
The quality of evidence is low in many experiments. Only 3 random seeds were used in the simulated human experiments, and the reported variances are quite high. Only one replication was performed on the human data, which is understandable for a pilot in present circumstances, but it makes it hard to strongly believe your results.

If I understand things correctly, the interpolation experiments have a significant conceptual flaw. Considering a uniform prior over strategy variables seems very inappropriate when we expect the strategy encoder to eventually concentrate around a few modes.
summary_of_recommendation: I cannot recommend this paper until further replications are performed. The simulated experiments only use three replications, and the human trials only use four subjects each. The spread in data values is very often large enough to be concerning. The method itself is reasonable, and qualitatively CoGAIL does appear to match the human's trajectory distribution better, but I simply cannot be confident that the method actually outperforms DYIAN in all cases.
issues: My biggest issue is that three replications simply isn't enough to be confident that your method is actually superior to the baselines. In 2D-FetchQ the 95% confidence intervals (assuming a Gaussian, which is hard to do when n=3) of DYIAN and Co-GAIL overlap. This is not the case in the 3D environments, which is something, but the variances of your method in those environments is also quite high, especially for n=3. It's also not reassuring that your method did worse than the others in the interpolation setting. It is true that the other methods that learn mode collapsed policies have an easier task here, but it betrays a weakness in your policy learning step if you can't learn a robot policy that is near optimal under your own model of the human.

Ultimately both of the simulated human cases provide some evidence, but aren't conclusive. Replay is limited in that the human can't react to the robot's actions, which was a major part of the motivation for this paper. Interpolation does have a reactive human, but lacks external validity. The human pilot study is the best evidence available, but unfortunately there were only four subjects available. At least on the 2D environment, you can get subjects over the internet. It is hard to view four replications as strong evidence. Admittedly the scatterplots for 2D-FetchQ and HR-S look fairly separate, but there is definitely room for DYIAN to come out on top in the handover task with more data. I would also recommend you produce those scatterplots, and also report mean scores across different subjects, at least in the appendix.

I also have a conceptual issue with how the interpolation experiments were done in terms of the reported success rate. When training, you learn a psi function to map human actions to a latent strategy. An ideal psi function captures multiple human modes of behavior, even at identical states. A psi that is spread out over all the different values in the strategy space is not very informative for the policy. So we expect that the empirical values of z will concentrate around a few different modes. If the goal of the reported success rate is to give a self-consistency check on the co-policies, taking a uniform prior over strategies seems inappropriate. What I would do in your position would be to record the z values along the fixed human trajectories you have, visually (or automatically) identify a few mode points for each method, and then use those mode strategy values to simulate the human's policy. Using a uniform prior to identify the diversity of human trajectories and compare that to the prerecorded trajectories is reasonable.

As an extremely minor issue, some implementations of GAIL use log(D(.)) as the proxy reward, and some use -log(1-D(.)). For the sake of completeness it would be nice if you could mention this somewhere in the appendix.

There are several grammatical issues, notational inconsistencies, and inconsistencies between the appendix and the main paper. I expect all of these can be fixed quite simply, and I provide them mainly for the benefit of the authors.

62: "agents are considered" -> "agents are assumed"
84: "which is crucial especially" -> "which is especially crucial" or "which is crucial, especially"
112, 126: $\mathcal{M}$ is both the Markov game and the set of demonstrations, pick a different name for one or the other
131: "that the start state is $s_i \sim \rho_0(\cdot)$" -> The way this is written was confusing, and this is such a standard assumption that I don't think you need to mention it at all.
143: "$\mathbf{z}_t \sim \psi(\mathbf{z}_t | ...)$" -> ""$\mathbf{z}_t \sim \psi( \cdot | ...)$"
155: "when producing action" -> "when producing actions"
191: "Equation1" -> "Equation 1" You're missing a space.
192: "infoGAIL" -> "InfoGAIL" Capitalization.
212: "collaboratively manipulate" -> "collaboratively manipulating" or rework the sentence more broadly.
245, 249, 277: You appear to have flipped table 1 and table 2 in several places.
298: "MA-InfoGAIL overfits to" -> "MA-InfoGAIL overfit to" tense issues.
315: "to the real robot" -> "to a real robot"

Table 1, 2, appendix: The main paper says you report variances, but the appendix says you report standard deviation. As a notational issue, X +- Y is only for something which is an interval, like a confidence interval. If you want to report a mean and variance or a mean and standard deviation, the notation is X (Y). Additionally, if you're going to bold specific items that are the best, you need to bold all items that COULD be the best, meaning that you bold the item with the best mean, and all other rows whose confidence intervals overlap with the confidence interval of the row with the best mean.
review_title: null

Meta-review: The paper received 4 reviews which are on average leaning slightly towards an accept. The paper is well written and well motivated, the evaluation is thorough. The paper heavily builds on prior work, result in rather low impact/contribution scores. Some details need to be clarified. The most important concerns circle around the experiments. Some of the results (in particular why the proposed method did worse in one setting) need a deeper analysis. Related to that there are also some doubts about the validity of the setup and results for some of the experiments.

## Update after author response and AC & reviewer discussion
The reviewers appreciated the detailed responses and additional experiments and analysis. They are now all in favor of accepting the paper.
Decision: Accept (Poster)