Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
Authors: Chen Wang
Claudia Pérez-D'Arpino
Danfei Xu
Li Fei-Fei
Karen Liu
Silvio Savarese
Venue: CoRL2021 Poster (2021)
Keywords: Learning for Human-Robot Collaboration
Imitation Learning

Abstract: We present a method for learning human-robot collaboration policy from human-human collaboration demonstrations. An effective robot assistant must learn to handle diverse human behaviors shown in the demonstrations and be robust when the humans adjust their strategies during online task execution. Our method co-optimizes a human policy and a robot policy in an interactive learning process: the human policy learns to generate diverse and plausible collaborative behaviors from demonstrations while the robot policy learns to assist by estimating the unobserved latent strategy of its human collaborator. Across a 2D strategy game, a human-robot handover task, and a multi-step collaborative manipulation task, our method outperforms the alternatives in both simulated evaluations and when executing the tasks with a real human operator in-the-loop.

Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
Chen Wang, Claudia Pérez-D's Pérez-D'Arpino, Danfei Xu, Li Fei-Fei, C. Karen Liu, Silvio Savarese Stanford University
Abstract: We present method for learning human-robor collaboration policy from numan-human collaboration demonstrations. An effective robot assistant must learn to handle diverse human behaviors shown in the demonstrations and be robust when the humans adjust their strategies during online task execution. Our method co-optimizes human policy and robot policy in interactive learning process: the human policy learns to generate diverse and plausible collaborative behaviors from demonstrations while the robot policy learns to assist by estimat- ing the unobserved latent strategy of its human collaborator. Across 2D strategy game, numan-robol handover task, and multi-step collaborative manipulation task, our method outperforms the alternatives in both simulated evaluations and when executing the tasks with real human operator in-the- Supplementary materials and videos at https: : //s: sistes... /ooggeeooogglle......./oo.o. com/view http//ogaallhhheaeeee
Keywords: Learning for Human- Collaboration, Imitation Learning
1 Introduction
Advancing technologies for human-robot collahoration (HRC) has the potential to unlock applica- tions with large societal impact in manufacturing, hospitals, and home settings [L, 2]. However, robots that are designed to work around humans are still limited in versatility when performing col- laborative tasks. Recent advances in robot learning focus on robots that work in isolation [3, 4, 5, 6] or alongside other agents that do not exhibit human traits [7, 8, 9, 10]. Learning to collaborate with humans presents unique challenges to existing robot learning methods: instead of optimizing only for efficient task completion, the robot assistant must act in coordination and adapt to the diversity of strategies and movements of their human counterparts. This work aims to develop robot assistants that adapt to diverse human strategies and movements in collaborative manipulation tasks.
While past works in multi-agent reinforcement learning show that agents could learn to develop collaborative behaviors [7, 9, 11, 8], sand-engineering reward functions for collaborative goals is non-trivial [10] and the strategies learned from robot-robot collaboration may not align with that of the human users. On the other hand, learning from demonstrations human-humas collabora- tion could allow the robot to develop plausible collaboration strategies, as well as avoiding reward engineering. However, most prior works in this setting [12. 13, 14] treat the human as a stationary part of environment during the learning process and fail to model the human's 's reactions to the evolving robot policy. Modeling such reciprocal behavior adjustment during training is essential to develop robust assistive policies, the human collaborator are likely depart from the exact strate- gies they previously demonstrated during online task execution. In addition, human behaviors tend to be highly diverse both in high-level intents and low-level movements. Learning robot policy that operates effectively with diverse human behaviors is also crucial for HRC tasks.
In this work, we propose method to learn policies that generate diverse human and robot collab- oration behaviors from human-human demonstrations with an interactive co-optimization process. Instead of treating humans part of the environment, the human policy is an active participant in the training process and co-evolves with the robot policy as the learning progresses. This allows the robot policy to adapt to the movements of the human teammate while assisting the human to complete the task. To overcome the challenge of diverse strategies presented in the human demon- strations, we introduce latent representation of human collaboration strategies. On one hand, latent strategy learning objective allows the policy to discover and disentangle factors of variations in the demonstrations, ensuring that the robot assistant is proficient all collaborative behaviors
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

(a) 2D-Fetch- FD-Fecch-Quest
(b) IIR-Handover
(c) HagSSgaaanp
Human-robot collaboration tasks: (a) ID-Fetch-Quest. human plays with a learned agent to unlock and retrieve objects with strategic dependencies; (b) HR- HR--Handver: human hands over an object to a robot; (c) Manie: Pick an object and open a drawer to drop the object on it. Human and robot might lead different steps of the task.
Figure I:
shown by the expert even if certain strategies are rare. On the other hand, the latent strategy code serves as a communication protocol that enables the robots to anticipate future behaviors of their human collaborators and become more effective during comman-in-the-lon task execution.
We test the method in three challenging collaborative domains illustrated in Figure 1: 2D game that requires coordination among two agents (2D-Fetch-Quest) human-to-robes handover (HR- Handover), and sequential collaborative manipulation with handovers (HR-SeqManip). These domains range difficulty from 2D grid-like game to high-dimensional manipulation tasks with 7-DOF arms and task execution within a physics-based simulation environment using Bullet [15] and iGibson [16]. We conclude by presenting brief demonstration of the method in interaction with real human collaborators, showing for the first time a learning method that succeeds at adapting to real users in these challenging tasks.
Related work
Multi-agent Reinforcement Learning: A large body of prior works utilize multi-agent RL (MARL) to learn collaborative strategies for tackling multiplayer games [17, 11], multi-robot ma- nipulation [18, 19], traffic control [20], and social dilemmas [7]. However, in the HRI setting, the robot needs learn to adapt to a wide range of human collaborative behaviors, for which it non- trivial to design effective reward functions. Translating the benefits of MARL to HRI in real-world manipulation tasks is limited because agents are assumed to have similar policies or don't 't exhibit human traits [21]. Tampuu et al. [10] use multi-agent reinforcement learning to model both agents and demonstrate how collaborative behavior can arise with the appropriate choice of reward struc- ture. In [8] the robot learns to influence the collaborator's 's action by optimizing long-term reward over repeated interactions and estimating their policy per interaction episode through a learned latent space. Hand-engineered collaborative rewards cause the overfitting of the Iearned robot policy [9], limiting the robot's 's adaption to diverse human behaviors.
Inverse reinforcement learning (IRL), on the other hand, shows promising results in learning collaborative behaviors learning from expert demonstra- tions [22, 23). Notably, recent progress in generative adversarial imitation learning (GAIL) [24, 25] further shows effectiveness cooperative settings [26]. However, as noted prior works [25], the learned collaborative policy tends to overfit strategies that are easy to execute and neglects the more challenging cases. This becomes problematic when the robot encounters a real human collabora- tor, who may prefer strategies outside of the distribution that the policy focuses on. Our method addresses this problem by explicitly disentangling the behaviors shown in the demonstrations and encourage the robot policy to learn from diverse collaborative strategies.
Multi-agent Inverse Reinforcement Learning:
Modeling diverse behavior: Learning diverse behaviors has been a long-standing challenge in re- inforcement learning research [27, 28, 29]. Recent works show that by maximizing the entropy of states and minimizing the conditional entropy of states with respect to the actions, policies can learn diverse skills [30, 31] even without reward signals [32]. Rather than training policies to behave as diverse as possible, our method optimizes for covering the behaviors shown in demonstrations. This allows the robot to focus collaborative behaviors that plausible during human-robot explo-

---

Eval with real human
Humen al dagy
roal human speratir
L
luman-Fummn collab data
2
a
earple
H
Tox
M
cooodiyy
colub
M
reconstruction laxs
Human bohavior
La
a
a
colimaaioo
(b)
(0)
Figure 2: Architecture Overview: (a) The co-policy Teo observes the states and strategy code 2 and outputs actions for both human and robot (a a a²). Teo is trained with the GAIL algorithm using dataset of human-human collaboration M. The strategy recognition network &b is jointly trained to estimate the latent strategy from human behavior, using the reconstruction loss Lg to recover 2. (b) The & and Teo networks are also trained as an an auto-encoder using the behavior reconstruction loss La to learn the inverse mapping from the demonstration to the strategy space and (c) System for testing human-robot collaboration. The model estimates the human strategy 2 from its motions and outputs the robot's action a².
encourage the coverage of diverse strategies.
ration, which is especially crucial in large environments where behavioral diversity is unbounded. Clegg et al. [33] introduces a method to learn diverse and plausible assistive dressing behavior by hand-coding human motor capabilities in the model. In contrast, our method does not assume ex- plicit domain knowledge and learns collaborative policies from demonstration data. Wang et al. [34] proposes to learn diverse behaviors through imitation but requires access to demonstrations during online execution, whereas our method only assumes access to demonstration at training time.
Human behavior modeling in human-robot collaboration: Prior works approach the problem of modeling human behaviors in interactive learning settings in multiple ways. Most works introduce intermediate representation of the human behaviors such as user type classification [12, 35), future action prediction [36]. hidden goal estimation [37], information communication [38] and human motion prediction [39]. These methods show promising results in HRI applications, including assis- tive robot for dressing [33, 40), feeding [41, 42, 43], assembling [14] etc. However, most existing works requires tasks-specific knowledge such as user types [12. 35]. Our method learns latent space of collaboration strategy from demonstrations without explicit task-level supervision.
3 Method
Our method is designed to overcome two unique challenges imposed by human-robot interaction: (1) human behaviors difficult model because they follow their own policy while reacting the robot's 's actions, and (2) human behaviors are highly diverse both in terms of the high-level strategies and the low-level actions. Below we introduce the general HRC problem setup (3.1), an interactive
imitation learning formulation to the problem (3.2), and how to uncover diverse collaboration behav- iors by learning a latent strategy space (3.3). Finally, we introduce the learning algorithm Co-GAIL. (Algorithm 1) and the numan-in-the-looe task execution setup (3.4).
3.1 Learning human-robot co-policy
Although human and robot have different state space and action space, their underlying motivation of completing the collaboration task is shared. Instead of treating the human part of the en- vironment with a fixed policy, allowing the human and the robot to learn and explore concurrently has the potential to achieve more robust robot policy. We consider a human-robot collaboration task as an extension of Markov Decision Process (MDP) called Markov games [44] for two agents (N 2), MG AS.A" R. T.A. R, 7, Pu), with state space S, human action space A robot ac- tion space A R transition distribution 11111/,, a." a. reward function R(s,, a R $441), discount factor 7 [0, 1), and initial state distribution Pp-
The solution to this MDP co-policy Teo with a shared policy network that outputs action for both agents. At every step, the co-policy observes the current state 8t, chooses set of actions
3

---

(a". a.a)) ~ [ a a,)ss,,, and observes the next state 8141 T(- a a and reward r: R(s,, at a." 81+1) shared by both agents. The goal is learn a co-policy that maximizes the shared expected return -| 7' R(st, of at 8t+1]]. With slight abuse of notation, we denote the part of the policy that outputs human actions as no and that for robot actions as T
3.2 Learning strategy-conditioned co-policy from demonstration
While we could attempt to solve for the co-policy using existing RI. methods, designing an effective and comprehensive reward function that properly evaluates the quality of human-robot collaboration non-trivial. On the other hand, imitation learning has the advantage of learning the policy directly from demonstrations, bypassing the need to explicitly design a reward function. As such, we record a set of human-human demonstrations M of two users performing a cooperative task and formulate the co-policy learning problem based on MA-GAIL [26], multi-agent extension of the Generative Adversarial Imitation Learning (GAIL) [24] framework.
Let TE and TE, denote two expert policies to which we only have access to demonstrations, M, a dataset of N collaborative task demonstrations M = { J Each demonstration is trajectory
Fi Fi Fiz "i,a _01 .... sT.). The goal of MA-GAIL [26] is train the co-policy Too to imitate expert policy (TE.. TE,) by minimizing distance between the generated state-action distribution c( and the expert's distribution : measured by Jensen-Shannon diver-
T, (s): ao -00
Fe'
gence. We optimize the following objective:
min max Be- Tz flog D(x)] Ey-o(r_.) [og(( D(y)))
Teo D
where D is discriminative classifier which tries to distinguish state-action pairs from the trajecto- ries generated by Too and (TE, TFF.)
However, directly applying MA-GAIL and equation 5 results in a poor co-policy due to the di- verse behaviors in human-human demonstration. Given the same environment state s, human might perform different types of action a" based on their choice of strategy, which is hidden from the observable state. To disentangle the diverse human behaviors associated with the same state, we in- clude a latent representation, x, called strategy, to the input of the co-policy, aao(a" " a s z). We assume that the strategy can be inferred from the history of observations and the the human action and define recognition model
~ (6(-(s:-Kat K a="i). where K is the length of the history. To simplify the notation, we denote the history at time step L as h. (st- Kit, af_1). The recognition model va is trained jointly with the co-policy (Figure 2a). Our framework allows the strategy to evolve as the task progresses in interaction episode. As such, the robot must constantly update its estimate of the human's 's latent strategy using (-))
3.3 Uncovering diverse human behaviors
Introducing the latent strategy alone does not lead to successful co-policy learning. One challenge is that the human policy can simply ignore the latent strategy z and always opt to generate the most common behaviors. To this end, we use high mutual information between the latent strategy and the human action (Figure 2a). Given a prior distribution of the strategy space p(z), we minimize the objective:
ceammation-thhorrte regularization [45, 25] to enforce
L =
211. -
(2)
where h is the length-K K trajectory generated from the acting policy nhd given sampled code z. While L3 indeed incentivizes the human policy to utilize the latent strategy when producing actions, it does not address another issue due to the imbalance distribution of training data, which results in e-(-) ignoring less frequently seen a" " and mapping them to the same strategies that reflects the majority of a !.. Whereas in practice, the human collaborator may interact with the robot with any strategy included in the demonstration. To address this issuc, we introduce another objective that measures the reconstruction error of human actions in the training dataset (Figure 2b):
La Eh,..... papm II (s:, (hh)) -a.1l
(3)
In each training iteration, batch of expert trajectory snippet h will be sampled from the dataset M. The recognition model v-)) first estimates the hidden strategy è from h. The human policy

---

(s, 2) is used as decoder to reconstruct the expert action from the predicted strategy 2. This loss encourages (b()) encode different a" different order to minimize the reconstruction envor, enforcing the inverse mapping from the human behavior space to the strategy space. Putting together, the final objective for learning diverse collaboration behaviors becomes:
min max Be--p [log [xx)] By--(==.)) [log(1 D X1 AzLa,
(4)
where A1, A2 are the hyperparameters for the intention and expert behavior reconstruction regular- ization term. The overview of the proposed algorithm is shown in Algorithm 1.
Algorithm 1: Co-GAlL. Learning diverse collaboration behavior
Data: M < given human- -umman collaboration dataset
Input: Te -- initialize co-policy, f- initialize strategy recognition network, D +- initialize the discriminator, B <- 0 initialize the replay buffer
while not done do
for trujectory i=l,.., ..., do
2, ~ p(z) sample strategy code from prior
B B-BUT {(ss a a, 2;))T= append experiences from Teo 2,) to the buffer
for update step = I,.., do bMM 8 + sample batch of trajectories from M and B
T.z calculate strategy reconstruction loss with 1 and Teo (Eq. 2) with data from 8
La + caleulate behavior reconstruction loss with & and Teo (Eq. 3) with data from b- update & and т. with the gradient of Y Ls + Ly
update discriminator D according to Eq. 4 using b and 1
10
11
update co-policy Too with PPO[46]
3.4 Human-in-the-loop task execution
During test time, we discard the human policy and only use the robot policy to control the robot who interacts with a real human user (Figure 2c). Different from prior work [25] where the latent code reconstruction network only used during the training process for mutual information max- imization, our learned recognition model v/(-) is an important component for the robot policy to recognize human collaborators' strategy and anticipate their future behaviors each time step, (--) takes history of states and human's 's previous action a H as input and outputs a strategy estimation 2. The robot policy will then generate the robot action, It T (s, ( 2).
real-time. At
Experiment Setup
The major advantage of Co-GAIL is its capability of uncovering diverse collaboration behavior from the given demonstrations. In the experiment section, aim compare Co-GAIL with state-of- the-art methods for learning collaboration skills. We focus on whether each method can successfully complete the task and successfully learn the diverse collaboration strategies. We also invite four real users to interact with the trained robot policy and evaluate the performance for each method.
4.1 Baselines
We start by comparing with imitation learning designed for single agent (i.e., treat human as part of the environment) and build towards the proposed method progressively.
BC-single: Behavior-cloning algorithm that only learns robot policy. During training, human behaviors are regarded as part of the environment and played back in an open-loop fashion.
BC-GAIL: Train the human and robot policy in isolation. First learn human policy with BC. Then use GAIL algorithm[244 to learn the robot policy by interacting with the learned human policy.
MA-GAIL: Implementation of MA-GAIL [26]. Both human and robot policies are learned with a shared GAIL. objective (Equation 5) during the training process.
MA-InfoGATI.: Add the latent strategy code learning objective based on InfoGAIl[[25) (Equation 2) as an additional learning objective to MA-GAIL.
5

---

MA-GAIL
MA-InÍGGAIL
DIAYN
Co-GAIL
Strategy
17%
Strategy
17%
Stratlegy
33%
Strategy
38%
REROKT sphindic
waschc cpiooka
CAANTOU spiied
Stralogy
33%
1000000 1000030
200 1000
Stratogy 33%
Strategy 17%
Strategy 17%
Figure 3: Interpolation evaluation on 2D-Fetch-Quest for model trained with two differently unbalanced dataset. Plots show the proportion of strategies generated by the model. Co-GAIL outnumbers the recovery of the two less represented strategies in all cases.
DIAYN: Implementation of [32] in a collaboration setting. Use a diversity reward to encourage the human policy to be diverse as possible.
Co-GAIL: The complete implementation of our proposed method. Our method differs from MA- InfoGAIL in the inclusion of behavior reconstruction loss (Equation 3) encourage the model to cover all strategies in the given demonstration.
4.2 Collaborative fasks
We evaluate the methods three continuous-acoo action space domains that involve collaboration, illus- trated in Figure 1. See Appendix for more details.
2D-Fetch-Quest (2D-FetchQ.): A collaborative 2D game inspired by the Fetch Quest minigame in Super Mario Party. Two agents must coordinate their strategy to fetch treasures from two locked rooms. Agents have complementary roles with temporal dependencies: one agent presses button to unlock the room while the other agent fetches the treasure, allowing four different strategies depending on the order of the actions and roles. The human-human demonstrations exhibit four distinctive strategies. Strategy 1 and 2 differs in which party takes the treasure first. Strategy 3 and are when the two parties decide switch to the other side of the space to complete the task.
HR-Handover (HR-H.): human-to-robot object handover task, in which the human holds an take place in any region between the two
object and hands it over to the robot. The handover agents with any approaching angle at any speed. The action space of each agent is the 6D position of the agent's hand relative the previous time step and a float value describing the gripper state. The human-human demonstrations exhibit diverse handover locations, varying in all three dimensions. We visualized the distribution of the position when handover happens (X-Y locations) in Figure 4a.
HRg (HR-S.): Place an object in a drawer by collaboratively manipulating both the object and the drawer. There two possible strategies solve this task: the robot assists by opening the drawer while the human drops the object, or the human hands over the object to the robot and opens the drawer, while the robot drops the object into the drawer. Other forms of diversity include the object handover locations and details motion such as ways to open the drawer. The action space is the same as in -In-deedoer..
These domains are implemented in an interactive simulation environment in which the human users control the agents via control interface. Demonstrations in 2D-Fetch-Ques are collected through a joystick interface. Handovev and MR-Se are implemented in iGibson environments [16] with Bullet physics engine [15]. Collaborative demonstrations are collected from two human users controlling the two agents using the Roboturk telcoperation interface [47, 48]. More details of the environment design could be found in Appendix.
4.3 Experimental setup
We assess the performance of म in terms of task completion (success rate) and coverage of diverse strategies. We employ three evaluation protocols to show different facets of the problem of learning diverse and plausible collaborative behaviors.

---

Uniformly sample the learned strategy space and simulate a robot- trajectory for each strategy using the learned co-policy. We evaluate the success rate and diversity of these trajectories for our method and the baselines. This evaluation provides an insight into the effect of each strategy on the co-policy, and the coverage of diversity presented in the training data.
Interpolation:
Replay: We subject the robot policy to interact with the open-loop replays of human trajectories previously unseen during training. Unlike interpolation evaluation, the replay evaluation tests of the policy can correctly respond to diverse and previously unseen collaborative behaviors. The setting allows scalable evaluation as we do not need a real human in the loop, but limited because the human policy does not react reciprocally to the robot.
Real- heal-human evaluation: Proof-of-concepe evaluation with task execution with real human user in-the-loop interacting with the robot's policy as illustrated in Figure 2c. We report the success rate for all domains and methods.
5 Results
Modeling human behavior improves
Table 1: Replay evaluation: Success rate (mean and 95% confidence interval over three seeds).
:h: The results show that modeling human behavior improves the robust- ness of the learned R when facing un- seen human behavior. Our proposed Co- GAIL outperforms BC-single in replay evaluation Table for more than 20% in all three tasks.
Replay HR-H.
2D-FetchQ.
HR-S.
BC-single
21.1 + 1.9
13.3 3.3 15.0 + 2.3
BC-GAIL
27.2 + 5.0
14.1
16.6
7.1
MA-GAIL[26]
30.0 3.4
23.7
21.0 1.7
MA-InfoGAIL/25)
40.0 1 1.7
27.8
L 4.4 26.6
8.8
Co-optimization outperforms learn- ing in isolation: Comparing the replay evaluation results (Table 1) for Co-GAIL and MA-GAIL indicates that co-optimizing the Tea has benefits over the sequential training of human and robot policy in BC-GAIL.
DIAYN[32]
44.4 8.4 22.7
1.5 18.3 7.1
Co-GAIL (ours)
53.3 + 4.4 43.9
+ 6.3 40.0 - 8.2
Co-GAIL learns more diverse strategies compared with the baselines: Unlike exist- ing methods, we designed Co-GAIL to obtain a R that can interact in the scenario that real human approaches the task with diverse strategies. Using the 2D-Fetch- Quest domain, which can be solved with 4 distinct strategies (see Appendix), we de- signed two datasets containing different distribution of demonstrated strategies follows: Dj = (17%, 17%, 33%, 33%} and D2 = (33%, 33%, 17%, 17%), as illustrated in Figure 3. We train relevant baselines over each dataset and use the interpolation eval- uation analyze the distribution of learned strategies in the latent space. We observe that Co-GAIL is able recover all the strategies, while the baselines are not able to generate the less represented strategies or have comparatively fewer of them. Similar results are also ob- served in the digensiinnni task HR-
Table 2: Interpolation evaluation: Success rate (mean and 95% confidence interval over three seeds).
Interpolation HR-H.
2D-FetchQQ
HR-S.
MA-GAIL|26]
78.5 + 7.2 99.2 + 1.4 46.2 + 10.3
MA-InfoGAIL/25)
80.2 + 3.0
96.4 + 4,1
51.7 7.5
DIAYN[321
80.2 + 3.6
86.2
9.0
45.4 7,9
Co-GAIL (ours)
82.1 3.1 81.2
3.1 31.6 2.5
Handover. We analyze the range of 3D locations in which the model interpolation performs suc- cessful human-to-robot handover (HR-Handover). A top view of the space between the human and the robot is shown in Figure 4a for two different random seeds. Handover locations of MA- InfoGAlL. tend to concentrate instead of covering the demonstrated distribution. We also observe that, since DIAYN encourages the learned policy as diverse as possible, it also explores strategies that are less plausible in real human-robot collaboration. Co-GAIL (green dots) shows better cover- age across the range of the dataset on both the X and dimensions (Z in Appendix due to space). We further investigate the interpolation results of each method and find that MA-GAIL, MA-InfoGAIL, DIAYN have high success rates during interpolation (Table 2) while performing poorly in replay evaluation (Table 1), indicating that previous methods tend to overfit, while Co-GAIL eo-lorse the dataset more broadly.
The learned latent space maps to qualitatively different human strategies (Figure 4b): In this experiment, we verify if regions of the learned latent space correspond to distinct recognizable human strategies. For Handove, divide the handover locations projected on the horizontal
7

---

Datnset
MA-InEoGAIL
DIAYN
Co-GAIL
MA-InfoGAIL
DIAYN
Co-GAIL
Seed
HRR- Handenner
Stratogy
Stratogy
Strategy 3
Xekentoniin) Seed 2
HR- Scayhuiii
Yo (r)
-0.00
-30
Strilgg
Stralegyy
(a) Handover positions
(b) Latent strategy visunlization
Figure 4: Diversity and strategies encoded latent space. (a) Top view of successful handover locations from HR-Hardover interpolation. (b) Visualization of the correspondence of learned latent strategy space (2D domain) to different type of manipulation strategies (colors).
plane XY into three regions, obtaining handovers executed at the left (S,), middle (S2), and right (Sa) sides. For Ma-Se we determined two possible strategies (S1, S2) depending on the human and robot roles (see Appendix). We use the recognition model of each method compute an average latent code for each unseen trajectory and visualize the code in Figure 4b, where each code is colored by its strategy category. Codes for the same strategy tend to cluster in the same region. We observe a better separation of latent strategy learned using Co-GAIL than the other two methods. This shows the value of our diversity objective in Equation 3.
Real human user interacting with Th: We performed proof-of-coneep analysis with real users to assess rep- resentative methods. Users interacted with all methods in randomized order. We report the success rate out of 20 tri- als per user per method per domain in Table 3. Co-GAIL consistently outper- forms the other methods except for one case in the HR-Handover. In this case, the user performed handover in small area that overlapped with the region that MA-InfoGAIL overfit Co-GAlL. showed the best adaptation to diverse strategies across users and trials. While a small sample size, these promising results are the first demonstration of learning method that succeeds at adapting to real users in these challenging tasks with continuous motion and varied strategies.
Table 3: Real- evaluation
2D Fe-cct
HR-H.
HR-S.
User id
1
4
2
2 3
4
BC-single
70 65 50 65
65
10 20
10
5
10
15
MA-GAIL(24]
65 50
45
50
35
25
20
30
25
35
25
MA-InfoGAHI.(25)
85 80 40
50
55
90
65
25
35
40
40
50
Co-GAIL
100 90 85 80 75 70 70
60
50
65 70
6 Conclusions and Future Work
We present Co-GAIL, method for training robot assistants that can handle diverse human behav- iors. Our method represents novel approach for taking advantage of simulated environments in HRI research. This is achieved by leveraging data of human-human collaboration demonstrations as guidance to concurrently generate simulated interactive behaviors and train a human-robot col- laborative policy. We test Co-GAIL in a 2D game and two high- high-dimensional manipulation tasks, with both procedural evaluation and online human-in iietaan-i--thee in-the-loop evaluation. While Co-GAIL outper- forms other state-of-the-arr methods in challenging HRC tasks, a few challenges remain open for future works. In current implementation, the roles of leaders and followers must be assigned to the human agent and the robot agent in advance. Enabling the agents to flexibly switch roles online would further improve the collaboration experience. Another future direction is apply the learned policy to real robot. A possible approach is to align the observation space of the simulated environment and the real world through state estimation and employ domain randomization [49] to allow the agent to be robust against domain shift in sim-to-real transfer.
8

---

Acknowledgments
We thank the Stanford Institute for Human-Centered Artificial Intelligence and the HAI-AWS Cloud Credits for Research program for its support with compute resources. Chen Wang thanks Lihua Wang and Zhongqing Zheng for helping with the development of the data collection system.
References
[1] J. A. Marvel, S. Bagchi, M. Zimmerman, and B. Antonishek. Towards effective interface designs for collaborative hri in manufacturing: metrics and measures. ACM Transactions on Human-Robot Interaction (THRI), 1-):1155,, 2020.
[2] W. Yang, C. Paxton, A. Mousavian, Y.-.. Chao, M. Cakmak, and D. Fox. Reactive human- to-robot handovers of arbitrary objects. In IEEE International Conference Robotics and Automation (ICRA). IEEE, 2021.
[3] T. Zhang. Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality telcoperation, In 2018 JEEE International Conference Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018.
[4] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
[5] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement lcarning for vision- based robotic manipulation. arXiv preprint arXiv:1806. 10293, 2018.
[6] Akkaya, M. Andrychowiez, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
[7] J. 7. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv: (000::7020003333 2017.
[8] A. Xie, D. P. Losey, R. Tolsma, C. Finn, and D. Sadigh. Learning latent representations to in- fluence multi multi-agent interaction. Proceedings of the 4th Conference on Robot Learning (CoRL), 2020.
[9] M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou. Tuyls, Pérolat, D. Silver, and T. Grae- pel. A unified thme-theoretti approach to multiagent reinforcement learning. arXiv preprint arXiv:1711.0083: 2017.
[10] A. Tampuu, T. Matüsen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4): c0172395, 2017.
[11] T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.08531, 2020.
[12] S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In 2015 10th ACM/EEEE International Conference Staman-Roban Interaction (HRI), pages 189-196. IEEE, 2015.
[13] J. Lee. survey of robot learning from demonstrations for human-robot collaboration. arXiv preprint 17 2017.
[14] D. Vogt, Stepputtis, S. Grehl, B. Jung, and H. B. Amor. A system for learning continuous human-robot interactions from human-human demonstrations. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2882-2889. IEEE, 2017.
[15] E. Coumans et al. Bullet physics library. Open source: bulletphysics. org, 2013.

---

[16] B. Shen, F. Xia, C. R. Martn-Marrtn, L. Fan, G. Wang, S. Buch, C. D' Arpino, S. Srivastava, L. P. Tehapmi, K. Vainio, L. Fei-Fei, and Savarese. igibson, simulation environment for interactive tasks in large realistic scenes. arXiv preprint, 2020.
[17] P. Peng, Q. Yuan, Y. Wen, Y. Yang, Z. Tang, H. Long, and J. Wang. Multiagent bidirectionally- coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv: 10069, 2:2, 2017.
[18] L. Matignon, L. Jeanpierre, and A.-I. Mouaddib. Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, 2012.
[19] R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta. Efficient bimanual manipulation using learned task schemas. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1149-1155. IEEE, 2020.
[20] E. Vinitsky, A. Kreidieh, L. Le Flem, N. Kheterpal, K. Jang, C. Wu, F. Wu, R. Liaw, E. Liang, and A. M. Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference Robot Learning, pages 399-409. PMLR, 2018.
[21] U. Jain, L. Weihs, E. Kolve, A. Farhadi, S. Lazebnik, A. Kembhavi, and A. Schwing, A cordial syne: Going beyond marginal policies for multi-agent embodied tasks. European Conference Computer Vision, pages 471-490, 2020.
[22] S. Natarajan, G. Kunapuli, K. Judah, P. Tadepalli, K. Kersting, and J. Shavlik. Multi-agent inverse reinforcement learning. In 2010 Ninth International Conference on Machine Learning and Applications,, pages 395-400. IEEE, 2010.
[23] X. Lin, P. A. Beling, and R. Cogill. Multiagent inverse reinforcement learning for two-person zero-sum games. IEEE Transactions on Games, 10(1):56-68, 2017.
[24] Ho and S. Ermon. arXiv:1606.034777 2016.
Generative adversarial imitation learning. arXiv preprint
[25] Y. Li, Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. arXiv preprini arXiv: 100::/703.0000 778.00840, 2017.
[26] J. Song, H. Ren, D. Sadigh, and S. Ermon. Multi-agent generative adversarial imitation learn- ing. arXiv preprint arXiv: htt.:8807.09938 2018
[27] S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically moti- vated reinforcement learning. arXiv preprint arXiv:1509.0873: 2015.
[28] Jung, D. Polani, and P. Stone. Empowerment for continuous envirnenvironment systems. Adaptive Behavior, 19(1):16-39, 2011.
[29] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Curiosity-driven exploration in deep reinforcement learning via bayesian neural networks. 2016.
[30] Achiam, H. Edwards, D. Amodei, and Abbeel. Variational option discovery algorithms. arXiv preprint 1 2018
[31] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint 1611::611 2016.
[32] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills 2018.
without a reward function. arXiv preprini
[33] A. Clegg, Z. Erickson, P. Grady, G. Turk, C. C. Kemp, and C. K. Liu. Learning to collaborate from simulation for robot-assisted dressing. IEEE Robotics and Automation Letters, 5(2): 2746-2753, 2020.
[34] Z. Wang, J. Merel, S. Reed, G. Wayne, N. de Freitas, and N. Heess. Robust imitation of diverse behaviors. In Proceedings of the s1 International Conference on Neural Information Processing Systems, pages 5326-5335, 2017.
10

---

[35] A. J. Lockett, C. L. Chen, and R. Miikkulainen. Evolving explicit opponent models in game playing. In Proceedings of the 9th annual conference on Genetic and evolutionary computa- lion, pages 2106-2113 13, 2007.
[36] Z. Wang, K. Mülling, M. P. Deisenroth, H. Ben Amor, D. Vogt, B. Schölkopf, and J. Peters. Probabilistic movement modeling for intention inference human-robot interaction. The International Journal of Robotics Research, 32(7):841-858 2013.
[37] R. Raileanu, E. Denton, A. Szlam, and R. Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning, pages
PMLR, 2018.
[38] V. V. Unhelkar, S. Li, and J. A. Shah. Decision-making for bidirectional communication in quential human-robot collaborative tasks. In Proceedings of the 2020 ACM/IEEE International Conference on Human- Human-Robot Interaction, pages 329-341, 2020.
[39] C. Pérez-D's Arpino and J. A. Shah. Fast target prediction of human reaching motion for coop- erative human-robot manipulation tasks using time series classification. In 2015 IEEE inter- national conference on robotics and automation (ICRA), pages 6175-6182, 2015.
[40] Zhang, A. Cully, and Y. Demiris. Probabilistic real-time user posture tracking for personal- ized robot-assisted dressing. IEEE Transactions on Robotics, 35(4):873-888 2019.
[41] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa. Towards robotic feeding: Role of haptics fork-based food manipulation. IEEE Robotics and Automation Letters, 4(2):1485- 1492, 2019.
[42] D. Park, Y. Hoshi, and C. C. Kemp. A multimodal anomaly detector for robot-assisted feeding using an 1stm-based variational autoencoder. IEEE Robotics and Automation Letters, 3(3): 1544-1551, 2018.
[43] S. Schröer, I. Killmann, B. Frank, M. Völker, L. Fiederer, T. Ball, and W. Burgard. An tonomous robotic assistant for drinking. In 2015 IEEE international conference on robotics and automation (ICRA), pages 6482-6487. IEEE, 2015.
[44] M. Littman. Markov games a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[45] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets. arXiv preprint 1600::066777 2016.
[46] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization 2017.
algorithms. arXiv preprint 1
[47] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, Gao, Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. Conference on Robot Learning, 2018.
[48] A. Tung, J. Wong, A. Mandlekar, R. Martin-Martin, Y. Zhu, L. Fei-Fei, and S. Savarese. Learning multi-arm manipulation through collaborative teleoperation. arXiv preprint arXiv:2012.067385 2020.
[49] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810, IEEE, 2018.
[50] X. Mao, Q. Li. H. Xie. R. Y. Lau, Z. Wang, and S. Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[51] C. Lugaresi, J. Tang. H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, Lee, al. Mediapipe: framework for building perception pipelines. arXiv preprint arXiv: 08172, 2019.
11

---

[52] C. Wang, R. Martín-Martin Martin, D. Xu, J. Lv, C. Lu, L. Fei-Fei, S. Savarese, and Y. Zhu. 6- pack: Category- level 6d pose tracker with anchor- keypoints. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059-10066. IEEE, 2020.
12


Reviews:
[{"review_id": "zUjQgB64Gd", "reviewer": "Reviewer_qTpv", "summary": "This paper presents a method for learning two models for human-robot collaborative tasks: one that reproduces human-like behaviors, and one that optimizes the robot's behavior for efficient collaboration. In learning the former, the human's strategy is represented as a latent representation and used to produce trajectories that maximize informativeness. The results indicate that the proposed method produces trajectories that reflect a variety of strategies demonstrated by a human (even when those strategies are unbalanced in the dataset). Furthermore, the proposed method resulted in more successful handovers during a real robot task.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and easy to follow.\n* The motivation is convincing, and the related works section clearly explains why the proposed method provides a benefit over current SOTA.\n* The evaluation is very thorough as a result of comparing the proposed method to four different baseline approaches.\n* The results themselves are compelling, and test each part of the system (latent space learning, policy learning, and execution).\n\nWeaknesses:\n* More discussion on the effect of the hyperparameters in eq 4 would be useful.\n* During training, what defines the ground truth for z in eq 2?\n* Axis labels in Figures 3 and 4 are way too small", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper is clear and presents thorough, compelling results. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* More discussion on the effect of the hyper parameters in eq 4 would be useful.\n* During training, what defines the ground truth for z in eq 2?", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "Qz7eR31PyP2", "reviewer": "Reviewer_RH4p", "summary": "This paper proposes a learning approach to train control policies that allow robotic agents to assist human partners in collaborative tasks. \nThe main insight is to incorporate the diverse range of behaviors/strategies that humans may follow during collaborative tasks into robot control policies for a more effective joint task execution.\nSimilar to prior work, the proposed approach first trains a collaborative policy using human-human demonstrations that results in a shared policy network that outputs actions for both agents (human and robot). \nThis policy learning is then extended, again similar to prior work, by integrating a latent representation of human strategies, i.e., by incentivizing the human policy to condition on the strategy to replicate demonstrated human actions. \nTo improve capturing diverse human behaviors, this latent representation and the human policy are also co-optimized as an autoencoder using demonstrations. \nThe results highlight comparable or better performance in terms of strategy coverage and responsiveness on the three tasks being evaluated.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this work is its learning architecture that effectively integrates human strategies. \nThis integration allows modeling human behaviors, which in turn improves the robustness of trained robot policies that rely on them.\n\nOne critical weakness of this work is that the diversity of human strategies is not analyzed and discussed within the main text. It's not clear for continuous problems (such as handover or sequential task) how to identify human strategies and then analyze them. The appendix includes further info but I believe as a core aspect of this work, the main manuscript should provide a better explanation on those details. \n\nSecond weakness is the experimental setup used for HRI scenarios. The user interface is based on a teleoperation interface (using mobile phones), which I assume would not provide representative human movements, and thus spoils their strategy selection and execution preferences. \nA more representative teleoperation interface would make the experiments and results more convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel extension of prior work which improves human-robot collaboration performance for three different tasks. This is achieved by incentivizing the encoding of diverse human strategies while learning robot policies. The experimental setup in terms of how the data was acquired remains to be a critical issue in order to justify their representativeness of real human movement and strategy selection behaviors.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Please also see above (Strengths & Weaknesses).\\\nSome further improvement points:\\\nThere is a strong emphasis on capturing diverse human behaviors, but even though this is supported by the interpolation evaluation on the 2D-Fetch-Quest task, looking at the other tasks' results (e.g., Fig4), I couldn't identify a significant (qualitative) difference between the proposed Co-GAIL approach and DYIAN. \nAlso, I'd recommend to not only focus on the positive results of the proposed approach but also where it underperforms, e.g., success rate of interpolation evaluations for the HRI experiments (Table 2).\nI'd suggest using a better/fairer phrasing and/or quantitative comparison, especially to justify diversity claims.\n\nReal human user interacting with robot policy: Even though it's a proof-of-concept, it's not clear what the protocol for this experiment was:\n- who were the subjects? how much they had experience with robotics and/or this particular setup? are they PhD students, friends, random people? etc.\nAs a qualitative analysis, it might be fine to report on, but a quantitative analysis would be unreliable without a proper user study protocol.\n\nsmall issues:\n- typo: caption of Fig.1: 'Pick and object ...', line 212: manipulate -> manipulating, line 231: reply -> replay, line 271: '... instead (of) covering ...'\n- misreference: line 152: Figure 2b -> 2a, line 160: Figure 2c -> 2b, line 245: Table 2 -> 1, line 277: Table 1 -> 2 and Table 2 -> 1\n"}}, {"review_id": "EDVh2tXI_Ct", "reviewer": "Reviewer_4Wjg", "summary": "This paper presents a technique to generate robot behavior that can collaborate successfully with humans by learning from human-human interaction. They model this as a two-agent imitation learning problem where behaviors for both the human and the robot are learned concurrently. Their main contribution is to extend the MA-GAIL [26] framework by introducing a latent representation of the human's strategy, also learned from the data, which enables the robot to adapt to the diverse set of behaviors exhibited by the human during training. They test this model in three collaborative tasks and show that it outperforms several baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n - The paper is well-written.\n - The technique is motivated by human behavior and grounded in past work.\n - Tested in multiple domains.\n - Experiments included their model interacting with humans which is rare.\n\nWeaknesses: \n- The experiment descriptions are missing some details like the user recruitment procedure including for the training data. Also, were the different strategies that the humans used pre-decided? Was any form of communication permitted between participants?\n- The superior performance of the baselines in table 2 is explained in the paper with overfitting. Can you provide some more detail why that should be the case since Interpolation is not necessarily playing trajectories from the training data? If I understand correctly,  the interpolated trajectories should also contain multi-modal behavior that MA-GAIL should find difficult to adapt to. Also, would be helpful if it is explained how the strategy sampled here was learned.  \n\nThat makes sense, seems to indicate that the proposed method underperformed the baselines. It would be useful if the authors presented some reasons for this.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well-written, they describe the problem clearly. Their solution seems logical given the explanation of past work. Their experiments provide a thorough test of their method. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- More details relating to the experiment design as well as the participants used for the study as described in the weaknesses.\n\n- Space permitting, it would be good if authors can mention more detail about their experimental setup for reproducibility\n\n- Is there a plan to release the software and data? \n\n- Analyze and explain the results from table 2.\n\n- The description of Fig 4b is difficult to understand.\n\n- In 4.3, which learned strategy space is being used for Interpolation?\n\n- line 24, not 30% in the HR-S\n\nTypos/ minor issues:\n- (line 116) an -> a\n- (line 231) reply -> replay, if -> of \n- (line 244) table 2-> 1\n- (line 249) table 2-> 1\n"}}, {"review_id": "Bx0xsc5PfQp", "reviewer": "Reviewer_wpop", "summary": "The authors made a good faith attempt to improve the quality of their evidence, and to improve the means by which they analyzed the evidence. As a result of this, I am more confident that Co-GAIL is better than their baselines, although I am still somewhat confused by the evidence. In particular, a T3 Dunnett post-hoc test found that Co-GAIL was only significantly superior to MA-InfoGAIL in one of three test environments, and was not significantly superior in another two. However, a GLM predicting success rate against the model, task, and user found that the weight on Co-GAIL was significantly higher than the weights on the other method terms. The only differences between these comparisons is that the GLM aggregates across tasks, and has a per-user term (which seems justified with n=20 trials per user). I am not an expert statistician, but these results taken together seem to say that Co-GAIL is a lot better on some tasks, and not obviously worse on the rest. The authors also gave a strong explanation for why my interpretation of Table 2 was incorrect, and the explanation in their paper was likely to be correct. This is enough for me to shift my judgement from weak reject to weak accept. The original review follows unchanged.\n\n===\n\nDoing imitation learning in an environment that requires interaction with a human is not handled well by most imitation learning methods, as they tend to assume any human involvement is in the form of a fixed human policy that does not react to the robot. This is either false (if the robot's state is not well captured in the state) or difficult to model and collect data on (when it is).\n\nIn order to address this issue, there exist a collection of methods that attempt to estimate both a human and a robot policy from a collection of demonstrations where humans act as both the human in the environment and the robot helping them accomplish a task. The most successful of these methods are MA-GAIL, which adapts the GAIL framework to a multi-agent setting, MA-InfoGAIL, a modification of the above that considers a latent strategy variable, and DYIAN, which tries to maximize a diversity criteria. MA-InfoGAIL and DYIAN have policies that are conditioned on a strategy latent variable that attempts to capture the fact that humans will act according to different strategies even in the same state.\n\nThe novel contribution of this paper is to introduce Co-GAIL, a modification of MA-InfoGAIL that includes an additional loss designed to prevent mode collapse in the estimation of the latent strategy variable. This new method (and several baselines) are evaluated on a 2D-gridworld and two 3D physics based object manipulation tasks. The authors find that, according to some evaluations using historical (fixed) human data and a small pilot study of humans reacting live to the robot's actions, co-GAIL outperforms all of the baselines overall. They also prevent evidence that mode collapse is a problem in prior methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper has a very strong introduction, and a very clear explanation of their method. I was not familiar with the MA-GAIL line of work, but was able to follow their paper.\n\nThe novel contribution is a natural solution to the problem at hand.\n\nThe paper provides evidence that the problem it is trying to fix actually exists, which is something many neglect to do.\n\nThe paper compares its method to a fairly comprehensive set of baselines.\n\nWeaknesses:\nThe quality of evidence is low in many experiments. Only 3 random seeds were used in the simulated human experiments, and the reported variances are quite high. Only one replication was performed on the human data, which is understandable for a pilot in present circumstances, but it makes it hard to strongly believe your results.\n\nIf I understand things correctly, the interpolation experiments have a significant conceptual flaw. Considering a uniform prior over strategy variables seems very inappropriate when we expect the strategy encoder to eventually concentrate around a few modes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I cannot recommend this paper until further replications are performed. The simulated experiments only use three replications, and the human trials only use four subjects each. The spread in data values is very often large enough to be concerning. The method itself is reasonable, and qualitatively CoGAIL does appear to match the human's trajectory distribution better, but I simply cannot be confident that the method actually outperforms DYIAN in all cases.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "My biggest issue is that three replications simply isn't enough to be confident that your method is actually superior to the baselines. In 2D-FetchQ the 95% confidence intervals (assuming a Gaussian, which is hard to do when n=3) of DYIAN and Co-GAIL overlap. This is not the case in the 3D environments, which is something, but the variances of your method in those environments is also quite high, especially for n=3. It's also not reassuring that your method did worse than the others in the interpolation setting. It is true that the other methods that learn mode collapsed policies have an easier task here, but it betrays a weakness in your policy learning step if you can't learn a robot policy that is near optimal under your own model of the human.\n\nUltimately both of the simulated human cases provide some evidence, but aren't conclusive. Replay is limited in that the human can't react to the robot's actions, which was a major part of the motivation for this paper. Interpolation does have a reactive human, but lacks external validity. The human pilot study is the best evidence available, but unfortunately there were only four subjects available. At least on the 2D environment, you can get subjects over the internet. It is hard to view four replications as strong evidence. Admittedly the scatterplots for 2D-FetchQ and HR-S look fairly separate, but there is definitely room for DYIAN to come out on top in the handover task with more data. I would also recommend you produce those scatterplots, and also report mean scores across different subjects, at least in the appendix.\n\nI also have a conceptual issue with how the interpolation experiments were done in terms of the reported success rate. When training, you learn a psi function to map human actions to a latent strategy. An ideal psi function captures multiple human modes of behavior, even at identical states. A psi that is spread out over all the different values in the strategy space is not very informative for the policy. So we expect that the empirical values of z will concentrate around a few different modes. If the goal of the reported success rate is to give a self-consistency check on the co-policies, taking a uniform prior over strategies seems inappropriate. What I would do in your position would be to record the z values along the fixed human trajectories you have, visually (or automatically) identify a few mode points for each method, and then use those mode strategy values to simulate the human's policy. Using a uniform prior to identify the diversity of human trajectories and compare that to the prerecorded trajectories is reasonable.\n\nAs an extremely minor issue, some implementations of GAIL use log(D(.)) as the proxy reward, and some use -log(1-D(.)). For the sake of completeness it would be nice if you could mention this somewhere in the appendix.\n\nThere are several grammatical issues, notational inconsistencies, and inconsistencies between the appendix and the main paper. I expect all of these can be fixed quite simply, and I provide them mainly for the benefit of the authors.\n\n62: \"agents are considered\" -> \"agents are assumed\"\n84: \"which is crucial especially\" -> \"which is especially crucial\" or \"which is crucial, especially\"\n112, 126: $\\mathcal{M}$ is both the Markov game and the set of demonstrations, pick a different name for one or the other\n131: \"that the start state is $s_i \\sim \\rho_0(\\cdot)$\" -> The way this is written was confusing, and this is such a standard assumption that I don't think you need to mention it at all.\n143: \"$\\mathbf{z}_t \\sim \\psi(\\mathbf{z}_t | ...)$\" -> \"\"$\\mathbf{z}_t \\sim \\psi( \\cdot | ...)$\"\n155: \"when producing action\" -> \"when producing actions\"\n191: \"Equation1\" -> \"Equation 1\" You're missing a space.\n192: \"infoGAIL\" -> \"InfoGAIL\" Capitalization.\n212: \"collaboratively manipulate\" -> \"collaboratively manipulating\" or rework the sentence more broadly.\n245, 249, 277: You appear to have flipped table 1 and table 2 in several places.\n298: \"MA-InfoGAIL overfits to\" -> \"MA-InfoGAIL overfit to\" tense issues.\n315: \"to the real robot\" -> \"to a real robot\"\n\nTable 1, 2, appendix: The main paper says you report variances, but the appendix says you report standard deviation. As a notational issue, X +- Y is only for something which is an interval, like a confidence interval. If you want to report a mean and variance or a mean and standard deviation, the notation is X (Y). Additionally, if you're going to bold specific items that are the best, you need to bold all items that COULD be the best, meaning that you bold the item with the best mean, and all other rows whose confidence intervals overlap with the confidence interval of the row with the best mean."}}]

Author rebuttal:

Meta-review: The paper received 4 reviews which are on average leaning slightly towards an accept. The paper is well written and well motivated, the evaluation is thorough. The paper heavily builds on prior work, result in rather low impact/contribution scores. Some details need to be clarified. The most important concerns circle around the experiments. Some of the results (in particular why the proposed method did worse in one setting) need a deeper analysis. Related to that there are also some doubts about the validity of the setup and results for some of the experiments.

## Update after author response and AC & reviewer discussion
The reviewers appreciated the detailed responses and additional experiments and analysis. They are now all in favor of accepting the paper.
Decision: Accept (Poster)