Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a general methodology for sim-to-real transfer learning for reinforcement learning policies in human-robot interaction scenarios. The method, Iterative-Sim-to-Real (i-S2R), attempts to simultaneously learn a model of both the target policy and the corresponding human behavior in an iterative fashion. An initial estimate of the human behavior is made and used to train the target policy, after which the policy is deployed and fine-tuned in the real world against an actual human. The resulting data is then brought back to simulation to update the human behavior model and the target policy with further training. This process iteratively repeats as needed. This method proposes a solution to the problem of leveraging simulation when humans are involved – how do we model the associated human behavior in simulation? The resulting method is applied to a game of table tennis, and shows improvement over a standard sim-to-real approach. 
strengths_and_weaknesses: Strengths:
* The paper is well-written and well-motivated; the sim-to-real problem being addressed is very relevant to human-robot interaction and so contributions in this area are of great interest to the HRI community.
* The iterative training methodology which interleaves real-world policy rollouts with simulated data to improve the fidelity of the human behavior model is fairly novel and interesting. While the concept of interleaving real-world policy rollouts itself is not (as pointed out in the related work), this has typically been focused on non-HRI scenarios and used to improve robot/environment dynamics models.
* The inclusion of the s2r-Oracle model in evaluation is very insightful. The effort required to perform multiple real-world policy rollouts with human partners as in i-S2R may be overly prohibitive, but if this process only needs to be performed once to learn a sufficient human model which can then be used to train new policies from scratch, this would go a long way to making such a methodology more broadly applicable. However, this does beg the question: how does this compare to collecting human data up front (with another human partner) and learning the human behavior model with imitation learning? It is not entirely clear to me that the robot is required to collect human-data in this table tennis scenario. However, in cases where the robot’s presence is required for human data collection (e.g., there would be significant distribution shift when a robot is present vs a human), such an approach seems worthwhile.
* The results are quite impressive! The included video showing the robot executing its table tennis policy with a human opponent for up to 150 hits in a single rally leaves an impression.

Weaknesses:
* The small sample size in human participants somewhat limits the conclusions that can be drawn from the experimental results. While this is clearly acknowledged in the paper and is somewhat inherent to human-robot interaction, the outlier that is the advanced player introduces many questions about under what conditions i-S2R is preferable over standard S2R.
* No conjecture seems to be made about why i-S2R performs worse for the advanced player. Is there possibly some aspect of their play that may have changed? Are there any simulation reward curves that might indicate a failure during the training process during the latter iterations? Did the fine tuning process go awry?
* The fine-tuning in the real-world process doesn’t necessarily make sense to me. Presumably, one needs to fine-tune the policy in the real-world because there is some slight distribution shift between simulation and real? But then why take the fine-tuned policy back into simulation? It seems like this would constantly be introducing distribution shift into the training process and make things more difficult. Prior works like [2] attempt to adjust the simulation models to make it more realistic, thus reducing the distribution shift on every iteration. But in this case, I’m not sure what the fine-tuning is accomplishing or why one would want to bring it back to simulation.
* It would have been interesting to train a single s2r-oracle policy with the aggregated data from all players. Would it generalize better than the individual policies given that the behavior model represents a wider distribution of player data? It is not entirely clear to me why individual policies were trained for each player rather than mixing and matching data.
* Minor: The right side of Fig. 2 is a bit difficult to understand. After reading the textual description of the different variants (Sec. 6) and spending a significant amount of time trying to understand the figure, I think I understand it, but it is not intuitive at first glance.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper addresses an important research problem, proposes an interesting method, is well-written, and invites interesting discussion about the results. While aspects of the method are not entirely novel and the small sample size in human participants yields a lot of variance in the data, I think this paper is of interest to the broader robotics -- and in particular HRI -- community given the challenge in leveraging reinforcement learning when humans are involved. The trained table tennis policies are also quite impressive and demonstrate some significant steps forward, particularly with successful play against novel opponents in the real-world.
issues: * Why is there such a significant difference between s2r-oracle and s2r-oracle-out-of-sim? This seems like it would imply that the human behavior model doesn't actually make that much difference, more that it is the fine-tuning process that makes a significant difference. Are there any results for the intermediate iterations? Results for 45k sim and fine-tune 30% might shed some more light on this, i.e., is it the behavior model or sim that impacts things.
* As noted above in Weaknesses, why is the fine-tuned model from the real-world brought back to simulation? What exactly is being fine-tuned here? One would assume that this is correcting for distribution shifts in the dynamics, but then it wouldn't make sense to bring these changes back to simulation. More information or intuition here would be beneficial.
* Are there statistical significance results for the comparisons between the different S2R variants?
* What distinguishes a beginner, intermediate, and advanced player? This information would be useful to know, either in the paper or appendix.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Enabling robots to imitate human behavior in robot-human interactive domains can be challenging, since a robot policy must already know how to interact with humans in order to obtain useful examples of human behavior to learn from. The work presents a novel method, iIterative-Sim-to-Real (i-S2R), that provides a structured way for a bootstrapped human model prior to be trained in simulation and then further refined in the real world in an iterative loop. The method is evaluated on the table-tennis task across 5 human subjects with varying skill levels, and is shown to outperform a non-iterative variant in most cases, resulting in respectable rally lengths on average (22) and significantly longer rallies (up to 150) as well.
strengths_and_weaknesses: Strengths:
- The model performance on real robot hardware is impressive, achieving respectable rally lengths on average (22) and significantly longer rallies (up to 150) as well.
- The presentation is clear, and details are easy to follow.
- The subject evaluation method (blind comparisons between subjects and trained models) is well-grounded, and removed the potential for biased performance between subjects.
- Overall experimentation is very thorough, with clear motivations for each setup.

Weaknesses:
- The method seems to require strong human prior modeling – i.e.: a distribution that can be both expressed analytically and simulated accurately in simulation. This may be a strong assumption, and could limit the method being generally applied to more complex settings where such modeling might be intractable (e.g.: the cluttered navigation problem described in L61-65 – see related comment below).
- It does not seem clear that the table tennis setup actually represents more ecological tasks, such as the busy hallway navigation task claimed in L61-65. The domain shift seems rather large and nontrivial, especially since the human model would have to account for N other humans (instead of just one), and will probably be in constant “generalization” mode since it is unlikely to have the exact same configuration of humans repeated.
- The advanced player’s data is obviously surprising. The relative outperformance of S2R over i-S2R performance isn’t necessarily concerning (it could perhaps be explained by the consistency of the professional player), but the i-S2R score in comparison with the other plays seems very counterintuitive – suggesting that the method somehow performs better on “worse” players (intermediate, ~40 hit mean rally length) compared to “better” players (advanced, ~20 hit mean rally length).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a novel iterative method that exhibits impressive empirical results on real robot hardware. The experimentation method is extensive and thorough, and showcases the strength of the method across varying human subjects. Further discussion on the advanced player’s surprising data, as well as more explicit justification of the applicability of the proposed method to more complex robot-human interactive setups, would further strengthen the core arguments presented in the paper.
issues: Questions:
- How dependent is the method of accurate modeling? E.g.: can i-S2R still succeed with less-accurately tuned ball parameters (L189-190)?
- What was the motivation behind choosing 3 fine-tuning stages, vs. additional stages? It does seem necessarily straightforward that the learned human model has “converged” by iteration 3 (which raises another question – what exactly defines “convergence” in this setup?).

Suggestions:
- An ablation study testing a completely “random” human model (fully randomized start pose, vel, landing x, y, etc.) would help validate that humans are actually necessary in the data bootstrapping process, and help contextualize the claim in L277-278.

Technical Concerns:
- The variable scaling in the y-axis between subfigures in Figure 4 seems a bit misleading.
- From what I understand, the initial data from the human used in the first simulation iteration consists solely of single hits without rallying (L164-165). If this is the case, then Fig 6 seems a bit misleading – there is an obvious domain shift between sim1 and sim2 since rallying only starts to occur in sim2+, which may explain the larger delta values seen in that figure.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes an iterative training scheme where a policy is trained in simulation and then deployed on a real-robot to fine-tune and collect data from a human player. This iSim2Real system was implemented on a ABB robot arm and demonstrated to collaboratively play table tennis (a rallying task) with beginner and intermediate human players.
strengths_and_weaknesses: Positives:
- A big plus of the paper is that the approach was demonstrated and tested on a real-world 8-DOF system on the relatively complex task of table-tennis.
- The iSim2Real framework is simple, but it is not immediately obvious that it would work with real human players.
- Ablation studies are presented showing that the system attains a good policy relatively quickly, along with the impact of improving the human model and play performance across human players.  
- The paper is generally well-written and I enjoyed reading it. 

Negatives:
- The human model in this setup is relatively simple; it is the ball distribution and the demonstration is limited to this setting. There are no “human aspects” that are directly modeled (e.g., fatigue, expertise, trust) though potentially, this can be captured by the iterative setup at the cost of sim iterations. It would be helpful if the authors discuss this limitation and how the method can be extended towards more complex human models.
- A related issue is that humans change over time, e.g., they can get better over time or can adapt to the robot’s policy. This is not discussed with sufficient depth in the paper but is important in collaborative settings. In the experiments, did the human and robot find a good *joint* policy? Does the i-S2R system facilitate this? Is the system able to adapt to a changing human? Although I understand experiments into these issues may not be possible given the difficulties associated with human-subject experiments, some discussion is required. 
- The experimental results are for 5 human subjects (4 of which are the papers authors) and on a single task; it is difficult to draw very strong conclusions from the limited participant pool. 
- Something that was unclear in the paper is the key differences between advanced v.s. beginner/intermediate players. Perhaps the authors can analyze the differences between the learnt model distributions? This could also lead to further insights as to why i-S2R did not work well with advanced players. 

Minor:
- I did not understand why there are 5 human subjects since each model was evaluated by “(a) the model’s trainer and (b) two other players”.
- it is a little odd that the authors refer to the model as having “converged” since the human may continue to change over time.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, the strengths of this paper outweigh the negatives. I especially appreciate that the authors conducted experiments with a real table-tennis robot. The major downsides for me were that the human model (for this setup) is simplified to an extent that the “human” elements appear to be missing, and that the participant pool was limited in size. The latter is somewhat of a missed opportunity; with a larger human-subject experiment, I would have recommended a strong accept. 
issues: Please see the weaknesses outlined above, and I hope the authors can provide additional discussion around the potential of iS2R to more complex human models and other settings beyond table tennis.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed an iterative sim-to-real method for learning to play robotic table tennis. This method trains a policy in simulations, obtains data of rallies between a human and a robot using this policy, and updates the human ball trajectory model using obtained data.
Experimental results indicate that the proposed method outperforms a fine-tuning method and demonstrates the effects of the parameters of the method and levels of a player on the performance of the method.
strengths_and_weaknesses: **Strengths**

- Introduction describes the background and motivation. This reviewer agrees that it is important for a robot to learn to interact with a human using simulations while updating the simulation using data from trials in the real world.
- Related work section introduces related topics and compares the proposed method with previous methods.
- The sections of the method, system, experiments, and limitations are written clearly and easy to follow.

**Weaknesses**

- All issues have been resolved.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The reviewer recommends Weak Accept because the authors resolved all issues that the reviewer concerned.
issues: - All issues have been resolved.
review_title: null

Decision: Accept (Oral)