Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work shows that a PR2 trained to fence against another PR2 can used the learned policy to compete with humans, challenging them with a Fencing Game to encourage physical activity. The paper shows that not only were the robots able to compete physically with human users, the users enjoyed the experience. This work is a good start for a “robot workout buddy” that becomes competitive through training with other robots, and then encourages people to workout using competitive games.
strengths_and_weaknesses: This paper is well written and clear. The idea of competitive robotic partners to encourage exercise is really interesting, and it seems like the execution worked quite well. I appreciate that this work trained agents with different gameplay styles, and also tested whether people would learn how to play against the robot quickly and become bored. The results of this paper seem to support the claim that a robot can train to compete with a person in a physical game, and that people enjoy playing against the robot.

While the results show that the system worked and was enjoyable for people, it could have been useful to see baselines to compare against. Maybe measuring enjoyment/performance for people playing against a robot moving randomly, rather than with training, to show that the robot was able to learn a better policy than random movements? Or compare enjoyment/human gameplay performance against playing the Fencing Game with another person instead of a robot? 

Overall, this paper is interesting and makes a strong argument for competitive robots that learn how to play games in a challenging way
summary_of_recommendation: I am recommending a weak accept because this paper is interesting and makes a strong argument for competitive robots that learn how to play games in a challenging way. However, it does not have baselines to compare against to show that the training was necessary to challenge the users. Overall I think this is interesting, well written work.


Edit post-rebuttal: The paper was updated during the rebuttal period to include a baseline comparison for the proposed system, which changed my assessment of the technical quality. This experimental addition was helpful to understanding the system, and has changed my recommendation to a strong accept.

issues: Issues to address during the author response/revision period:
-	Is there any data recorded during training, etc. to suggest that the robot’s training improved its gameplay and potentially human enjoyment over an untrained robot?
-	Was there a significant visually observable difference between the distinct gameplay styles?
-	What metrics were used to determine that the Fencing Game was played “sufficiently well” after two training sequences?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a novel human-robot interaction task — the Fencing Game — with a focus on building a system for this task that encourages healthy *human-robot competition*, an area that has historically been understudied in the HRI literature.

Generally, the goal of the system is to build a robot capable of “fencing” (simplified, safe, no real swords) with a human for encouraging physical exercise. In addition to defining the rules of this two-player zero-sum game in which a human needs to attempt to move their sword to a “target area” without contacting the robot’s sword (straightforward attack/defense style gameplay), this paper introduces a system for learning diverse robot policies for playing this game, trained in simulation via a variant of self-play, then transferred to the real world.

The system is evaluated with a user study that measures quantitative metrics like increase in users heart rate while playing the game (proxying the efficacy in a physical exercise setting) and average game score, as well as quantitative metrics like enjoyment, usefulness, ease of use, etc. (adapted from the standard set of technology acceptance model questions).

A crucial part of the proposed approach is building a “catalogue” of diverse robot policies for the humans to play against; the idea here is that diversity prevents humans from “overfitting” too much to a given policy and maximizes the sustained efficacy of such a system. However, I found this part of the paper hard to understand (when are different policies being shown to users during the study), and the results do not seem to back up the hypothesis, which I found interesting.
strengths_and_weaknesses: The biggest strength of this paper is the exploration of competitive human-robot interaction, and formalizing such a paradigm through this fencing game. I believe this is an interesting axis for human-robot interaction, and I hope to see more work like this in the future!

However, I have serious concerns about the experimental procedure, and weaknesses in the study, and the corresponding conclusions. Notably:
- There is no basis for comparing the proposed learned robot policy. I have no idea how good these human scores are, and looking at qualitative metrics like “ease of use” and “perceived usefulness” without any point of comparison makes it hard to draw any conclusions. 
- For the given setup of the game, it seems like a random heuristic that just waves the bat back and forth really quickly across the target area would provide a substantial challenge for humans, while giving them a similar workout (but perhaps at the cost of “enjoyment”). A random baseline where you inject some noise into this heuristic might actually considerably beat the learned baselines.
- The process for learning robot policies via self-play is interesting, but it’s again hard to see 1) why the multi-phase learning process is necessary (especially since the “warm-start” policy performs as well as the other diverse policies), and 2) how diversity in policies is actually being quantified/the guiding principle that defines this part of the learning process.

---
**Update Post-Rebuttal**

The authors did a lovely job addressing my concerns during the rebuttal. I still have some questions regarding the multi-phase learning policy and would love to see more of an ablation/intuitive explanation of the policy diversity, but the basis of comparison question has been answered with the heuristic. I am updating my score to a weak reject -- while I think the comparison is good, I think there's still room to expand the set of experiments to fully study these questions (I'm still curious about a fully random baseline, or a baseline that just moves the bat back and forth quickly).
summary_of_recommendation: In its current form, this paper tackles a well-motivated question in building a system for healthy human-robot competition. However, while the initial part of the paper is well-motivated, there are several concerns around the learning and evaluation involved in the proposed system that give me significant reservations.

Specifically:
- Lack of critical details and motivation around some of the learning algorithm design choices, and in how the user study was run.
- [Critical] Lack of a basis for comparison - the proposed system seems technically sound, but I have no idea how to evaluate its efficacy — without this, it’s hard to even argue that this particular game (the Fencing Game) is the *right* paradigm for studying human-robot competition. Notably, there are simple heuristic/random baselines that I think could easily be implemented that would outperform the existing system.

---
**Update Post-Rebuttal**

See above - I am pleased by the rebuttal and updating my score to a weak reject.
issues: There are several missing details in the learning section of the paper that also contribute to my recommendation above. Specifically:

- What is the continuous reward used to optimize the self-play policies (beyond game score)? Where does this come from?
- Could you provide more details about the “diversity” phase of training - are you just continuing training from the same “warm-start” point with different randomness? 
- How do you shortlist the 12 policies you add to the “diversity catalogue”
- How would the straightforward baseline (phase 1 training, but for longer) perform?
- How does the diverse class of 3 policies show up in the user study? Do you switch policies on a user within the 5 games they play, or do you do this across users. 
- Generally, would appreciate a lot more clarity around the user study setup!
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors develop a competitive fencing game to motivate humans to exercise while playing against a robot partner. To make the game interesting, the authors had to use state of the art robotics techniques such as sim2real and population-based training to produce a set of proficient and diverse policies. The experimental results showed that the human subjects enjoyed the game and did get some exercise, quantified by an increase in heart rate.
strengths_and_weaknesses: The authors provide a brief but very interesting literature review about how competition improves motivation for humans performing a variety of physical and cognitive tasks. I’m surprised that there hasn’t been more work in the field of competitive HRI. It seems like a natural fit to make physical therapy more fun for patients. 

The explanation of the scoring in section 2 was hard to understand, but watching the video linked in the caption of Figure 1 was very helpful. From what I understand, the video has a high pitched beep when the human scores points, and a low pitched beep when the robot scores (when the foam rods touch).

The authors develop a fencing game that is enjoyable and physically demanding. 
The robot opponent’s capabilities must be sufficient for the game to be challenging and interesting, so the authors explain the technical aspects of the development of the robot's fencing policy. The authors first use self-play in a Mujoco simulation to train a protagonist and antagonist. The authors use population based training to encourage diversity of policies. 
The authors select the 3 most distinctive policies out of the population using PCA analysis of trajectory roll-outs. The authors effectively bridge the Sim2Real transfer using the CMA-ES algorithm. A key part of the approach is that the robot switches between 3 different policies, so that the game is more challenging because the robot is less predictable.

The explanation of the experimental methods was very clear. It is good that the authors were able to obtain baseline heart rate data as a control for the study. Also, the test subjects had 5 minutes to learn about the scoring of the game before the start of scoring. 
The results of the survey are impressive, especially in the perceived enjoyment category of the survey. 
The finding about the changes in emotional and cognitive reactions was also very interesting. 
It was surprising that humans were not able to improve at the game over 5 episodes without the use of the diverse policies. Maybe the game is more challenging than expected. The human players did explore several strategies.

summary_of_recommendation: The paper is very well motivated, extremely novel, and has great technical execution and experimental methods & results.
issues: I noticed at least a few typos:
Line 33 heading should be “System Design and Implementation”
Line 202 heading could be “Experiments”
review_title: null

Meta-review: ### Final Meta-Review

Reviewers unanimously express their appreciation of the authors' effort and thoroughness in responding to their concerns during the discussion and revision phase. The authors' inclusion of a new user study to measure differences between the proposed learning-based approach and a hand-crafted policy has been well received. Nonetheless, reviewer DiX9 maintains some unresolved concerns, which the authors should endeavor to address in the final version of the paper, by discussing possible confounding factors and limitations of the user study.

All in all, the final manuscript has been significantly strengthened and, given the overall novelty and appeal of its ideas, I am happy to recommend it for inclusion in the CoRL proceedings and Oral Presentation at the conference.

### Original Meta-Review

The paper proposes a Fencing Game as a human-robot competition setup meant to encourage physical exercise. The reviewers agree that the paper is well written and explores an interesting direction, making a compelling argument for this type of application.

Two of the three reviewers express concerns with the experimental evaluation of the proposed system: in particular, without a control group of participants engaging with a baseline robot policy, it is unclear how much of the objective and subjective results should be attributed specifically to the proposed learned approach. This is issue is accentuated by the little justification provided for some of the (presumably heuristic) design choices, as well as some missing technical details. 

The authors should thoroughly address the reviewers' concerns and questions during the response period, especially those regarding experimental evaluation and design decisions.
Decision: Accept (Oral)