Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: * The paper provides an extension on prior work with the cotribution of a new Quality diversity environment generation algorithm that receives feedback via surrogate models for a human and a robot that are trained on the generated environments
* In particular, providing a quality diversity algorithm with (a) feedback on the feasibility of the generated scenario, via corrections, and (b) the "true" quality of the scenario via simulation, the authors show that scenario generation quality is better, the algorithm is more sample efficient, and for complex domains, the algorithm is time efficient too.
review: **Strengths**

* The paper provides a novel and promising method to "Oz the Wizard" (Steinfeld, Aaron, Odest Chadwicke Jenkins, and Brian Scassellati. "The oz of wizard: simulating the human for interaction research." Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. 2009.)
* The paper builds on prior work in the field and yet seems to be making a significant contribution
* The paper surveys the literature on this topic adequately
* The paper's evaluations are appropriate and the results are great
* The paper characterizes its limitations appropriately

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I strongly recommend that this paper be included in the technical program.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper considers the problem of generating diverse/challenging scenarios for evaluating HRI tasks.  Previous works formulated the problem as a quality diversity (QD) problem which has been shown effective in a simple teleop HRI domain. However, previous methods require directly evaluating robot policies and human actions, which is expensive and limits the scalability to more complex tasks. The authors propose to instead train a model to predict task outcomes and use that model in place of expensive evaluations.

The authors evaluate their approach in two domains: a shared control teleoperation task and a shared workspace collaboration task. The reported experiments show the proposed approaches outperforming baseline QD algorithms in sample efficiency. Additionally, wall clock time improvement is shown on the collaboration task where evaluation is particularly expensive.
review: Strengths:
- In general, I found the paper to be well written and easy to follow.
- The proposed approach seems like an elegant combination of the ideas in [1] and [21] with sensible advances to support application to more complex HRI systems
- The empirical results presented in the paper are promising

Scope for improvement:
- The approach is only evaluated on two (relatively simple) domains. I would like to see the approach evaluated on a larger and more diverse set of domains to better support the claims that the algorithms generally apply to HRI systems.
- Because the paper is motivated by HRI systems, I would like to see some empirical evidence that the distribution of synthesized challenging human actions is similar to the distribution of challenging human actions that you would actually encounter “in the wild”. For the real world demo reported in the paper, a user is asked to reproduce scenarios extracted from the generated archives. This shows that the generated scenarios are reproducible, but it does not indicate that the generated scenarios cover situations that you would naturally encounter. I think the case for the proposed approach would be much stronger if the authors included some larger scale user study that shows that the types of difficult scenarios encountered naturally are actually covered by the distribution of synthesized scenarios.
- The paper references an appendix multiple times (e.g. “See App. A for the complete pseudocode”), but there is no appendix included in the submitted document. After downloading the supplementary material I see the appendix is included in the zip file. The other submissions I am reviewing have the appendix included in the document (which makes the reference links work). Please include the appendix in the submitted document.
- I personally find the diagram in Figure 2 to be difficult to follow. You might consider breaking it up into multiple diagrams. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The problem is well motivated and relevant. I think the proposed approach is in some ways an incremental work combining the ideas of [1] and [21]. However, the authors propose sensible advances to adapt the ideas for application to more complex HRI systems and the empirical results presented in the paper are promising. For these reasons, I think the paper is interesting and worth sharing with the community.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper's premise is that the evaluation of human-robot interactive systems (HRI) is difficult and costly. Therefore, the paper proposes a surrogate model to predict the behaviour of the human and the robot. The overall goal is to find diverse scenarios for training. The work is evaluated on two scenarios using a Kinova JACO robot arm.
review: HRI scenarios are highly relevant. The manuscript is clearly written and the work is properly motivated. The experiments show that the proposed methods provide more diverse scenarios compared to the chosen baseline methods. A small subset of the scenarios found by the proposed approach is evaluated on a real setup.

The concrete collaborative tasks in section 6 are not properly described. Without looking at Figure 5 and the accompanied video, it stays unclear what the actual task is. Even with these visual guides, it is unclear what an optimal solution to the HRI task would look like.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the graphs of the QD score over time show an improvement for every point in time for the proposed approach and therefore would support the author's claims, the real-world impact of this score is not clear to me. However, I am not an expert in this area and will adjust my verdict given the recommendations of reviewers more confident in this area.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This work addresses the challenge of generating datasets containing challenging scenarios for shared control and human-robot interaction. To tackle this issue, the authors propose two variations of surrogate-assisted scenario generation algorithms that formalize model-based quality diversity (QD) problems. Unlike previous approaches that either do not leverage a surrogate model [1,2] or do not consider continuous state and action spaces [3], the proposed methods combine these elements effectively. The proposed approach follows a hierarchical process, with the inner loop utilizing the QD algorithm to identify diverse and challenging scenarios based on the surrogate model, while the outer loop evaluates and adds these scenarios to the dataset used for surrogate model training. The experimental results demonstrate that both methods outperform baselines in terms of sample efficiency and wall-clock time.
review: I highlighted my comments below:

Major:
From the perspective of shared control and human-robot interaction (HRI), this work makes a valuable contribution to the field by addressing the creation of challenging datasets for meaningful algorithm testing and evaluation. While previous work primarily focused on shared autonomy (SA), extending it to HRI is a reasonable direction. However, I am curious about how the proposed scenario generation approach would adapt to more complex and difficult tasks. As mentioned by the authors, improving realism would require incorporating more complex environments, human actions, and sophisticated simulators. Nonetheless, for the current work, the low-dimensional workspace was discretized. Considering that more complex tasks could lead to the curse of dimensionality, it may potentially impact performance. Could you please elaborate further on this aspect?

Furthermore, the formulation of a quality diversity (QD) problem seems reasonable and yields notable results. However, in the current work, the optimization is performed over two user-defined measures, leading to convergence after a considerable amount of time (>12 hours). I am curious about the scalability of the proposed method in relation to the dimensionality of the given measures. Specifically, I would like to understand whether the proposed methods exhibit similar sample inefficiency as seen in Domain Randomization when faced with high-dimensional search spaces. Exploring how the proposed methods perform and handle increased dimensionality would provide valuable insights into their efficacy and potential limitations considering more complex HRI tasks. I kindly request the authors to address these questions and provide further clarification on the scalability of the proposed methods with respect to measure dimensionality.

Both proposed methods, DSAS and SAS, exhibit superior performance compared to the baselines in terms of sample efficiency and wall-clock time. However, when compared to prior work [1, 2], the introduction of the surrogate model increases the computational demand. As stated in line 198, the proposed methods were tested five times due to GPU usage constraints. It would be helpful if the authors could provide more insights into these limitations and discuss whether they could pose challenges in scenario generation for more complex HRI settings.
	
Minor:
The paper is well-written, and the visualizations are well-made. However, I would like to suggest that the authors consider adjusting the placement of the figures to avoid having two figures on the same page below each other. This adjustment could enhance the readability and overall quality of the paper. Additionally, it would be beneficial to revise the scale of Figure 5, as the images appear quite small and are challenging to perceive. By making these adjustments, the figures will be more accessible and contribute to a better understanding of the paper's content.

References:

[1] M. C. Fontaine and S. Nikolaidis. A quality diversity approach to automatically generating human-robot interaction scenarios in shared autonomy. In Robotics: Science and Systems, 2021

[2] M. C. Fontaine and S. Nikolaidis. Evaluating human–robot interaction algorithms in shared autonomy via quality diversity scenario generation. ACM Transactions on Human-Robot Interaction (THRI), 11(3):1–30, 2022.

[3] V. Bhatt, B. Tjanaka, M. C. Fontaine, and S. Nikolaidis. Deep surrogate assisted generation of environments. Conference on Neural Information Processing (NeurIPS), 2022
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is well-written and structured. Aside from some minor comments regarding the figures, my main concern revolves around the limitations of the algorithms in terms of scalability and computational demand. That being said, I recognize the value of the paper's contribution to the field of human-robot interaction (HRI) in terms of creating valuable scenarios for evaluating and benchmarking proposed methods. Therefore, I'm leaning towards accepting the paper, albeit weakly. I'm open to changing my decision to a full acceptance if the authors provide a compelling rebuttal that effectively addresses the concerns raised.
review_title: null

Meta-review: Summary of Paper:
The paper delves into generating diverse and challenging scenarios for evaluating Human-Robot Interaction (HRI) tasks. The authors extend previous work by introducing a new Quality Diversity environment generation algorithm, taking feedback from surrogate models trained on the generated environments for humans and robots. Their approach bypasses the expensive evaluations of direct robot policies and human actions that hampered the scalability of prior methods. Through evaluations in two domains, shared control teleoperation and shared workspace collaboration, the paper showcases enhanced sample efficiency and reduced wall clock time compared to baseline Quality Diversity algorithms.

Strengths:
- The proposed method offers an inventive approach to generating challenging datasets for HRI, rooted in the Quality Diversity algorithm, which receives feedback via surrogate models. This is particularly notable considering its grounding in the "Oz the Wizard" framework.
- An impressive build on existing research, the paper's contribution is significant, aptly drawing from and expanding upon the foundational literature.
- The paper is well organized, ensuring comprehension and presenting a clear narrative of the proposed method's efficacy, evidenced by empirical results that underscore its merits over QD algorithms in terms of sample efficiency and timing.
- Authors are transparent about the limitations of their work, which underscores the paper's credibility.

Areas for Improvement:
- Domain Limitations: The proposed approach, while promising, has been tested only on two domains. Expanding evaluation across more diverse and complex domains would bolster claims about the algorithm's broad applicability in HRI systems.
- Empirical Grounding in Real-World Scenarios: There's a palpable gap between the generated scenarios' reproducibility and their alignment with real-world HRI challenges. A more comprehensive user study would validate whether the synthesized scenarios encapsulate the range of challenges one would face in actual environments.
- Clarity and Documentation: Several aspects, such as a clearer description of concrete collaborative tasks and the inclusion of referenced appendices directly within the paper, could enhance the submission's clarity and completeness. Some visual elements, like Figure 2, could be reworked for better comprehension.
- Computational Concerns: While the paper boasts of improved sample efficiency and reduced time, there are implicit concerns about the computational demands, especially when juxtaposed with prior works. Addressing the scalability and computational constraints, especially in more intricate HRI contexts, would be beneficial.
- Dimensionality and Complexity: The current work's limited scope, in terms of its low-dimensional workspace, raises questions about the algorithm's performance as the complexity and dimensionality of tasks increase. Engaging with these concerns might provide a more comprehensive view of the approach's versatility.

Discussion during the rebuttal phase:
The authors responded effectively to the queries raised by the reviewers, resulting in a significant improvement in the reviewers' rating scores. The authors have indicated in the rebuttal phase that they will add further references to related works and provide more detailed data in the appendix for the camera-ready manuscript. Consequently, we can expect an even higher-quality paper in the end.

Recommendation:
The paper showcases a commendable effort in advancing HRI scenario generation. Given the paper's considerable strengths, especially its novel approach and strong empirical grounding, coupled with its addressable weaknesses, I recommend accepting the paper. However, it would benefit the authors to consider the discussion with reviewers, addressing key concerns to ensure the work's robustness and broader applicability.
Decision: Accept (Oral)