Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach to adapting control parameters in a prosthetic limb using a multi-agent Reinforcement Learning design that considers the human and the prosthetic as collaborating agents with the goal of achieving a symmetric gait pattern. 
The design is evaluated and compared in a number of simulation experiments.
strengths_and_weaknesses: The control of prosthetics in a way that is natural and does not put increased burden on the human is can have significant impact on the
life is amputees. Moreover, learning collaborative control tasks with humans in general is an important field. The paper is generally well written and lays out the rationale for the approach well.
The main weakness of the paper is in the experimental evaluation which, as it is done in simulation with a ver simple (and beneficial to the approach) human model. In particular it assumes that the human learns and adapts in the same way as the prosthetic (using the partial gradient of the prescribed control model with respect to the joined utility). This seems to remove some of the rationale of the interactive multi-agent approach that is to react to and address the human’s reactions to the behavior of the prosthetic, which, in turn, is likely based in past experiences and existing control schemes in the human. A discussion as to the assumptions in the simulation (especially with respect to the human model) both in terms of the utility used - don’t humans likely have more complex utilities?- and the policy- a human would start with a strongly pre-trained system with expectations about the other limb - or , better, inclusion of experiments with pre-trained human models that reflect likely human reactions to “un-natural” controls especially early on (and an evaluation of the difference this makes on learning speed and performance) would be very useful.
Also, it would be useful if the authors would include a comparison of the learned impedance parameter functions for the 3 scenarios and the base performance of the flat ground parameters in the robustness scenarios to give the reader an idea whether the approach would have to be trained for all terrain types separately (and maybe ultimately require a “terrain classifier”) or whether general parameters for variable terrains can be learned.
Also, to better understand the generality of the approach across walking scenarios, a discussion of whether and how it could be adapted to settings (like turning) where symmetry is not the main criterion would be useful.
questions: Some questions arise regarding the degree to which the evaluations are realistic:
To what degree is the simulation model for the human realistic ?
To what degree do the learned impedance parameters generalize to slight variations of the terrain ?
Is it realistic to assume that the human has the same objective to achieve symmetric patterns ?

To what degree can the approach be generalized to other non-straight walking tasks?
limitations: The paper addresses social impacts and considerations and discusses limitations in the conclusions. However, it would have been nice to see more discussion regarding the limitations (and potential ways to address them) earlier in the paper, especially in the experiments section to allow the reader a clearer picture regarding what the results show (and what they don’t capture).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies the problem of controlling robotic lower limb prosthesis for achieving better human walking assistance. The main idea is a formulation of multi-agent RL, where one agent represents the human controller and the other represents the robotic prosthesis. The robotic controller has access to some information about the human state and controller, which helps it better assist the human walking. To optimize the controllers, a Q-learning-based algorithm is deployed. The proposed method is evaluated in the OpenSim simulation environment and compared to alternative multi-agent RL algorithms. An ablation study is also carried out to investigate the impact of including human controller action as observation for the robotic controller.
strengths_and_weaknesses: Strengths:
- The paper studies an important and practical problem of controller robotic prosthesis for human motion assistance.
- The idea of multi-agent RL training for the robotic controller seems reasonable.
- The proposed formulation obtained promising learning performance in a simulated task.

Weaknesses:
- The technical contribution of the paper is not very clear. The main innovation seems to be the formulation of the observation and action spaces for the human and robotic agents and how the robot agent can observe certain information about the human. However, without evaluation on real humans, it’s hard to tell how realistic these assumptions are.
- The proposed formulation seems specific to the setup: if the task becomes walking up/down stairs, or traversing stepping stones, having desired and commanded velocity from the human controller might be insufficient.

questions: - During the walking, who decides the desired walking speed? If human determines that, how does the robot have access to it?
- How well does the method generalize to unseen human behavior? For example, if the thresholds in the human controller state machine is varied during testing.
- In the ablation it appears that without human controller input the model learns slower, but converges to a similar level of performance. Some discussion on this would be useful to better understand the impact of including the additional information.
- Human users can adapt their behavior to the prosthesis device. To account for this, one need to either model the adaptation process, or obtain a model that is robust to this. How would the proposed method handle that?
- How does the proposed method compare to a biomechanical engineering-based approach in terms of performance? 
limitations: The current discussion on limitations of the method is good in general.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents an application for human-robot collaboration and shared autonomy — the task of robotic prosthesis for symmetrical walking — which has not been studied extensively using RL. The authors propose a collaborative MARL framework to study the problem by defining a state-space problem and joint cost function, and show that an actor-critic policy can jointly optimize this problem. Compared to some relevant baselines like COMA and MADDPG, the proposed agent performs better in terms of learning a stable gait. Useful ablation analysis justifies some of the design decisions made.

strengths_and_weaknesses: _Heads up_: I am not familiar with the space of human-robot prosthesis, and my review is from the perspective of a general robotics/HRI/ML researcher. A quick skim of the space suggests some prior work in using RL for limb/knee prosthesis, but the proposed collaborative formulation seems novel to me.

Strengths
---
- The paper does a great job of describing the problem of human-robot prosthesis to an outsider (me), and it was possible to follow along the terminology and problem formulation with some effort. It’s always hard to write an accessible paper bridging communities and this paper does a good job of that.
- The experiments seem very well designed and answer specific questions I have as a reader. The ablation and reliability studies were particularly useful in convincing me that (i) the cost function is doing what it should, (ii) the human input is actually being used in a non-trivial manner, and (iii) the method can adapt to more general target signals (velocity, slope etc.). Learning curves alone do not capture this story, and the analysis included by the authors seems really comprehensive and useful.
*** 
Weaknesses
---
- [Interpreting Results] From the quantitative results in the paper, it is very hard to interpret what is a “good enough” error/symmetry. Is the lowest always better, or is there a point below which it does not matter and should be treated as the “target”. It would really help to include some qualitative result denoting what it means to “do better” in the task, e.g. stability of learned gaits, or energy consumption estimate etc. Also, the learning curves seem to only denote the training curves and not the validation curves — can the authors provide validation curves instead/alongside to convey the full picture? Training loss going down does not capture the entire process.
- [Implementation Details] I found the paper severely lacking in implementation details, and in its current shape, the paper is not reproducible. Beyond providing the form of the agent and objective, the paper has no information on the agent architectures used, training algorithm, optimizer, amount of training time etc.
- [Related Work: Shared Autonomy] The discussion on related work seems to miss a very important body of research in shared autonomy [1, 2, 3, to name a few…]. The task of shared autonomy very closely resembles the problem setup in the paper — human input combined with a (semi-)autonomous agent to achieve a common goal — and should be discussed when positioning the paper. While the proposed approach is shown to be better than other MARL baselines, it would also benefit from a comparison with a more shared autonomy-esque viewpoint to the problem — seeing an empirical evaluation against this, or a discussion of why this is not feasible, would really strengthen the paper.
***
References
---
[1] Nikolaidis and Shah, “Human-Robot Cross-Training: Computational Formulation, Modeling and Evaluation of a Human Team Training Strategy” (2013) 

[2] Javdani, Srinivasa, and Bagnell, “Shared Autonomy via Hindsight Optimization” (2015) 

[3] Reddy, Dragan, and Levine, “Shared Autonomy via Deep Reinforcement Learning” (2018) 

questions: Drawing from the discussion of weaknesses above, here are some clarifications/suggestions that would help me re-evaluate the paper after authors’ response. I am more than willing to engage with the authors and modify my review score contingent on the author discussion phase.
1. Improving interpretability of results, with a better metric or a qualitative visualization/analysis.
2. Adding implementation details, or ideally, releasing the code with the paper.
3. A discussion on positioning wrt shared autonomy; ideally, added comparison against some existing approach(es).
4. [Line 355] What are some other, more natural, objectives that may be used for providing shared goals in the proposed cMARL framework for robot prosthesis? Beyond the task of prosthesis, what broader set of collaborative objectives in HRI can the proposed method be applied for? A discussion on this would greatly improve how the broader community receives and builds on this work.

--------

After a productive round of responses from the authors, I have updated my score to recommend acceptance.
limitations: 

- It does not seem that the implementation/code is shared in the supplemental material. The data being proprietary is understandable, but that is not a sufficient reason to not release the code. The paper is also sparse in implementation details of cMARL, as well as the baselines, and it may be really difficult for people in sibling disciplines of HRI to draw useful insights and quickly try out the proposed algorithm.
- I could not find an explicit discussions on societal impacts of the work. While the immediate impacts of the work are not particularly troubling, I believe any paper on HRI, esp. involving prosthesis, should include a discussion on the potential negative implications and a discussion of how the proposed algorithms may be susceptible to malicious attacks. While this is not a criticism/limitation of the research proposed (and does not factor into my review), it helps people outside the community contextualize the research and its potential limitations.
review_title: null

Meta-review: metareview: This paper present an approach for control of prosthesis under a novel collaborative multi-agent formulation. The approach is demonstrated in simulation. 

All the reviewers agree that the paper contribution is novel and interesting.
The reviewers also provided several suggestions on how to improve the manuscript and pointed out the limitations intrinsic with evaluating the approach only in simulation.
Evaluating the approach in the real-world seems a natural and desirable next step.

I invite the authors to carefully integrate all the feedback received and, in particular, better highlight limitations and societal impact.

Personal comment: The references would benefit from some work: some of the references are poorly formatted; [30] and [31] cite the same paper; and finally you are not really citing 110 papers in the main text, so I think you might have forgot a \nocite in the code.
confidence: Less certain
Decision: Accept