Human-Robotic Prosthesis as Collaborating Agents for Symmetrical Walking
Authors: Ruofan Wu
Junmin Zhong
Brent Abraham Wallace
Xiang Gao
He Huang
Jennie Si
Venue: NeurIPS 2022 Accept (2022)
Keywords: collaborative multi-agent reinforcement learning
continuous control
human-robotic prosthesis collaboration
real world AI application

Abstract: This is the first attempt at considering human influence in the reinforcement learning control of a robotic lower limb prosthesis toward symmetrical walking in real world situations. We propose a collaborative multi-agent reinforcement learning (cMARL) solution framework for this highly complex and challenging human-prosthesis collaboration (HPC) problem. The design of an automatic controller of the robot within the HPC context is based on accessible physical features or measurements that are known to affect walking performance. Comparisons are made with the current state-of-the-art robot control designs, which are single-agent based, as well as existing MARL solution approaches tailored to the problem, including multi-agent deep deterministic policy gradient (MADDPG) and  counterfactual multi-agent policy gradient (COMA).  Results show that, when compared to these approaches, treating the human and robot as coupled agents and using estimated human adaption in robot control design can achieve lower stage cost, peak error, and symmetry value to ensure better human walking performance. Additionally, our approach accelerates learning of walking tasks and increases learning success rate. The proposed framework can potentially be further developed to examine how human and robotic lower limb prosthesis interact, an area that little is known about. Advancing cMARL toward real world applications such as HPC for normative walking sets a good example of how AI can positively impact on people’s lives. 

Human-Robotic Prosthesis as Collaborating Agents for Symmetrical Walking
Ruofan Wu Arizona State University
Junmin Zhong Arizona State University
Brent Abraham Wallace Arizona State University
Xiang Gao Arizona State University
He Huang North Carolina State University
Jennie Si Arizona State University
Abstract
This the first attempt at considering human influence in the reinforcement learn- ing control of a robotic lower limb prosthesis toward symmetrical walking in real world situations. We propose a collaborative multi-agent reinforcement learning (cMARL) solution framework for this highly complex and challenging human- prosthesis collaboration (HPC) problem. The design of an automatic controller of the robot within the HPC context is based accessible physical features mea- surements that are known to affect walking performance. Comparisons are made with the current state-of tfate-of-the-as robot control designs, which are single-agent based, as well as existing MARL solution approaches tailored to the problem, including multi-agent deep deterministic policy gradient (MADDPG) and counterfactual multi-agent policy gradient (COMA). Results show that, when compared to these approaches, treating the human and robot as coupled agents and using an estimated human adaption in robot control design can achieve lower stage cost, peak error, and improved symmetry to ensure better human walking performance. Addition- ally, our approach accelerates learning of walking tasks and increases learning success rate. The proposed framework can potentially be further developed examine how human and robotic lower limb prosthesis interact, area that little is known about. Advancing cMARL toward real world applications such as HPC for normative walking sets good example of how Al can positively impact on people's 's lives.
Introduction
The concept, design, and applications of human-robot cooperation have advanced rapidly due to new demands in Al-enabled applications fueled by powerful deep learning and reinforcement learning algorithms 2 B.. Human-robot collaboration can take on a variety forms depending on tasks to be solved, how information is shared [122, and the nature of interaction collaborative tasks may include picking up or carrying objects together 4 5 cooperating on a
Examples of complex
production line, in which cases robot learn to imitate human demonstrations
[6 7 g Other
application scenarios may include intermittent robotic correction of human driving, or vice versa [9]. In essence, most of these recent studies involve interactions between a human and robot in way such that there is either space between the agents, or there is time for predictive counter measures interfere. By contrast, in the HPC problem the human and robot agents are physically coupled together, and there is often little time for the human agent to react to prevent from falling or injury.
"qqual Contribution. Coorespponding author: si 1asaasu.ee edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

As result, innovative solution frameworks are needed to first address the design, analysis, and testing of complex control systems to restore locomotion for amputees. Based on these platforms, human- robot interactions can be studied, and fundamental issues such as user preference and prosthesis embodiment can be examined. The knowledge in turn, will further help design seamless automatic robot control systems. As such, challenges arising from controlling a wearable robotic lower limb prosthesis to meet the human user's s needs are unique, and in some aspects, these challenges are greater than what have been studied [10
13)
State-of-the automatic control of robotic prosthesis has been reported recently with successful human subject testing [14, [5 16 [7]. These approaches are single agent- reinforcement learning controls, specific tasks such as level ground and ramp walking were accomplished via designer prescribed or specified robotic joint movement profiles [14] [5]. A most recent progress in single-agent based reinforcement learning solution demonstrated that human amputee subject can perform level ground and ramp walking with the robot controller aimed at mimicking the intact joint movement 117. Note, however, that all previous results have not directly considered human influence on the human-robot system performance in tuning the robotic prosthesis controller, a phenomena that has direct impact not only on restoring walking but also on human health [[8]
Achieving normative walking is fundamentally a real time control problem that involves continuous states and continuous controls of the numan-robor system. Continuous state and control problem has received great attention. Approaches based on (deep) reinforcement learning have shown promise to substantively address real world applications. Several algorithms, such as Deep Deterministic Policy Gradient (DDPG) [19]. Proximal Policy Optimization (PPO) (20). Soft Actor-Critic (SAC) 21), and Twin Delayed DDPG (TD3) [22], have demonstrated success with solving complex control problems. For example, in simulated human locomotion control (33 24], deep reinforcement learning solved over 20 independent control signals to facilitate a humanoid robot to achieve different walking tasks. Additional single agent-based continuous control has also demonstrated promise in engineering applications such as stabilization, tracking, and aeconfiguring control of Apache helicopters 25 26
227, stabilization and control of large power grids [28 29 30), robotic manipulation and locomotion
via MuJoCo and OpenAI Gym 31 32). and wearable robots with human in the loop [4
15
16 17
While these works are encouraging toward solving realistic single-agent continuous control problems, not obvious how they can directly address the multi-agent human-robot normative walking problem as needed when we consider human influence in the robot control design. From a physical human-robot interaction (pHRI) perspective, robotic upper-limb control has undergone intense development, especially in the realms of patient rehabilitation [33] and industrial applications [34]. However, this type of pHRI problems differ fundamentally from the human-robot walking problem, as their control target usually consists of well-defined end points generated by a decoupled trajectory generation exosystem [35. On the other hand, human-robo. robot walking tasks are difficult to associate with end point task goal due to tight dynamic coupling between the human and the robotic lower limb, and many factors can affect the human's ' performance goal. As such, even though multi-agent reinforcement learning (MARL) control is a natural candidate to address our HPC challenge, feasible solution is yet to be developed.
2 Related Work and Challenges
Single agent RL for automatic control of robotic limb. Simulation, Most ofate-of-the-art RL control design approaches to enable continuous human-robot walking are single-agent based. Important milestones have been achieved by two major classes of RL algorithms: actor-critic algorithms including direct heuristic dynamic programming (dHDP) [36] 37, 38 33, and variants of policy iteration algorithms such as flexible policy iteration (FPI) [40]. Both types of single- agent based control algorithms were developed and demonstrated in simulated environments first, Human Tests. Then these algorithms were tested on human subjects walking with robotic knee prosthesis [5] 16] 41 Note that all of the above methodologies and tests used designer-preseribed robot joint movement profiles generated a priori for the specific subjects and walking tasks. In real-world use scenarios, joint movement profiles evolve dynamically in real-time to accommodate internal human walking objectives [1 A more recent single-agent RL control work [17] replaced besigner--presrrbed joint movement profiles with the intact joint motion. Note also that, none of the above results have directly taken into account human influence on human-robot walking in the control design, novel contribution of this work. Multi-agent reinforcement learning (MARL).

---

A general cMARL problem usually cannot be transformed to an equivalent single-agent problem. as additional agent(s) and their control policies introduce uncertainties and/or -tatttttionaitty to the environment 43 Partial observability is common in cMARI. problems, which are usually addressed by the decentralized POMDP framework. Centralized training decentralized execution (CTDE) is popular paradigm to address multi-agent coordination problems. A central feature of most CTDE approaches is factorization of the joint state-action value function into individual utility functions. MADDPG [44] is a popular CTDE-based method to solve both cooperative and competitive MARL problems. COMA 45 is another popular MARL algorithm which utilizes a single centralized critic by using global state information and actions of all agents. We therefore use them in benchmark evaluations. Shared Autonomy (SA). The human- camman-roosthsii problem under our consideration falls into the area of physical human-robo interaction (pHRI) as the human and the robotic prosthesis are physically coupled at all times. It is not, however, the extensively studied pHRI problem archetype (e.g., cooperative object manipulation, human operating in a remote environment), wherein interactions are usually mediated by a third object. For similar reasons, our pHRI problem is different in several important aspects from the approaches the existing shared autonomy, such as cross-training [46). bounded memory adaptation [47), predict and blend [48], and model-free RL [49]. The wearable exoskeleton control problem seems relevant to our HPC problem. Yet, there are still fundamental differences. Recent exoskeleton approach also takes into account human- human-exoskeleton interacting effects [50] 5T. However, these exoskeletons mainly focus on end point performance of a foot or lower limb joints where user intent is quantified by estimating human joint torque or interactive torque. Thus.
system performance goal is for the robot to produce joint or endpoint trajectory [52 53 54]. For HPC problem we consider, there is no clear end point goal as there is in most existing current pHRI problems, including robotic upper limb that has been intensively addressed in literature. Therefore, how to define a performance goal for our HPC problem is a challenge. We address this study from existing limited yet proved knowledge 14 42 55). Modeling Challenges. The HPC walking problem involves two strongly coupled agents, the interacting dynamics of which are difficult or nearly impossible to describe by ordinary difference or differential equations in large part because there is no clear performance goal in HPC as in studied cases of shared autonomy. HPC may be affected by several factors, such as lower limb mechanics, inter- limb neuromechemical coupling, and physical structure of the human body including the lower limb [56] [7]] Physical and physiological differences in individual human subjects further complicates modeling and control design. Even though our knowledge of human motor control and motor learning in cases such as post-stroke or general loss of normative locomotion capabilities have expanded greatly [58]. little is known about how active robotic prosthesis, not traditional stick-type passive prosthesis, affects human and vice versa in walking tasks. This is because that the human-robotic prosthesis interacts continuously, the intricate human neurocontrol circuits including sensing, perception and feedback control, which are necessary to facilitate normative walking, is disrupted after amputation. Human Utility Challenges. Because of the reasons above, also because of no clear end point target as performance goals as in most studied shared autonomy cases, seamless collaboration between a robotic lower limb prosthesis serially attached its human user poses new issues that requires to be answered [10] [T]. In this study, we based on latest understanding on the HPC problem, to provide an innovative solution approach to account for human influence in HPC.
Human-robot collaborative tasks will continue to play an increasingly vital role in modern life, and existing single- and multi-agent control frameworks have only covered small subset of the problems. The contributions of this work are as follows. 1) We introduce an innovative approach automatically control robotic lower limb prosthesis by treating the human user as collaborating agent. We thus address a new challenge in the domain of shared autonomy problems. 2) To solve the control problem, we introduce new cMARL approach to solve our HPC problem, a problem that cannot be readily solved to satisfaction by existing MARL approaches such as COMA or MADDPG. 3) This the very first attempt in the field of wearable lower limb robots that human influence is explicitly considered in the robot control design.
Contributions.
3 Fundamental control problem statement
In powered lower limb prosthesis, the finite state impedance control (FS-IC) framework most frequently serves to provide intrinsic control of a robotic joint, i.e., it is the built-in controller from robotic prosthesis manufacturers. While position control is common in industrial robots, when a human is affixed to robotic joint, robot trajectory tracking using position control may preclude any
3

---

STr
STE
SWF
Qxx v,
goit cycle
69
Stited
up
πu
Intacl
Knee Angle
23
Z
STF
STE
SWE
Figure 1: Left: human gait and prosthesis FS-IC characteristics. Right: cMARL solution approach to the HPC problem.
dynamic interaction of the robot with its human user and the environment [59] This may cause an amputee to react to the awkwardness of the prosthesis rather than to interact constructively with [60] As FS-IC impedance control framework is considered to provide compliant control for human use of the robot, it expected to generate stable and predictable human-robot walking behavior, and thus, it is adopted in this study which aims for its controller designs at real world applications.
The FS-IC treats gait cycle, or step, as four consecutive gait phases (Figure L. Left): stance flexion (STF) as Phase 1, stance extension (STE) as Phase 2, swing flexion (SWF) as Phase 3, and swing extension (SWE) as Phase 4. In what follows, we design four individual controllers for each of the four phases. Since the four controllers are conceptualized similarly and designed using the same approach, for the sake of clarity we carry out subsequent discussion without explicitly referring to the specific phase numbers (but we emphasize that each of the four phases requires its own mndependently-trainee transition are provided in Appendix A.3
controller). Additional details on FS-IC such as phase detection and
At the k-th gait cycle, robot controller (solved by eMARL in this paper) is determine three impedance parameters, J = KK, B5, (0+)) TR3 under the FS-IC framework, representing stiffness K2, damping coefficient BR. and equilibrium position (0c) respectively. The prosthetic joint motor torque Tk then generated based on joint kinematics (knee joint angle 0 and angular velocity w) according to the following impedance control law,
T = K2 00- (0e,)) + BAW.
The control problem formulation requires automatically determining 12 control inputs or impedance control parameters for each of the 4 gait phases) for individual users aiming at walk in real world situations. The initial set of feasible baseline impedance control parameters lo can be obtained from manufacturers or rehabilitation clinies. Impedance updates take place according
I + uk:
where up 3 is to be determined from our proposed cMARL approach.
4 Method
The human user and the robotic limb are considered collaborating agents. Our cMARL solution approach toward human-robot symmetrical walking problem is formulated based physical features and measurements that have been shown affecting human-robot walking performance, and they are respectively available to the human and the robot, but not necessarily to each other.
State and control variables. Refer to Figure At k-th gait cycle, let of and Xg. respectively, represent the stance time (time of foot on the ground) and step length (length between two consecutive steps when toe touching the ground) the human intact leg. Similarly, Tf and X₁- respectively, for the prosthetic leg. Let Ark, Ad denote respectively the difference of stance time and step length,
ATR- The
T
Ad Xh X
(3)

---

We thus define human state variable 24 E RR as as = [ATk, AAa]".
Robot states include kinematic features determined from the knee movement profiles well as step length and stance time (or zt). For robot kinematic state variables, we extract gait kinematic features from knee motion profiles of both limbs (Figure I. Let Pj and Dj represent the knee angle and time duration of the intact limb, and similarly, PK and DI for the prosthetic limb. Let AP' and ADR denote the error of peak knee angles and the error of duration time between the intact and the robotic knees, respectively,
A = P P AD2 D Dj.-
(4)
We then define robot state Ta E RR to include the following four variables, the first two of which are shared between the human and the robot, 3& = [mr A A Pr; ADx]"
As such, human states are available to the robot, but subset of the robot state (the kinematic features) is not available to the human. This reasonably reflects practical situations in real life applications.
The robot control policy is state feedback law from robot state to robot impedance control parameters, namely, control 4g consists of increments to the impedance control parameters (Eq. 2
A AR, (A0+)
(5)
Determining human control policy which originates from human neurocontrol eircuit is daunting task, especially now that sensing, perception and feedback control circuits are interrupted after a lower limb amputation. While agents' policy solutions are solved from Bellman optimality- equations, a human cannot interpret and implement an MARL solution via their complicated neural circuits. But this is not our intention. We instead use the human control variable as an estimated input to the robot controller design, i.e., the robot is informed by an estimated human influence when they share the same goal of symmetrical walking.
As the very first step to demonstrate this idea, we consider the human user of the wearable robot intentionally or voluntarily walk at a reference step length, which may also be from an instructional feedback. For real life scenarios, such behavioral cues can change over time, task, or environment. But it important to validate such formulation of considering human influence in the robot control design. Toward this end, we consider a practical reference cue denoted as desired step length Ao We let Ui represent an endogenous control signal of the human which is function of human physical and mental states that involve activities ranging from neural level to muscular and joint level. Accordingly, we represent the endogenous human influence solved from the MARL design as a step length Xg. We therefore define human control as,
U/
Ao-
Including this estimated human control into the problem of robot control design, we take into account human influence on human-robot walking performance while they share the same symmetrical walking goal.
Symmetrical walking as shared task goal.
We consider the stage cost to be shared between the human and the robot.
U(Kk, Uk, u&) = R2. + Ro + ul R. + ph?
where R2 E R4x44 and R. E R3x3 are positive semi-definite weighting matrices, R2 and ju are positive weighting constants, and ha t un. In the above, the difference between actual human step length Aj. and reference Xg. denoted as 32 = A Ao is considered practical and available.
In this formulation, the shared control objective is represented in two ways. First, the robot kinematic state variables in 2g are match the intact knee, i.c., the robotic joint angle profile is to match that of the intact joint. Additionally, a walking symmetry measure is directly considered by the differences step length and stance time between human and robot, commonly used gait symmetry measures 61 62 63 The human is assumed to perceive the gait symmetry but do not have access to or understand robot kinematic data (peak knee deflection error APs and peak knee time error A DD). The works [14, 64] show from test subject data that human control and adaptation has direct influence on robot kinematics through dynamic interactions [59]. To account for this dynamic learning phenomenon, this framework penalizes the estimation error b4 U3 Up
X X
5

---

between human perceived difference in step length. An unexpected or undesired human action will increase the cost represented by h4s while diminishing h2 indicates that human imput that originated from the neuromuscular system matches actual human performance. Minimal controller energy expenditures are also included the cost structure.
The human-robot control objective to be solve by MARL as function of the state variables and the the controls is formulated as an infinite horizon, discounted cost Q(22, Uk uy) 7 vy. Uz) where 7 (0 < Y 1) is the discount factor for the infinite-horizn
problem.
Solutions to control actions. Let U2 Tu (zx) and ug nu (2x), where Ty and Tu are the human and the robot control policies solved from cMARL, respectively. Solving optimal control policies T and =. requires solving the optimal Q function that satisfies Bellman optimality equation:
(Irke Uky ug) U (1)) Uke (&)) + Q" (IR+11 T. (2++1), (f*++1))
(8)
When using a neural network-based actor-critic solution framework, we approximately solve the optimal control problem using the following iterative procedure (i is the iteration index),
Qi+1 (IFR, 0k: ux) (3rk, Uk, u4) ( No, (#k+1)) Fu; (1R++1)).
where Tr, (zx) arg min., Qi( uks ug), and Ta, (ax) = arg mina, Qi(TR, Uk, uk), and (z Uks uk) are iterative actor policies and iterative Q value function.
During training, the actor and critic back-propagate their respective squared error update their weights. The prediction error of actor k, Cau.k E R is,
1 (Qi (IR, Uk, mx:))
Eagyh Eng,k
(10)
2
The prediction error for the critic Ca is formulated based on the Bellman error,
Ec = U + +0Q T.. (2x+1), Tu, (xk+1)) Qi+1 (Ik, Uk. ug),
(11)
and the critic neural network is trained to minimize ec. 2
The optimal late-actionn tost-to-go function (xrk Uky u&) is approximated by critic neural network which learns the Q function by minimizing the Bellman error on the shared cost signal, not on a local cost signal for either the human or the robot the human and the robot physically coupled. We use our established direct heuristic dynamic programming (dHDP) algorithm [36] to solve this approximation dynamic programming problem. The critic neural network is three-layer MIP with 6 hidden units and uses linear activation function in the output layer. Therefore, we have the approximated cost to go value represented by:
(Ik, us) = We2,49 (WeLs [x] u "
(12)
where WA1, E R6x8 denotes the weight matrix between the input layer and the hidden layer, and W2,,, E RIx6 the weight matrix between the hidden layer and the output layer during the ith learning update. The weight updates of the hidden layer matrix Wa are according to
dea,
AW 2,; Le
(13)
aw c2
and the weight updates of the input layer matrix Wa1 are according to
dec
AWa lo
(14)
aw c1
where 4,
is the learning rate of the critic network.
Similar to the critic network, the actor networks for the human and the robot, respectively are three- layer MLP with 6 hidden units with hyperbolic activation function in the output layer to bound the action output. The same SGD optimizer (36] can be applied to the actor networks as well.
The two actors, u and respectively are
Uy = 4 (W 2,1 1 (W 1.2k))
(15)
02 = x (Wo.2. * 4 (Wa, (Wa))+2)))

---

where W E R6x4 Wa.1 E R W E R and W E R 1x6 are the weight matrices, and (p)) is the hyperbolic tangent activation function used in the hidden layer and the output layer. The weight updates of the hidden layer matrix Wa2, are according to
Dea k
AW 2,2
(16)
Dca...
AWa22. la
2.a
The weight updates of the input layer matrix Wa1,i are according
Dean & aWa, 1,
A Aaal 1, la
(17)
dcank
AW la
a.a
where La > 0 is the learning rate of the actor.
Further details the cMARL automatic control solution are provided in Appendix D
5
Experiment
We conduct design evaluations of the proposed eMARL for symmetrical walking using OpenSim, well-established open source biomechanical modeling tool for conducting biomechanics research and motor control science (65] The robot knee control is realized within an FS-IC framework. In real life, initial impedance parameters can be selected based on manufacturer and/or rehabilitation clinician' s recommendations. Similar care is given to OpenSim simulated walking yet in all evaluations, the initial impedance values vary from large range of settings for fair examination. As in (37], we enforce realistic safety constraints to prevent the human from stumbling or falling. Impedance parameters are reset to initial impedance values if any of the state variables exceed the safety bounds. The safety protocols followed in this work are fully described in Appendix AA.4
To make the simulations reflective of real world conditions, sensor and actuator noise data extracted from real human experimental testing sessions applied all the simulations in this study. Appendix A.2 provides the complete procedure followed of extracting noise data from experiments involving human subjects and injecting into all the simulations.
In this section, we provide results of a large set of simulation studies aiming at answering the following questions: 1) Does our eMARL solution framework provide better performance than state-of-the-an baselines, including MADDPG and COMA? 2) Does including human inffuence in robot control design accelerate learning and improve success rate of policy in comparison to single-agent based approach (wout/human)? 3) Is our cMARL applicable to different and realistic walking tasks? To provide answers, we show three sets of evaluations: benchmark, ablation and reliability, Benchmark and ablation evaluations are based on a level ground walking task with pace of Im/s. Reliability evaluations are based two new walking tasks: slope walking on 11.5 degree ramp and level ground walking at an increased pace of 1.12m/s.
Performance Criteria. In order to ensure that amputee subjects walk safely and continuously, we consider several performance metrics: 1) As an optimal control problem, the objective is to minimize state regulation cost (smaller is better). 2) Peak knee error can directly reflect amputee safety, preventing falling and stumbling (smaller is better). 3) Symmetry in walking can prevent secondary injury (closer to 0 is better). 4) Fast learning in terms of fewer tuning steps is practically important to amputees (fewer steps is better). 5) High success rate boosts amputees' confidence and hence walking performance (higher is better). Details of how the data was obtained can be found in Appendix E.2 The main evaluation results are presented in Table 1. The second value in each entry (i.e., after the symbol) represents the standard deviation of the performance metrics.
Benchmark study. MADDPG and COMA differ from our proposed cMARL approach toward symmetry walking. MARL problems can vary greatly, same expected of their solutions [66 67
To perform benchmark studies, we tailor MADDPG and COMA, respectively to our HPC problem. Details are provided in Appendix D
7

---

Table 1: Performance of the implemented algorithms in terms of the five performance criteria. The results of the perf-perf algorithm for each criterion are boldfaced
Performance
w/human
wout/human
COMA
MADDPG
Stage Cost
0.002 + 0.001
0.008 + 0.003
0,004 E 0.002
0.38 = 0.306
Peak Error
0.003 0.001
0.007 + 0.002
0.003 0.001
0.025 + 0.014
Symmetry
0.001 + 0.001
0.006 + 0.003
0.003 + 0.002
0.022 + 0.012
Converge Steps
97 88.3
187.5 + 105.2
136 + 121.8
Success Rate
0.7
0.58
0.5
0.8
whuman
0.6
Success Rate
0.4
wout/human
0.2
w/human
COMA
wout/human
COMA
0.0
100
200
300
50
100
150
200
250
Steps
Steps
Figure 2: The total number of steps needed reach convergence in training (left). The success rate
during evaluation (right).
Figure 2 shows that our cMARL solution has the best convergence profile and success rate over the baselines. Further results training and evaluation shown in F and which compare the environment sample efficiency as well as algorithm performances. To answer the first question based benchmarking, our cMARL solution outperforms the baselines both in terms of kinematic and symmetry measurements as shown in Figures 3 and 4 center and right panels for training and evaluation. respectively. Additionally, the left panels of Figures solution outperforms benchmarks with at least 30% less environment samples.
3 and show that our cMARL
wchmman
wout/human
COMA
MADDAG
1.0
0.10
0.06
0.05
Training Stage Cost
0.5
0.08
Symmerry
0.04
0.06
0.10
0.0
100
200
30g
Peak
0.03
0.04
0.02
0.05
0.02
0.01
0.00
0.00
100
200
300
Steps
Steps
Steps
Figure 3: Learning curves of stage cost during training (left), peak angle error (middle) and sym- metry of step length (right) for benchmark (w/human, COMA, MADDPG) and ablation (w/human, wout/human) studies. Each learning curve is averaged over 16 different random seeds and shaded by their respective 95% confidence interval.
Ablation study. gain insights on how human control influences human-robot walking performance, an ablation study is carried out with direct human influenced terms removed, including fgg and Uk.

---

0.10
0.06
wihuman COMA
0,08
0.85
Averqel Toall Cost
60
Symnetty
0.04
40
0,92
V0k
0.02
50
100
1.50
200
2.50
300
30
100
1.50
200
2.50
300
100 150
200
250
300
Steps
Steps
Stegs
Figure 4: Learning curves of averaged total cost (terms associated with human influence removed) during evaluation (left), peak angle error (middle) and symmetry of step length (right) for benchmark (w/human, COMA) and ablation (w/umann, wout/human) studies. Each (smoothed) learning curve is averaged over 5 different random seeds and shaded by their respective 95% confidence interval.
from the problem formulation. This only leaves the kinematics and symmetry measurements in the stage cost. Similar to the benchmark study, the ablation study has a training session and evaluation session. For comparable results, the total cost in evaluation is obtained U (z)" R2g, 2 u R. u. Figures and 4 show that with an estimated human control influence accounted for in the robot control design, our cMARL solution (the green curves) outperforms the one without it (orange curves). Treating human and robot collaborating agents toward a shared performance goal, our cMARL. solution approach achieves increased success rate and accelerated learning speed (Figure [ This result makes sense as an estimated human control provides predictive signal to the robot control which aims at duplicating the intact human joint movement. Additional information on the quality of estimated human control is given in Appendix B.2
Reliability study. To make the proposed cMARL method practical and useful in real life, we setup two new walking tasks: (1) slope walking (1 (15.5 degree ramp) and (2) walking at an increased pace (1.12m/s). They will result in different walking patterns from those used in the baseline study, and thus different knee joint profiles. For slope walking, knee flexion will be more pronounced during the stance phase since it walks inclined. In the case of faster pace, stance time will compressed. To carry out the tests, the same training procedure is used in benchmark and ablation studies.
Figure. shows the performance of cMARL during slope walking and increased pace walking tasks. Performance of the two new tasks follow the same trend as that of the level ground walking at nominal pace. These results again validate our design approach of using the intact knee movement trajectory as the target for the robotic knee to copy. By doing so, we have removed a major control design barrier in the way of performing different walking tasks by automatic control.
Training Stage Cost
0.10
0.10
0.06
11
Poak Error
Symmetry
walking
0.04
0.05
0.05
0.02
0.00
0.00
0.00
100
200
300
100
200
300
100
200
300
Steps
Steps
Steps
Training Stage Cost
0.10
0.06
pacz
Paak Eroor
Symmetry
walking
0.04
0.05
0.05
0.02
100
200
300
100
200
300
100
200
300
Steps
Steps
Steps
Figure 5: Learning curves of stage cost during training (Left), peak angle error (Middle) and symmetry of step length (Right) for different walking tasks (level ground walking increased pace and ramp walking). Each learning curve is averaged over 16 different random seeds and shaded by their respective 95% confidence interval.

---

Limitations of This Study. Here we conduct an in-depth iiaulattion-baseed analyses as an important first step to evaluate this novel cMARL solution to the HPC problem. Simulations are critically important as exploration of problem formulation, control algorithm design, and systematic evaluation are necessary to performed prior to human experiment due to factors such as human fatigue, human safety, human loss of interest/confidence caused by repeated trial-and-erron time spent, and significant cost associated with testing amputee subjects. However, our framework is still to be tested in human experiment. Based on several important works in the literature 37 40 68) 66], extensive simulation studies followed by real life human test studies 47 have proven a highly successful development procedure for human/robot control. This will be the next step of this study.
For scenarios in which the terrain or task has changed significantly, a task planner will become necessary. making the problem a planning and control problem (as opposed to automatic control, the focus of this work). This expanded automatic control algorithm must be extensively verified in simulation and then in human tests before it can be integrated into a real-world planning framework for daily use cases. Such planning frameworks constitute intended future works.
6 Conclusion and discussion
1) In the US, approximately 1.7 million people live with limb loss. The amputee population is expected double by 2050 as the population ages and incidence of dysvascular disease increases. As most lower limb amputees use prosthetic legs to restore basic bipedal locomotion, our solution to the prosthesis control problem can potentially help improve the function and quality of life of lower limb amputees. 2) In this work, we develop novel cMARL framework towards systematically integrating the human and robot as collaborative agents achieve normative walking toward solving real world problems. With reaching symmetric locomotion as shared control performance goal, we demonstrate improved walking performance. 3) Symmetry is selected as the shared goal for the collaborating agents because asymmetric walking has been linked to secondary health complications including back pain and osteoarthiritis [7] 72 73). Although human-robot walking performance goals are difficult to systematically catalogue, additional considerations such as embodiment of the robot into the human will considered future works. 4) By breaking apart the shared cost for the human and the robot (cf. Section 5. the symmetrical walking task is treated and evaluated by MADDPG and COMA, respectively. Simulation results show that the factorization- to address the human-robot problem. The observed performance issues with factorization likely stem from the intrinsic coupling between the human and robot agents. 5) While ensuring human user safety has been carefully considered during control design, additional analysis of important properties such as convergence of learning, (sub)optinality of control policy, and human-robot closed- llose stability is still needed for this framework. Encouragingly, previous related works [74 that these theoretical results are very likely to be provable.
CTDE paradigm struggles
75
76
77
indicate
7 Acknowledgments and Disclosure of Funding
This research was supported in part by NSF under grants 1563921, 1808752 and 2211740 for Si; grants 156454, 1808898 and 2211739 for Huang. Zhikai Yao participated in early stage discussion involving the formulation of shared performance goal.
References
[1] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 -09060900666606000000000 2020.
[2] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):3543353 2017.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):599-533 2015.
10

---

[4] Martin Lawitzky, Alexander Mörtl, and Sandra Hirche. Load sharing in human-robe co- operative manipulation. In 19th International Symposium Robot and Human Interactive Communication, pages 185-191. IEFE, 2010.
[5] Antoine Bussy, Pierre Gergondet, Abderrahmane Kheddar, François Keith, and André Crosnier. Proactive behavior of a humanoid robot in haptic transportation task with a human partner. In 2012 IEEE RO-MAN: The 2 IEEE International Symposium on Robot and Human Interactive Communication, pages 962-967. IEFE, 2012.
[6] Hongbo Wang and Kazuhiro Kosuge. Control of robot dancer for enhancing haptic human- robot interaction in waltz. IEEE transactions on haptics, 5(3):264-27)) 2012.
[7] David Vogt, Simon Stepputtis, Steve Grehl, Bernhard Jung, and Heni Ben Amor. A system for learning continuous human-robot interactions from human-human demonstrations. In 20/7 IEEE International Conference on Robotics and Automation (ICRA), pages 2882-2883 IEEE, 2017.
[8] Shuhei Ikemoto, Takashi Minato, and Hiroshi Ishiguro. Analysis of physical human-robin interaction for motor learning with physical help. Applied Bionics and Biomechanics, 5(4):213 223, 2008.
[9] Biao Ma, Yulong Liu, Xiaoxiang Na, Yahui Liu, and Yiyong Yang. A shared steering controller design based on steer-by-wire system considering human- human-machine machine goal consistency. Journal of the Franklin Institute, 356(8):4397-4419, 20119
[10] Dylan Losey, Craig G McDonald, Edoardo Battaglia, and Marcia K O'Malley. review of intent detection, arbitration, and communication aspects of shared control for physical huban-robot interaction. Applied Mechanics Reviews, 70(1), 2018.
[11] Thomas B Sheridan. Huban- 532, 2016.
interaction: innttnnacootittaiiiiiooo
status and challenges. Humannfccooss 58(4):525-
[12] Agostino De Santis, Bruno Siciliano, Alessandro De Luca, and Antonio Bicchi. An atlas of physical human-robot interaction. Mechanism and Muchine Theory, 43(3):253-270, 2008
[13] Panagiota Tsarouchi, Sotiris Makris, and George Chryssolouris. Hluman-robo interaction review and challenges on task planning and programming. International Journal of Computer Integrated Manufacturing, 29(8)996-993) 2016.
[14] Yue Wen, Minhan Li, Jennie Si, and He Huang. Wearer-prosthesis interaction for symmetrical gait: study enabled by reinforcement learning prosthesis control. IEEE transactions neural systems and rehabilitation engineering, 28(4):904-913, 2020.
[15] Minhan Li, Yue Wen, Xiang Gao, Jennie Si, and He Huang. Toward expedited impedance tuning of robotic prosthesis for personalized gait assistance by reinforcement learning control. IEEE Transactions on Robotics, 38(1):407-420, 2022.
[16] Minhan Li, Xiang Gao, Yue Wen, Jennie Si, and He Helen Huang. Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning. In 2019 International Conference on Robotics and Automation (ICRA), pages 2831-2833 IEEE, 2019.
[17] Ruofan Wu, Minhan Li, Zhikai Yao, Wentao Liu, Jennie Si, and He Huang. Reinforcement learning impedance control of a robotic prosthesis to coordinate with human intact knee motion. IEEE Robotics and Automation Letters, 7(3):7014-7020, 2022.
[18) Kenton R Kaufman, Serena Frittoli, and Carlo A Frigo. Gait asymmetry of transfemoral am- putees using mechanical and micropprocessor-ccomtrleee prosthetic knees. Clinical Biomechunics, 27(5):460-46) 2012.
[19] Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1509::1550000000 2015.
11

---

[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06447 2017.
[21] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprini arXiv: /8/2.0995, 2018.
[22] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMI.R 2018.
[23] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics- baseed character skills. ACM Transactions on Graphics (TOG), 37(():1-14, 2018.
[24] Seungmoon Song, Lukasz Kidziński, Xue Bin Peng, Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G Atkeson, and Scott L Delp. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. Journal of neuroengineering and rehabilitation, 18(1):1-17, 2021.
[25] Russell Enns and Jennie Si. Apache helicopter stabilization using neural dynamic programming. Journal of guidance, control, and dynamics, 25(1):19-25 2002.
[26] Russell Enns and Jennie Si. Helicopter trimming and tracking control using direct neural dynamic programming. IEEE transactions neural networks, 14(4):929 999,,999999 2003.
[27] Russell Enns and Jennie Si. Helicopter flight-control reconfiguration for main rotor actuator failures. Journal of guidance, control, and dynamics, 26(4):572-584, 2003.
[28] Chao Lu, Jennie Si, and Xiaorong Xie. Direct heuristic dynamic programming for damping oscillations in large power system. IEEE Transactions Systems, Man, and Cybernetics, Part B (Cybernetics)), 1804)):008-1000 2008.
[29] Wentao Guo, Feng Liu, Jennie Si, Dawei He, Ronald Harley, and Shengwei Mei. Approximate dynamic programming based supplementary reactive power control for dfig wind farm to enhance power system stability. Neurocomputing, 170:417-427, 2015.
[30] Jingyi Zhang, Yonghong Luo, Boya Wang, Chao Lu, Jennie Si, and Jie Song. Deep reinforce- ment learning for load shedding against short-term voltage instability in large power systems. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[31] Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. arXiv preprint arXiv: 100..007709999 2017.
[32] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[33] Qingsong Ai, Zemin Liu, Wei Meng, Quan Liu, and Sheng Q Xie. Machine learning in robot assisted upper limb rehabilitation: A focused review. IEEE Transactions on Cognitive and Developmental Systems, 2021.
[34] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint 204.::2004.08122 2021.
[35] Isidori. Nonlinear Control Systems: An Introduction. Springer Verlag, Berlin, Germany, 1985.
[36] Jennie and Yu-Tsung Wang- Online learning control by association and reinforcement. IEFE Transactions Neural networks, 12(2):264-276, 2001.
12

---

[37] Yue Wen, Jennie Si, Xiang Gao, Stephanie Huang, and He Helen Huang. A new powered lower limb prosthesis control framework based on adaptive dynamic programming. IEEE transactions on neural networks and learning systems. 28(9):2215-2230) 2016.
[38] Ruofan Wu, Zhikai Yao, Jennie Si, and He Helen Huang. Robotic knee tracking control to mimic the intact human knee profile based on actor-critic reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 911):1933, 2021.
[39] Xiang Gao, Jennie Si, Yue Wen, Minhan Li, and He Helen Huang. Knowledge-guided reinforee- ment learning control for robotic lower limb prosthesis. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 754-760. IEEE, 2020.
[40] Xiang Gao, Jennie Si, Yue Wen, Minhan Li, and He Huang. Reinforcement learning control of robotic knee with human-in-the-logg by flexible policy iteration, IEEE Transactions on Neural Networks and Learning Systems, 33(10):5873-555) 5887, 2022.
[41] Yue Wen, Jennie Si, Andrea Brandt, Xiang Gao, and He Huang. Online reinforcement learning control for the personalization of robotic knee prosthesis. IEEE transactions on cybernetics, 2019.
[42] Wentao Liu, Jummin Zhong, Ruofan Wu, Bretta L Fylstra, Jennie Si, and He Helen Huang. Inferring human-robot performance objectives during locomotion using inverse reinforcement learning and inverse optimal control. IEEE Robotics and Automation Letters, 2022.
[43] Lactitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforce- ment learners in cooperative markov games: survey regarding coordination problems. The Knowledge Engineering Review, 27(1):1-31, 2012.
[44] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed pooperative-competine environments. Advances in neural information processing systems, 30, 2017.
[45] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[46] Stefanos Nikolaidis and Julie Shah. Human-robol cross- cross-training training: computational formulation, modeling and evaluation of human team training strategy. In 2013 3 8th ACM/IEEE International Conference on Human-Robe Interaction (HRI), pages 33-40. IEEE, 2013.
[47] Stefanos Nikolaidis, Yu Xiang Zhu, David Hsu, and Siddhartha Srinivasa. Sluman-robor mutual adaptation in shared autonomy. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pages 294-302. IEEE, 2017.
[48] Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. Shared autonomy via hindsight optimization. Robotics science and systems: online proceedings, 2015, 2015.
[49] Siddharth Reddy, Anca Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv: 2018.
[50] Seyed Farokh Atashzar, Mahya Shahbazi, and Rajni V Patel. Haptics-enabled interactive neurorehabilitation mechatronics: classification, functionality, challenges and ongoing research. Mechatronics, 57:1-19, 2019.
[51] S Farokh Atashzar, Hsien- Hung Huang, Fulvia Del Duca, Etienne Burdet, and Dario Farina. Energetic passivity decoding of human hip joint for physical human-robot interaction. IEEE Robotics and Automation Letlers, 5(4):5953- 2020.
[52] Tingfang Yan, Marco Cempini, Calogero Maria Oddo, and Nicola Vitiello. Review of assistive strategies in powered lower-limb orthoses and exoskeletons. Robotics and Autonomous Systems, 64:120-136, 2015.
13

---

[53] Shraddha Srivastava, Pei-Chun Kao, Seok Hun Kim, Paul Stegall, Damiano Zanotto, Jill S Higginson, Sunil K Agrawal, and John Scholz. asseeeed robot-aided gait training improves walking function in individuals following stroke. IEEE Transactions Neural Systems and Rehabilitation Engineering, 23(6)956-963. 2014.
[54] Andres Martinez, Brian Lawson, Christina Durrough, and Michael Goldfarb. A velocity-field- based controller for assisting leg movement during walking with bilateral hip and knee lower limb exoskeleton. IEEE Transactions on Robotics, 35(2):30--36,, 2018.
[55] Wentao Liu, Ruofan Wu, Jennie Si, and He Huang. A new robotic knee impedance control parameter optimization method facilitated by inverse reinforcement learning. IEEE Robotics and Automation Letters, 7(4):10882- 10889, 2022.
[56] Charles T Leonard, RL. Craik, and CA Oatis. The neurophysiology of human locomotion. In Gait Analysis: Theory and Application. Mosby-Year, 1995.
[57] Gentaro Taga, Yoko Yamaguchi, and Hiroshi Shimizu. Self-organized control bipedal locomo- tion by neural oscillators in unpredictable environment. Biological cybernetics, 65(3):147-159 1991.
[58] Ryan T Roemmich and Amy J Bastian. Closing the loop: from motor neuroscience to neurore- habilitation. Annual review of neuroscience, 41:415 429, 2018.
[59] Neville Hogan. Impedance control: An approach to manipulation: Part i- 1985.
[60] Frank Sup, Amit Bohara, and Michael Goldfarb. Design and control of powered transfemoral prosthesis. The International journal of robotics research, 27(2):263-273, 2008.
[61] Slavka Viteckova, Patrik Kutilek, Zdenek Svoboda, Radim Krupicka, Jan Kauler, and Zoltan Szabo. Gait symmetry measures: A review of current and prospective methods. Biomedical Signal Processing and Control, 42:89-100, 2018.
[62] Antonio Nardone, Marco Godi, Margherita Grasso, Simone Guglielmetti, and Marco Schieppati. Stabilometry is predictor of gait performance in chronic hemiparetic stroke patients. Gait & Posture, 30(1):5-10, 2009.
[63] Chris Mizelle, Mary Rodgers, and Larry Forrester. Bilateral foot center of pressure measures predict hemiparetic gait velocity. Gait & Posture, 24(3):356-365 2006.
[64] Yue Wen, Andrea Brandt, Jennie Si, and He Helen Huang. Automatically customizing a powered knee prosthesis with human in the loop using adaptive dynamic programming. In 2017 International Symposium on Wearable Robotics and Rehabilitation (WeRob), pages 1-2. IEBE, 2017.
[65] Scott L Delp, Frank C Anderson, Allison S Arnold, Peter Loan, Ayman Habib, Chand T John, Eran Guendelman, and Darryl G Thelen. Opensim: open-source software to create and analyze dynamic simulations of movement. IEEE transactions on biomedical engineering, 54(11):1940-1955 2007.
[66] Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning, arXiv preprint 1908.0090...3999 2019.
[67] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
[68] Yue Wen, Ming Liu, Jennie Si, and He Helen Huang. Adaptive control of powered transfemoral prostheses based on adaptive dynamic programming. In 2016 38th Annual International Conference of the IEEE Engineering Medicine and Biology Society (EMBC), pages 5071- 5074. IEEF, 2016.
[69] Ruofan WI, Zhikai Yao, Jennie Si, and He Helen Huang. Robotic knee tracking control to mimic the intact human knee profile based on actor-critic reinforcement learning. JFFF/CAA Journal of Automatica Sinica, 91):11--0, 2022.
14

---

[70] Minhan Li, Yue Wen. Xiang Gao, Jennie Si, and He Huang. Toward expedited impedance tuning of robotic prosthesis for personalized gait assistance by reinforcement learning control. IEEF Transactions on Robotics, 2021.
[71] M Jason Highsmith, Casey R Andrews, Claire Millman, Ashley Fuller, Jason T Kahle, Tyler D Klenow, Katherine Lewis, Rachel Bradley, and John Orriola. Gait training interventions for lower extremity amputees: systematic literature review. Technology & Innovation, 18(2-
3):99-113, 2016.
[72] Alberto Esquenazi. Gait analysis in lower-limb amputation and prosthetic rehabilitation. Physi- cal Medicine and Rehabilitation Clinics, 25(1):153-167, 2014.
[73] George NS Marinakis. Interlimb symmetry traumatic unilateral transtibial amputees wearing two different prosthetic feet in the early rehabilitation stage. Journal of rehabilitation research and development, (144):881559) 2004.
[74] Feng Liu, Jian Sun, Jennie Si, Wentao Guo, and Shengwei Mei. boundedness result for the direct heuristic dynamic programming. Neural Networks, 32:229-235, 2012.
[75] Yury Sokolov, Robert Kozma, Ludmilla D Werbos, and Paul J Werbos. Complete stability analysis of heuristic approximate dynamic programming control design. Automatica, 59:9-18, 2015.
[76] Xiang Gao. Data-Efficient Reinforcement Learning Control of Robotic Lower-Limb Prosthesis With Human in the Loop. PhD thesis, Arizona State University, 2020.
[77] Qingtao Zhao, Jennie Si, and Jian Sun. Online reinforcement learning control by direct heuristic dynamic programming: From time-driven event-driven. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[78] SimTK. Tutorial 1 intro to musculoskeletal modeling.
[79] Daniel Jacobs. From the ground up: Building passive dynamic walker model, 2014.
[80] Mr P Kadaba, HK Ramakrishnan, and ME Wootten. Measurement of lower extremity kine- matics during level walking. Journal of orthopaedic research, 8(3):383-392, 1990.
[81] Ming Tan. Multi-agen reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[82] Peng Peng, Ying Wen, Yaodong Yang. Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent andirectionalll-cooriiii to play starcraft combat games. arXiv preprint arXiv: 1703. 10069, 2017.
nets: Emergence of human-level coordination in learning
[83] Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference on machine learning, pages 1146-1155. PMLR, 2017.
[84] Juanjuan Zhang, Picter Fiers, Kirby A Witte, Rachel W Jackson, Katherine L Poggensee, Christopher G Atkeson, and Steven H Collins. Human-in-the-lood optimization of exoskeleton assistance during walking. Science, $56(6344):1220-L11
2017.
[85] Stefanos Nikolaidis, David Hsu, and Siddhartha Srinivasa. collaborative tasks: Models and experiments. The International Journal of Robotics Research, 36(5-7):618 634, 2017.
Human-robot mutual adaptation in
[86] Jonathan Kofman, Xianghai Wu, Timothy Luu, and Siddharth Verma. Telcoperation a robot manipulator using vision- based human-robot interface. IEEE transactions industrial electronics, 52(6): 1219, 2005.
15


Reviews:
[{"review_id": "rbWG462PtEP", "reviewer": "Reviewer_bfd4", "summary": "The paper studies the problem of controlling robotic lower limb prosthesis for achieving better human walking assistance. The main idea is a formulation of multi-agent RL, where one agent represents the human controller and the other represents the robotic prosthesis. The robotic controller has access to some information about the human state and controller, which helps it better assist the human walking. To optimize the controllers, a Q-learning-based algorithm is deployed. The proposed method is evaluated in the OpenSim simulation environment and compared to alternative multi-agent RL algorithms. An ablation study is also carried out to investigate the impact of including human controller action as observation for the robotic controller.", "questions": "- During the walking, who decides the desired walking speed? If human determines that, how does the robot have access to it?\n- How well does the method generalize to unseen human behavior? For example, if the thresholds in the human controller state machine is varied during testing.\n- In the ablation it appears that without human controller input the model learns slower, but converges to a similar level of performance. Some discussion on this would be useful to better understand the impact of including the additional information.\n- Human users can adapt their behavior to the prosthesis device. To account for this, one need to either model the adaptation process, or obtain a model that is robust to this. How would the proposed method handle that?\n- How does the proposed method compare to a biomechanical engineering-based approach in terms of performance? ", "limitations": "The current discussion on limitations of the method is good in general. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper studies an important and practical problem of controller robotic prosthesis for human motion assistance.\n- The idea of multi-agent RL training for the robotic controller seems reasonable.\n- The proposed formulation obtained promising learning performance in a simulated task.\n\nWeaknesses:\n- The technical contribution of the paper is not very clear. The main innovation seems to be the formulation of the observation and action spaces for the human and robotic agents and how the robot agent can observe certain information about the human. However, without evaluation on real humans, it’s hard to tell how realistic these assumptions are.\n- The proposed formulation seems specific to the setup: if the task becomes walking up/down stairs, or traversing stepping stones, having desired and commanded velocity from the human controller might be insufficient.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C30D6QsFbqg", "reviewer": "Reviewer_Wv4V", "summary": "The paper presents an application for human-robot collaboration and shared autonomy — the task of robotic prosthesis for symmetrical walking — which has not been studied extensively using RL. The authors propose a collaborative MARL framework to study the problem by defining a state-space problem and joint cost function, and show that an actor-critic policy can jointly optimize this problem. Compared to some relevant baselines like COMA and MADDPG, the proposed agent performs better in terms of learning a stable gait. Useful ablation analysis justifies some of the design decisions made.\n", "questions": "Drawing from the discussion of weaknesses above, here are some clarifications/suggestions that would help me re-evaluate the paper after authors’ response. I am more than willing to engage with the authors and modify my review score contingent on the author discussion phase.\n1. Improving interpretability of results, with a better metric or a qualitative visualization/analysis.\n2. Adding implementation details, or ideally, releasing the code with the paper.\n3. A discussion on positioning wrt shared autonomy; ideally, added comparison against some existing approach(es).\n4. [Line 355] What are some other, more natural, objectives that may be used for providing shared goals in the proposed cMARL framework for robot prosthesis? Beyond the task of prosthesis, what broader set of collaborative objectives in HRI can the proposed method be applied for? A discussion on this would greatly improve how the broader community receives and builds on this work.\n\n--------\n\nAfter a productive round of responses from the authors, I have updated my score to recommend acceptance.", "limitations": "\n\n- It does not seem that the implementation/code is shared in the supplemental material. The data being proprietary is understandable, but that is not a sufficient reason to not release the code. The paper is also sparse in implementation details of cMARL, as well as the baselines, and it may be really difficult for people in sibling disciplines of HRI to draw useful insights and quickly try out the proposed algorithm.\n- I could not find an explicit discussions on societal impacts of the work. While the immediate impacts of the work are not particularly troubling, I believe any paper on HRI, esp. involving prosthesis, should include a discussion on the potential negative implications and a discussion of how the proposed algorithms may be susceptible to malicious attacks. While this is not a criticism/limitation of the research proposed (and does not factor into my review), it helps people outside the community contextualize the research and its potential limitations.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "_Heads up_: I am not familiar with the space of human-robot prosthesis, and my review is from the perspective of a general robotics/HRI/ML researcher. A quick skim of the space suggests some prior work in using RL for limb/knee prosthesis, but the proposed collaborative formulation seems novel to me.\n\nStrengths\n---\n- The paper does a great job of describing the problem of human-robot prosthesis to an outsider (me), and it was possible to follow along the terminology and problem formulation with some effort. It’s always hard to write an accessible paper bridging communities and this paper does a good job of that.\n- The experiments seem very well designed and answer specific questions I have as a reader. The ablation and reliability studies were particularly useful in convincing me that (i) the cost function is doing what it should, (ii) the human input is actually being used in a non-trivial manner, and (iii) the method can adapt to more general target signals (velocity, slope etc.). Learning curves alone do not capture this story, and the analysis included by the authors seems really comprehensive and useful.\n*** \nWeaknesses\n---\n- [Interpreting Results] From the quantitative results in the paper, it is very hard to interpret what is a “good enough” error/symmetry. Is the lowest always better, or is there a point below which it does not matter and should be treated as the “target”. It would really help to include some qualitative result denoting what it means to “do better” in the task, e.g. stability of learned gaits, or energy consumption estimate etc. Also, the learning curves seem to only denote the training curves and not the validation curves — can the authors provide validation curves instead/alongside to convey the full picture? Training loss going down does not capture the entire process.\n- [Implementation Details] I found the paper severely lacking in implementation details, and in its current shape, the paper is not reproducible. Beyond providing the form of the agent and objective, the paper has no information on the agent architectures used, training algorithm, optimizer, amount of training time etc.\n- [Related Work: Shared Autonomy] The discussion on related work seems to miss a very important body of research in shared autonomy [1, 2, 3, to name a few…]. The task of shared autonomy very closely resembles the problem setup in the paper — human input combined with a (semi-)autonomous agent to achieve a common goal — and should be discussed when positioning the paper. While the proposed approach is shown to be better than other MARL baselines, it would also benefit from a comparison with a more shared autonomy-esque viewpoint to the problem — seeing an empirical evaluation against this, or a discussion of why this is not feasible, would really strengthen the paper.\n***\nReferences\n---\n[1] Nikolaidis and Shah, “Human-Robot Cross-Training: Computational Formulation, Modeling and Evaluation of a Human Team Training Strategy” (2013) \n\n[2] Javdani, Srinivasa, and Bagnell, “Shared Autonomy via Hindsight Optimization” (2015) \n\n[3] Reddy, Dragan, and Levine, “Shared Autonomy via Deep Reinforcement Learning” (2018) \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-xEqeHEFwGz", "reviewer": "Reviewer_FF9f", "summary": "This paper presents an approach to adapting control parameters in a prosthetic limb using a multi-agent Reinforcement Learning design that considers the human and the prosthetic as collaborating agents with the goal of achieving a symmetric gait pattern. \nThe design is evaluated and compared in a number of simulation experiments.", "questions": "Some questions arise regarding the degree to which the evaluations are realistic:\nTo what degree is the simulation model for the human realistic ?\nTo what degree do the learned impedance parameters generalize to slight variations of the terrain ?\nIs it realistic to assume that the human has the same objective to achieve symmetric patterns ?\n\nTo what degree can the approach be generalized to other non-straight walking tasks?", "limitations": "The paper addresses social impacts and considerations and discusses limitations in the conclusions. However, it would have been nice to see more discussion regarding the limitations (and potential ways to address them) earlier in the paper, especially in the experiments section to allow the reader a clearer picture regarding what the results show (and what they don’t capture).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The control of prosthetics in a way that is natural and does not put increased burden on the human is can have significant impact on the\nlife is amputees. Moreover, learning collaborative control tasks with humans in general is an important field. The paper is generally well written and lays out the rationale for the approach well.\nThe main weakness of the paper is in the experimental evaluation which, as it is done in simulation with a ver simple (and beneficial to the approach) human model. In particular it assumes that the human learns and adapts in the same way as the prosthetic (using the partial gradient of the prescribed control model with respect to the joined utility). This seems to remove some of the rationale of the interactive multi-agent approach that is to react to and address the human’s reactions to the behavior of the prosthetic, which, in turn, is likely based in past experiences and existing control schemes in the human. A discussion as to the assumptions in the simulation (especially with respect to the human model) both in terms of the utility used - don’t humans likely have more complex utilities?- and the policy- a human would start with a strongly pre-trained system with expectations about the other limb - or , better, inclusion of experiments with pre-trained human models that reflect likely human reactions to “un-natural” controls especially early on (and an evaluation of the difference this makes on learning speed and performance) would be very useful.\nAlso, it would be useful if the authors would include a comparison of the learned impedance parameter functions for the 3 scenarios and the base performance of the flat ground parameters in the robustness scenarios to give the reader an idea whether the approach would have to be trained for all terrain types separately (and maybe ultimately require a “terrain classifier”) or whether general parameters for variable terrains can be learned.\nAlso, to better understand the generality of the approach across walking scenarios, a discussion of whether and how it could be adapted to settings (like turning) where symmetry is not the main criterion would be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept