Soft Robot Assisted Human Normative Walking: Real Device Control via Reinforcement Learning Without a Simulator

Authors: Junmin Zhong, Emiliano Quinones Yumbla, Seyed Yousef Soltanian, Ruofan Wu, Wenlong Zhang, Jennie Si
Primary area: reinforcement learning
Keywords: reinforcement learning, Real world application, imitation learning, no simulator

Abstract:
This study offers an innovative solution approach to soft robot-assisted human walking. The controller design of the  soft robotic exosuit aims at assisting human normative walking with reduced human physical effort. Achieving such optimal interaction between the human and robot agents presents a key challenge to the robot control design due to a lack of robust model of the soft inflatable exosuit and its interaction dynamics with the human user.  Moreover, to maximize user comfort, the robot assistance  should be personalized to individual users. Toward this goal, we propose an offline to online based approach that is referred to as AIP, which stands for online Adaptation from an offline Imitating expert Policy. Our offline learning mimics human expert actions through real human walking demonstrations without robot assistance. The resulted policy is then used to initialize online reinforcement learning, the goal of which is to optimally personalize robot assistance. In addition to being fast and robust, our online actor-critic learning method also posseses important properties such as learning convergence, system stability, and solution optimality. We have successfully demonstrated our simple and robust solution framework for safe robot control on all four tested human participants.

Decision: reject
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729993807577

thread_type:
official_review

<thread_object>
creator: reviewer_jbgv

<content>
summary:
This paper presents a system to learn controllers for robot-assisted human walking. Offline learning is used from human walking data, and online adaptation is used to further fine-tune the controller. The proposed system is demonstrated to generate controllers for a soft robotics exosuit that can help reduce human walking effort.

strengths:
A system that can generate controllers that reduce human walking effort using a soft exosuit. This can impact the wider adoption of machine learning techniques on such systems.

A comprehensive analysis of the online adaption controller that shows significant improvement and personalized adaption.

weaknesses:
There is no video of the results.

There is a lack of comparison to existing work. What is the state of the art in the literature regarding effort reduction with an exoskeleton, either using the same robot or other exoskeletons?

There is not much innovation in terms of methodology. Are there any limitations of the current methods? What is preventing this from scaling up as mentioned in future work? Is it just simply putting more time into doing experiments?

questions:
How necessary is the initial offline phase? Can the system work by just using the online adaptation phase with some reasonable initial guess of the policy?

What is the theoretical maximum effort reduction of the assisted walk can achieve? Is the proposed method getting close to that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696034938

thread_type:
official_review

<thread_object>
creator: reviewer_thbp

<content>
summary:
This paper tackles the key problem of how to make the robot work effectively with humans without having a computer simulation to test things first - a significant challenge since most robotics work relies heavily on simulation before real-world testing.  The paper presents a systematic investigation of the problem and shows that their method can successfully train soft robots to aid walking without relying on simulation, opening new possibilities for human-robot interaction in assistive devices.

The authors propose a two-step approach they call AIP (Adaptation from Imitating expert Policy). First, they learn from normal human walking patterns without the robot's assistance. Then, they use this as a starting point for real-time learning where the robot adapts to each individual person's walking style. Their method focuses on improving data quality through a technique called RIIV (Reducing Intra-person and Inter-person Variation) and ensures safety through careful constraint design. The researchers validate their approach with both theoretical analysis and practical experiments on four subjects, demonstrating reduction in muscle effort while maintaining natural walking patterns.

----
*Post Rebuttal*: Updated the Score.

strengths:
- The paper presents a clear, well-structured methodology for direct learning on physical devices, with precise problem formulation and experimental validation.
    
- The data-driven approach (AIP) is simple yet effective, demonstrating successful learning without simulation through a combination of offline imitation and online adaptation.
    
- Strong empirical results with clear metrics (20% EMG reduction) and thorough ablation studies validating design choices.

weaknesses:
- The investigation lacks analysis of system degradation effects (e.g., soft actuator wear and tear) on the learned timing shifts discussed in Q3. This raises questions about long-term reliability.
    
- The paper would benefit from a baseline comparison showing human adaptation alone without robot learning, to isolate the benefits of the co-adaptation process.
    
- Limited discussion on the generalizability of the approach - specifically, which components are specific to soft exosuits (like RIIV data processing) versus which are generally applicable to human-robot learning systems.
    
- While the paper demonstrates successful learning without simulation, it doesn't fully address whether this approach eliminates the need for simulation in this domain. Have we achieved oracle performance? A discussion of the domain specific need of simulation and tradeoff to direct learning approaches would strengthen the contribution.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705046623

thread_type:
official_review

<thread_object>
creator: reviewer_1qdu

<content>
summary:
This paper introduces a method for controlling a soft robotic exosuit to assist human walking.  The key idea is to use reinforcement learning to directly learn a control policy from real-world interactions between the human and the exosuit.  This is achieved through an offline-to-online approach called Adaptation from an offline Imitating expert Policy. In the offline phase, the robot learns to imitate human walking patterns from expert demonstrations.  This provides an initial policy for the online phase, where the robot personalizes the assistance by fine-tuning the controller through direct interaction with the user.  The method is evaluated with four participants, showing that it can effectively reduce muscle effort during walking while maintaining a normal gait.

strengths:
1. The paper addresses the challenging problem of controlling soft exosuits for human assistance, which involves complex dynamics and human-robot interaction.
2. The method is designed to work directly in the real physical environment, without relying on a simulator, which is often difficult to construct accurately for soft robots and human-robot interaction. This makes the approach more practical and potentially more effective than sim-to-real methods.

weaknesses:
1. The method used in this paper is naive BC and policy gradient, however, there are other related offline to online frameworks such as [1], and the authors are encouraged to perform experiments with other methods.
2. The experiments are conducted with a small number of participants. It is unclear how well the method would generalize to a larger and more diverse population, including people with different walking patterns, or physical impairments.
3. Missing of hardware details.
4. Missing comparison with previous methods in exosuits, does reinforcement learning really help?
5. This article is more suitable for the robot conference or journal since it focuses on field robots rather than general methods.

[1] Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization, Kun LEI and Zhengmao He and Chenhao Lu and Kaizhe Hu and Yang Gao and Huazhe Xu, ICLR 2024

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731277718980

thread_type:
official_review

<thread_object>
creator: reviewer_yg6k

<content>
summary:
The authors present online Adaptation from an offline Imitating expert Policy (AIP) algorithm to learn a personalized and generalizable exoskeleton controller for human subjects. The knee-assist exoskeleton-suit helps reduce the effort required by the human user. The key to their approach is to first learn an offline behavior cloned policy which acts as a good starting point for online adaptation. To boost learning performance, the paper also introduces RIIV which reduces intra and inter-person variations thus improving data quality.

strengths:
The paper is written with sufficient details. The results presented are interesting and does justify the approach as a practical solution to robot assisted locomotion. Successfully getting online reinforcement learning to work on safety-critical human-robot interaction task is commendable.

weaknesses:
Including a supplementary video would be extremely useful for the reader to evaluate the qualitative performance of the policy. For example, noticing how gait looked like with the exoskeleton active. (I understand the privacy concerns with human subjects involved, perhaps blurring out the users could be an option?)
Adding a section on the limitations of the proposed approach would be valuable.

questions:
1) If I understand correctly, the policy only controls the onset timing and duration and not the actual air pressure of the robot. Does the air pressure used in the exoskeleton vary with different users? 
2) Related to the above question, the authors mention that data from only one person was used for collecting data for the offline imitation learning phase of the algorithm. Which among the 4 users contributed to this data? or was there a different person? if so, what were the anthropometric data look like? 
3) In section 3.5 - the following statement " Unlike traditional RL formulations that aim to
maximize the expected reward, in wearable robotics, the objective is to minimize the overall cost
over policy π" is not very clear, how are these two different. One can also reformulate a cost function to be a reward function. On a related note, the algorithm used for online adaptation direct heuristic dynamic
programming (dHDP) is not one of the more popular online RL algorithms, was any other approaches like PPO, SAC etc.. considered? If so, what are the pros and cons?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732500123145

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We greatly appreciate the Reviewers and the ACs for their time and feedback in reviewing our work. 

We are very pleased that the Reviewers have recognized the significance of AIP for its several important contributions:

1. "The paper presents a clear, well-structured methodology with precise problem
formulation and experimental validation. The data-driven approach (AIP) is simple yet effective, demonstrating successful learning without simulation. Strong empirical results with clear metrics (20\% EMG reduction) and thorough ablation studies validating design
choices." (Reviewer tHbP)

2. "This can impact the wider adoption of machine learning techniques on such systems. A comprehensive analysis of the online adaption controller that shows significant improvement and personalized adaption." (Reviewer jBgV)

3. "The results presented are interesting and does justify the approach as a
practical solution to robot assisted locomotion." (Reviewer YG6k)

4. "Successfully getting online reinforcement learning to work on safety-critical
human-robot interaction task is commendable." (Reviewer YG6k)

5. "The paper addresses challenging problem of controlling soft exosuits for human assistance, ..., the method work directly in the real physical environment, without relying on a simulator. This makes the approach more practical and potentially more effective than sim-to-real methods." (Reviewer 1QdU)

6. "This paper .. without having a computer simulation to test things first - a significant challenge since most robotics work relies heavily on simulation before real-world testing. The paper presents a systematic investigation of the problem .. opening new possibilities for human-robot interaction in assistive devices. The researchers validate their approach with both theoretical analysis and practical experiments." (Reviewer tHbP)


For the issues raised by the Reviewers, we have taken great care to ensure all of the Reviewers' valuable feedback has been thoroughly addressed. A detailed explanation is provided in the individual responses below to each and every point raised by the Reviewers, and we will revise the manuscript accordingly.

>On Supplemental Video 

First, we would like to emphasize that the results of the 4 participants included in the original submission were not cherry-picked. Namely, we used 4 participants in the study and we obtained 100\% learning success rate.

We have now included video clips of each of the 4 participants in supplementary materials. 

As REVIEWER 1QDU requested data from more participants, we are making our best effort to collect a new dataset on a new participant, with which it brings our total number of participants to 5, comparable or more than the typical size used in most recent and relevant studies. The new data and related results that we have managed to process are included in Fig. 9 and Table 8 in the Appendix of manuscript. Please also note that, again, we obtained 100\% learning success rate for all 5 participants involved in the study, and the new data is consistent with previous results of the 4 participants.

>On Difference between Exoskeletons and Exosuits

We notice on occasions that the terms Exoskeleton and Exosuits were used interchangeably in the reviews. Although they both refer to wearable lower limb  technologies in the context of this paper, and are designed to augment human locomotion, but they differ significantly in several ways, from their structure, fabrication, to sensing, especially actuation. Exoskeletons are motor-driven rigid devices that provide physical support (directly via motor torque) and power to enhance strength, endurance, or mobility. They are often made of materials like metal or hard plastic, resembling a mechanical frame that partially or fully encases the body or limbs [Shi, Di, et al. 2019]. In contrast, exosuits are soft, lightweight, and flexible systems that use textiles, cables, and air inflation/deflation mechanisms for actuation to provide assistive movement without rigid structures. In terms of control, almost all exoskeletons are motor-driven. As the exoskeletons as ridig mechanical systems can be modeled through Euler-Lagrange mechanics, that makes model-based control methods (including classical control approaches) applicable [Sun, Yuanxi, et al. 2022]. However as we discussed in the related work, such model-based approaches are ENTIRELY not feasible for soft exosuit. Please refer to our Related Work section, lines 812 to 819 for more information. Controlling exosuits faces  unique new challenges. That is perhaps of the reason that soft exosuit control results are few in the literature even though conceptually one may prefer soft wearable such as exosuits over rigid mechanical devices such as exoskeletons.
</content>
</thread_object>

</thread>


Metareview: