Reinforcement Learning Control of a Physical Robot Device for Assisted Human Walking without a Simulator
Authors: Junmin Zhong
Emiliano Quinones Yumbla
Seyed Yousef Soltanian
Ruofan Wu
Wenlong Zhang
Jennie Si
Venue: ICML 2025 poster (2025)
Keywords: Soft Robotics
Human robotic interaction
reinforcement learning without simulator
Primary area: applications->robotics

Abstract: This study presents an innovative reinforcement learning (RL) control approach to facilitate soft exosuit-assisted human walking. Our goal is to address the ongoing challenges in developing reliable RL-based methods for controlling physical devices. To overcome key obstacles—such as limited data, the absence of a simulator for human-robot interaction during walking, the need for low computational overhead in real-time deployment, and the demand for rapid adaptation to achieve personalized control while ensuring human safety—we propose an online Adaptation from an offline Imitating Expert Policy (AIP) approach. Our offline learning mimics human expert actions through real human walking demonstrations without robot assistance. The resulted policy is then used to initialize online actor-critic learning, the goal of which is to optimally personalize robot assistance. In addition to being fast and robust, our online RL method also posses important properties such as learning convergence, dynamic stability, and solution optimality. We have successfully demonstrated our simple
and robust framework for safe robot control on all five tested human participants, without selectively presenting results. The qualitative performance guarantees provided by our online RL, along with the consistent experimental validation of AIP control, represent the first demonstration of online adaptation for softsuit control personalization and serve as important evidence for the use of online RL in controlling a physical device to solve a real-life problem.

Reinforcement Learning Control of a Physical Robot Device for Assisted Human Walking without a Simulator
Junmin Zhong'
Emiliano Quinones Yumbla"
Seyed Yousef Soltanian
Ruofan Wu Wenlong Zhang 23
Jennie Si 13
Abstract
1. Introduction
This study presents an innovative reinforcement learning (RL) control approach to facilitate soft exosuit-assisted human walking. Our goal is to address the ongoing challenges in developing re- liable RL-based methods for controlling physi- cal devices. To overcome key obstacles- such as limited data, the absence of simulator for human-robot interaction during walking, the need for low computational overhead in real-time de- ployment, and the demand for rapid adaptation to achieve personalized control while ensuring human safety-we propose online Adaptation from offine Imitating Expert Policy (AIP) ap- proach. Our offline learning mimics human expert actions through real human walking demonstra- tions without robot assistance. The resulted pol- icy is then used to initialize online actor-critic learning, the goal of which is to optimally person- alize robot assistance. In addition to being fast and robust, our online RL method also posses im- portant properties such as learning convergence, dynamic stability, and solution optimality. We have successfully demonstrated our simple and robust framework for safe robot control on all five tested human participants, without selectively presenting results. The qualitative performance guarantees provided by our online RL, along with the consistent experimental validation of AIP con- trol, represent the first demonstration of online adaptation for softsuit control personalization and serve as important evidence for the use of online RL in controlling physical device solve a real-life problem.
Goal of this study. Wearable robots such as rigid exoskele- tons and soft exosuits have been extensively researched and have shown great promise for gait rehabilitation (Rodríguez- Fernández et al., 2021) and for assisting human walking to reduce physical efforts (Collins et al., 2015). Unlike rigid ex- oskeletons, soft, garment-like devices made from materials like silicone elastomers and fabrics provide more com- fortable, safer, and adaptable user experience (Granberry et al., 2017; Save et al., 2025; Yumbla et al., 2021). Yet, effectively controlling the wearable robots to seamlessly work with human users locomotion tasks remain major challenge. This may be why deployment of the promising wearable technology still have limited success in real-world deployment. Reinforcement learning points to two potential solutions: sim-to-real approach or direct design in the real physical environment. However, devising a near-perfect simulator for human-soft exosuit walking is exceptionally costly if all possible given the unpredictable human be- havior while wearing foreign robotic device, and the ubiq- uitous presence of noise, delay, and other uncertainty in the environment. Challenges of controlling a soft wearable exosuit. First off, soft inflatable exosuits lack a robust model of their dynamics (Polygerinos et al., 2015), not to mention modeling the interaction dynamics between the user and the robot, a necessary step in building a high fidelity simulator. These unique challenges stem from that the pneumatic dy- namics of the soft inflatable actuators (Joshi & Paik, 2021) are complicated in part by the nonlinear nature of soft actu- ators due to material properties and design geometry. The fabric-based actuators result in highly compliant behavior that enables high levels of deformations (Hasan et al., 2022). The manufacturing process of the actuators also introduces significant variations and uncertainties (Joshi & Paik, 2021). Further wear and tear the fabric only makes problem more complicated. Lacking reliable model simulator of the soft robot has made controlling soft inflatable exosuit more complex than a traditional rigid exoskeleton (Polygerinos et al., 2015). Unlike rigid exoskeletons where the assistive torque is determined by motor actuators and can be directly used as a control parameter, for soft inflatables, the torque is generated from two collaborative sources: the human and
'chool of Electrical, Computer and Energy Engineering, Ari- zona State University. Tempe, Arizona. "School of Manufactur- ing Systems and Networks, Arizona State University, Mesa, Ari- zona. *Corresponding Authors. Correspondence to: Jennie Si asQ.suuduu>, Wenlong Zhang Swwellong.zhagg @ds.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AIP Assisted Human Walking Without a Simulator
the exosuit, which is nearly impossible quantify. Ad- ditionally, the inflation/deflattion of the actuators typically introduces longer actuation delays than motor-actuated ex- oskeleton, a factor that potentially reduces stability margins in control system design. Contributions of this study. Our approach of online Adaptation from an offline Imitating expert Policy (AIP) provides holistic framework that en- ables efficient, effective, and direct physical device control in order to overcome the challenges of lacking a simulator, limited data, and robustness in real time deployment, Specif- ically, 1) instead of an algorithm-centric approach that has been mostly developed and deployed based on extensive simulations or vast amount of data in successful to- online robot RL. control (Levine et al., 2020; Kumar et al., 2020; Lee et al., 2022) we rely on a data-centric approach to account for environmental noise, delay, and other un- certainties. 2) The actor-critic online adaptation approach has shown its capability of successfully addressing distri- bution shift as we adapt offline learned policy to individual new users. Validations are provided by real experiments involving human users walking with the soft exosuit. Re- sults successfully show that soft exosuit control has assisted human normative walking with reduced human effort. 3) Additionally, we provide qualitative online learning perfor- mance assurances such as learning convergence, dynamic stability and human safety, and solution optimality. The qualitative performance guarantees, along with the consis- tent experimental validation of AIP control, represent the first demonstration of online adaptation for softsuit control personalization and serve as important evidence for the use of online RL in controlling a physical device to solve real-life problem.
adaptation to different terrains. An offline cable control parameter optimization was developed in Li al. 2022b, which relies on an impedance model based on the geomet- ric relationship of ankle joint. While these studies have achieved coordination between the robot and the user, they have a strong prerequisite that the wearable robot possesses a robust dynamical model. They also lack the ability of online tuning and personalizing for different users. There- fore, new innovations are needed for those wearable devices ready for real life deployment.
Offline-to-online RL is an effective and practical approach for robot control. Extensive research effort has been put into addressing the distribution shift from offline to online. The issue has been explored through various approaches. Zhang al. 2023 introduces policy expansion which ex- pand the policy with another learnt policy and implements Boltzmann action selection strategy. Several studies (Lee et al., 2022; Zhao et al., 2023a;b) utilize an ensemble of pessimistic value functions to mitigate distributional shift. Zhong et al. 2024 develops a sim-to-real application on gantry crane. Nakamoto et al. 2024 proposes training an additional value function address over-conservatism. Li et al. 2023 introduces a policy regularization term for trust- region-style updates. Lei et al. 2023 uses an on-policy optimization strategy unify offline and online training with- out extra regularization. However, nearly all of the current dffline-to-onlig RL approaches rely on extensive simula- tions or a vast amount of offline data, and none of them have demonstrated directly learning from physical environ- ment using only limited data for offline policy (Levine et al., 2020; Kumar et al., 2020). Furthermore, almost all the demonstrated robot control applications are under struc- tured environment such as those in warehouse settings or performing specific tasks.
2. Related Work
Control of Soft Exosuit. A fundamental control challenge with wearable devices is modeling the interaction dynamics between the user and the robot for optimal coordination (Polygerinos et al., 2015; Nesler et al., 2018; O'Neill et al., 2022; Liu et al., 2022; Wu et al., 2022), and additional challenges as discussed in Appendix B. Several studies have achieved successful coordination through human-in- the-loop optimization methods. In Siviy et al. 2020, offline optimization of cable-driven ankle exosuit is performed to generate the assistive torque profile. In Ding et al. 2018, the authors perform buman-in-the- ln-the-loop optimization through Bayesian optimization to identify the peak and offset timing of hip extension assistance with a cable-driven hip exo- suit. In Kim et al. 2019b, the authors advance this frame- work by coupling Bayesian optimization with a Kalman filter metabolic estimator to deliver plantar flexion assis- tance the ankle with a cable-driven ankle exosuit. In Li et al. 2022a, the authors developed a hierarchical human-in- the-loop controller of cable-driven exosuit for impedance
The PPO and SAC methods have been widely used in online robotic learning, again, relying heavily on extensive simula- tions for under structured environment learning. Both algo- rithms have shown great ability to learn stochastic policies by parameterizing a probability distribution over actions. These Distribution-based policies facilitate exploration by sampling actions during training (Haarnjj et al., 2018 8; Schulman et al., 2017). However, applying PPO and SAC in soft exosuit robot that does not have a simulator or a dynamic model of human-robot interaction, presents true challenges. Additionally, in wearable robotics with human in the loop, user comfort and safety are of great importance. Human participants become frustrated if the robot operates erratically (Dani et al., 2020). This necessitates exploration of using deterministic control policies.
Data-centric approach in offline imitation learning (IL). IL is natural and effective part of reinforcement learn- ing (RL) to device a reasonable initial policy (Taylor et al.,
2

---

AIP Assisted Human Walking Without Simulator
Video
U =
(()
Knee Angle (Deg)
100
da
McCap
80
EMG sensors
Air Bags
5
Online RL
60
MoCap Marker
40
Offinee
IMU sensors
|
Policy Transfer R
20-
Sensor State
L
0%
20%
40%
60%
80%
100%
(c)
Figure 1. (a): Offline imitation learning using normative human walking data with ground truth provided by MoCap. The learned policy is then used to initialize online RI. (b): Online, personalized RI. control of the soft exosuit to achieve human-robot normative walking while minimizing human effort measured by EMG activity. Sensor data are acquired via IMU for real-time control. (c): Knee angle profile of a complete gait cycle (in %) with the four gait phases as shown.
201 To address the issue of distribution shift when ap- plied in online environments (Ross et al., 2011; Spencer et al., 2021), two typical approaches have been explored. Algorithm--entric approaches aim learn robust policies by imposing task-specific assumptions based on specific characteristics of the task (Galashov et al., 2022: Guhur et al., 2023; James & Davison, 2022), or acquiring addi- tional data to model environment dynamics for the agent to return to in-distribution states (Englert et al., 2013; Qi et al., 2022). Some approaches enhance action representation such as using Gaussian or mixture models to capture all expert actions (Chi et al., 2023; Mandlekar al., 2021). Others reduce the task length by employing temporal abstraction of the action spaces (Shridhar et al., 2023; Zhao et al., 2023c). Data-centric approaches prioritize data quality, primarily aiming to maximize state diversity. Numerous studies fo- cus on modifying data collection processes to expose the expert to a diverse set of state transitions through shared control (Cui et al., 2019; Kelly et al., 2019; Ross et al., 2011). Some methods allow human intervention to correct robot behavior when necessary (Gandhi et al., 2023; Man- dlekar et al., 2020). Active learning guides data collection toward more informative samples by prioritizing questions that maximize information gain while minimizing the diffi- culty of selecting queries(Bryık et al., 2019; Cui Niekum, 2018). However typical data-centric techniques, such as collecting more data, diversifying state transition, actively learning human walking dynamies, human intervention of natural and normative walking, are entirely unfeasible for easy to understand reasons. We focus on improving data quality with two specific considerations: 1) to avoid long data collection processes, and 2) effectively deal with environmental noise to address our unique problem chal- lenge. With such a data-centric framework, we expect to improve action divergence (Belkhale et al., 2024) between the learned policy and the demonstration policy, thereby to improve task success rates.
3. Method
Our online Adaptation from an offline Imitating expert Pol- icy (AIP) procedure consists of two main phases. First, we develop a data-centric offline learning approach by employ- ing IL. We aim to capture a baseline normative walking policy by focusing on improving data quality and thus the quality of the initial policy for online adaptation. Second, to personalize RI. controller for new users online, actor-critic online learning method, the direct heuristic dynamic programming (dHDP) (Si & Wang, 2001), which is deterministic policy gradient method. It is based on the idea that the actor adjusts the policy in the direction of the action-value gradient, and the critic updates the action-value function. The idea can also be found in NFQCA (Hafner Riedmiller, 2011), which in turn, in DPG (Silver et al., 2014) and DDPG (Lillicrap et al., 2015). A more detailed discus- sion on dHDP and its performance evaluation in comparison to DDPG, as well as its several significant applications in complex and realistic engineering systems can be found in Appendix B. Ultimately, due to the holistic considerations that define the AIP method, it one of the first realizations that does not rely on simulator or vast amount of offline data to train an offline policy and successfully adapt it online for new users.
3.1. Physical Setup
Refer to Figure 1, the AIP solution involves two main phases, offline imitation learning (Figure 1.a) and online personal- ized RL (Figure 1.b). In both phases, a participant walks on the treadmill at a constant speed of 1 m/s. The inertial mea- surement unit (IMU) sensors collect kinematic data while the electromyography (EMG) sensors measure muscle ac- tivity simultancously. A Motion Capture (MoCap) System, which provides ground truth measurement of the human joint motion, is time synced with the IMU and EMG sensors in the offline phase and the ground truth walking profiles were used to train an offline human policy as an initial policy
3

---

AIP Assisted Human Walking Without a Simulator
for online training. The complete anthropometric data of the subjects and IRB can be seen in Appendix A and details on the placement of the soft inflatable exosuit, its manufac- turing, wearable sensors, etc. can be found in Appendix D. Experiment protocol can be found in Appendix E.
and lcc By precisely adjusting these timings and durations, the RL controller ensures that the exosuit provides optimal assistance to the user's knee movement, enhancing over- all gait efficiency and reducing muscle effort quantified by EMG measurement (EMG Effort). Note that, it is natural to maintain consistency in the state and action spaces during both offline and online learning.
3.2. State and Control Variables
An analysis of knee joint kinematics reveals two critical regions associated with knee stance extension (from point A to B) and swing extension (from point C to D) during a gait cycle (Figure I.c). The four extrema mark the transition from one gait phase to another: marker A is the maximum knee flexion during mid stance, B the maximum knee tension during terminal stance phase, C the maximum knee flexion during mid swing phase, and D the maximum knee extension adjacent to heel strike. These transition points and their related characteristics are therefore considered for incluusonn in the state representation from the human walk- ing profile. Specifically, the state variables include the peak knee flexion angle at point C as denoted by fz. the time instances a and tc of peak knee flexion the stance and swing phases, respectively, the duration of the stance phase (between point A and point B) defined as dA tB- tA. and the duration of the swing phase (between point C and point D) defined as de = LD lc. Thus, we define the state variable as follows:
3.3. Safety Constraints
To ensure human participants walk continuously and safely, we consider several safety constrains: 1) the actuator pres- sure is limited to 206.8 kpa: 2) the control timings and inflation/deflationn durations are constrained by taking ref- erence of those during participant's 's normative walking as shown in Table 4, which are within realistic ranges (Zhang et al., 2020b). These physical constraints help prevent sig- nificant misalignment between controller timing and the respective gait phase during human walking. Without these constraints, it may trigger soft actuator deployment and cause discomfort or injury the user; and 3) the online training objective is set for the control timings to approach those during normative walking, and thus in a safe state. Details of the safety constraints and their physical repre- sentations are in Appendix C. Additionally, we provide a theoretical performance analysis of the online learning process to ensure learning convergence, optimal timing so- lution, and human-robot interaction dynamic stability under reasonable conditions and within these safety constraints (Appendix F).
= dA, dc,
(1)
Unlike rigid exoskeletons where the torque is generated by electrical motors and can be directly used as control pa- rameter, for soft inflatable actuators, the amount of assistive torque is determined by both human knee torque and the ac- tuator pressure, the two collaborative sources. It is therefore not feasible to use torque directly as the control variable for the exosuit. Instead, only properly timed inflation and defla- tion of the exosuit will provide the necessary and optimal assistance to the human user (Figure I.c). On the contrary, if the exosuit is not properly operated, it may cause discomfort or even injury to the human user. Toward this end, the RI. controller must determine the optimal timings to operate the exosuit and these control parameters are:
3.4. Offline Human Normative Walking Policy
We take different perspective of not focusing on develop- ing another offline method, but instead, focusing on how to improve data quality to make offline imitation learning more effective in developing a good quality offline policy, By doing so, we aim to demonstrate the generalizability and data efficiency of our method by ONLY collecting of- fline walking data from a SINGLE participant (participant 1) with N 150. Detailed information about the offline data collection process can be found in Appendix E. Our im- itation learning approach utilizes Behaviour Cloning (BC) (Torabi et al., 2018: Bain & Sammut, 1995; Daftry et al., 2017) to derive an effective imitation policy based on data D = (zk)), A: 1, 1,2, .... N), obtained from normative walking demonstrations under natural walking condition of human participant, where N represents the total number of gait cycles over which the state variable data is collected from the MoCap system.
u= [t2, d1. f2, dz)"
where 1 is the onset timing of inflation of the exosuit to assist stance extension, the duration for which the air pres- sure is maintained during this phase is dj. Similarly, t2 represents the onset timing of inflation of the exosuit to assist swing flexion, and the corresponding duration for maintaining air pressure during this phase is given by d2. As it takes time for the exosuit to inflate and deflate, it is expected that an optimal RL controller should successfully learn the optimal timings of t and f2, which are expected to be close to or ahead of the maximum flexion timings fA
A good offline normative walking policy should serve the following two purposes. First, it provides a reasonable ini- tial policy for online tuning tailored for individual users while both offline and online learning are subject to simi-
4

---

AIP Assisted Human Walking Without a Simulator
lar environmental uncertainties such as sensor and actuator noise. As such, this offline policy helps online policy tuning to be kept in reasonable and meaningful range. Second, this offline learned policy should capture key human loco- motion characteristics even under intra- and inter-person variations (Zhang et al., 2020a; Ahn & Hogan, 2012) as human locomotion (such as knee angle) exhibits similar patterns as shown in Figure (I.c).
walking policy T((s), which maps human state from IMU sensors control timings and durations of normative walk- ing with the ground truth provided by MoCap, namely,
क {EA. d.. dc} E D. We use "action divergence" to measure offline cost T2 in BC learning,
1
Ck (T((S)) Tx)
(6)
Therefore the actor with policy parameter (6) minimizes a supervised loss as:
Improve data quality. Based most recent results that data-centric approaches have greater impact than algorithm- centric approaches on the effectiveness of imitation learning (Belkhale et al., 2024), we aim to improve data quality and expect that to be especially effective in addressing our unique challenges associated with the human-robot system under study. We therefore propose a reducing intra-person and inter-person variation (RIIV) method to improve mea- sured data quality as it is likely to be the one and the most effective and efficient approach. As result, we compare our RIIV with a common benchmark approach that normal- izes the raw measurements of state variables into the [-1,1]
N
(
N
k=0
which is the distance between the RL policy and that used in human demonstration.
3.5. Online, Personalized Soft Exosuit Control
Personalizing soft robot control for individuals face the fol- lowing offline to online learning challenges: 1) Data out of distribution (OOD) due to inter- and intra-hummn variance; 2) Limited availability of human walking data; and 3) Hard- ware limitations including communication delays, sensor noise, and significant delay in actuation. 4) Human accep- tance of the physical device which is directly correlated with human trust in the device and comfort when walking under robot control. Our solution relies on a good data quality im- provement procedure and an efficient online reinforcement learning algorithm, the dHDP which can sccessfully address the above OOD problems as shown in Appendix G.
range.
Specifically, for each gait cycle of length T. let the origi- nal measurements of each state variable of s, as in Equa- tion (1), be denoted by Ç. The following computations are performed component-win wise for each of the state variables (j 0==1, -1,2, 5).
1) The benchmark DIRECT method normalizes the raw sen- sor measurements (() of states into [-1,1] by the following procedure,
The dHDP actor-critic learning has demonstrated online learning convergence under limited data conditions empiri- cally in several wearable robotics applications (Wen et al., 2016;;b; 2017a; 2019). Additionally, dHDP with expeirence replay and target network has shown compatible perfor- mance to DDPG in many DMC robotic control problems (Wu al., 2024). Along with nice theoretical properties of dHDP (Sectoin 3.6) and practical successes in previous studies, we employ dHDP in our current study to learn the exosuit control solution with 5-dim state space and 4-dim action space.
8 2 - (mmi(())
1.
maxx(() mán(((
2) The RIIV method.
The first step of RIIV reduces ations by converting gait timing from actual time into gait
step length vari-
percentage by normalizing over a gait cycle T", that is
{
(4)
T
The goal of online learning control of the soft exosuit to minimize the user's muscle effort by improving the ef- fectiveness of the exosuit assistance. We thus consider two necessary performance metries: gait normalcy thus safety constraint, and muscle effort. the online learning objective is therefore to minimize the overall cost over policy ग, defined as follows:
The second step reduces inter-person variation by transfer- ring state variables into the range of [-1,
E --in((E) s = ( - inf(E)
-,
(5)
sup(E)
where the values of inf(() and sup(Ş) are from established studies of biomechanics literature (Zhang et al., 2020b), which is shown in Table 6 in Appendix C.
QT uK) = E
f ux),
7
t-k
(8)
Imitation policy. Once real time measurements for of- fline policy training are obtained during normative human walking, BC is utilized train an offline imitation human
t=k
where 8y ~ p( sb-1, Up 1x-)), us = π (sx), c c(Sk, ux) is the stage cost, and the discount factor <
5

---

AIP Assisted Human Walking Without a Simulator
? 1. The stage cost Cj in the above is formulated to take into consideration of two important performance measures in online learning of personalized optimal policy to achieve robot-assisted normative walking with reduced energy ex- penditure.
efficient learning without a simulator. As result, we have shown promising first steps in addressing key challenges of RL control for real life applications. Furthermore, our
AIP as a data-centric, offline to online approach reveals its practical usefulness to address environment uncertainty due to variations in human, sensor and actuator noise and delay that are unavoidable in real physical environments. Videos of the experiments can be seen in supplemental files.
First, we embed normative walking and safety constraint e¹ as one of the important performance considerations in the performance index (Equation 10). Specifically, c² = (s -) 8)² where target state & is defined in Appendix C, and is extracted from the offline normative walking profile using MoCap data as in Table 4. Additionally, e² is bounded within safety constraint provided in Table 5. It ensures that the subject does not deviate significantly from the target, thereby preventing potential falls or discomfort. The sec- ond consideration of reducing human energy expenditure is reflected by reduced muscle activity, which is measured per gait cycle, namely, the EMG effort e is determined by
Performance Criteria. The results reported in this study were based on the following performance metrics: 1) The stage cost as shown in Equation 10 to reflect online learning performance; 2) Peak knee error as a kinematic measure of normative walking and also to reflect walking safety; 3) EMG activity (Equation 9) which reflects human effort during walking: 4) Time to convergence of RL online learn- ing (influencing human physical fatigue); 5) "Action diver- gence" to measure offline policy optimality as in Equation (6). For all the metrics, better performance is associated with smaller/shorter outcomes.
T
fE()))²
Questions Addressed. Our real experimental results aim at answering the following questions:
which ft()) is the EMG sensor value at time t of gait and E. (t) simulates the integral of the EMG signal under a complete gait cycle of length T.
1) Is RIIV effective method for improving IMU sensor data quality in our data-centered solution framework? 2) Can offline normative human walking policy be further adapted and customized for individual participants via robot online learning to achieve optimal human-robot interaction? 3) Is human adaptation alone sufficient to improve perfor- mance with reduced effort?
We thus have the stage cost Ca formulated by balancing the reduction of EMG effort and adherence state error tolerance and safety constraints, and it is consequently used in formulating the total cost in Equation (8):
4) there evidence that both human and robot co-adapted
to achieve optimal interaction
=
(10)
5) To achieve optimal interaction between the human and the robot, what are essential control cost objectives to be considered in RI. design?
The dHDP is then used to provide online learning of a personalized optimal policy for individual users. Further details about dHDP, its actor and critic network realizations, and its implementation can be found in Appendices E and F.
6) Is the method generalizable to other locomotion tasks beyond level-ground walking?
Q1: (Oflline Benchmark Study) Our RIIV method is practically effective in capturing invariant normative walking characteristies while directly accounting for sen- sor and actuator noise in real environments, thereby improving offline policy optimality or action divergence. From Figure 2 and Table 2, we can clearly see advantages of using RIVV procedure over the Direct method (Section 3.4) to process raw IMU sensor data. 1) Firstly, RIVV results in significantly lower training cost and faster convergence than the Direct Method. As illustrated in the four bar charts in Figure 2, the RIIV method (green bar) reduces the action divergence effect more greatly than the Direct Method (or- ange bar) does, indicating that RIIV more accurately aligns with true human walking characteristics. 2) Next, RIIV has shown to be capable of accounting for significant uncertain- ties inherent in physical sensing and actuation, as demon- strated by the green bar with its values closer to the ground
3.6. Qualitative Properties of the Learning Process and Control Performance Associated with dHDP Online Learning
We provide a theoretical analysis to characterize proper- ties of the learning process and the control performance, specifically those related to learning convergence, solution optimality, and control system stability as a result of online dHDP learning initialized by an offline policy obtained via imitation learning. Details are provided in Appendix F.
4. Results and Analysis
This study of directly learning to control a physical device, the robotic exosuit, to assist normative human walking aims at exploring the feasibility of RL. in achieving stable and
6

---

AIP Assisted Human Walking Without Simulator
mm1 0.0235
5-137
0.5906
6-110
I8
Soreer
HIN
Drect
Sensor
Dhrest
Dree:
Figure 2. Offiine learning outcomes evidence of the essential role of processing sensor mcasurements in AIP data-centric method. (Left): Comparison of cost performance, Equation (7), using Direct and RIIV, the shaded regions represent the 95 % confidence range of the five random seeds. (Right panels): The MoCap data used ground truth in the comparisons, where action divergence (AD) as in Equation (6) was measured (the closer to O the better.): "bluc" is AD between IMU sensed data and the truth; "orange" is AD hetween Direct and Truth; and "green" is AD between RITV and Truth.
Table I. Performance of AIP method in terms of stage cost, peak knee error, and EMG effort.
Performance Evalustions
Beginning Online Training (Oflinee IL policy)
Ead of Online Training
Human Particppant
3
5
Stage Cost
0.94
0.23
0.57
0.99
0.19
0.43
0.49 0
0.37
0.35 1
0.49 1.09
Truining Peak Knee Emor
NIA
N/A
N/A
NIA
NIA
0.39 1 0.05
0.23 0.04
0,16 1 0.14
0.37 1 0.27
0.22 1: 0.07
Training EMG Effint
N/A
0.54
0.59 0.13
0.43 +
0.57
0.65
Evaluation Peak Knee Error
0.48 = 0.09
0.33 = 0.06
0.28 + 0.05
0.2
0.15
0.41 0.02
0.23 0.04
0,18 : 0.18
0.38 :: 0.07
0.11
Evalution EMG Effort
0.66 0.01
0.96 + 0.3
0.55 + 0.09
088 - 0.27
0.92 0.17
0.52 + 0.05
0.32 + 0.02
0.42 + 0.07
0.38 + 0.02
0.41 + 0.07
Baselinc without Exesuit Assistance
Bod of Online Training
EMG Effort
0.631 + 0,06
1.14 = 0.03 0,68 + 0.04
0.6 = 0.03
0.71 0.06
-14.4%
-48.2%
-36.8%
-5%
-8.5%
Table 1, a significant intra-subject variance and inter-subject variance is apparent. At the initial online learning stage (gait cycle 1), the same offline policy produced varying per- formances across different participants. However, by the end of training, the cost consistently converged to similar values of around 0.5, which indicates that online training has effectively customized the initial offline policy for each indi- vidual, allowing all participants to reach normative walking patterns with at least a 20% reduction in EMG effort.
Table 2. Statistic data of Figure 2. The MoCap Video data is used
as ground truth the comparisons, where action divergence (AD)
as in Equation (6) was measured (the closer to 0 the better.)
Action Divergence
Sensor
DIRECT
RIIV
t
0.03 + 0.003
0.038 + 0.002
0.02 + 0.002
da
0.004 + 0.01
0.259 + 0.015
0,0003 + 0.007
lc 0.0926 0.03
0.21
0.025
0.02 1 0.003
dc 0.0963 0.02
0.02
0.009
0.02 0.01
Q3: Human adaptation alone cannot sufficiently im- prove human walking performance. To isolate the effect of co-adaptation, feasible approach robot's adaptation in order to observe human adaptation. This is implemented by freezing policy updates. The re- sults are shown in Figure 7, where in row A (baseline), participants walked naturally and showed their natural gait pattern; in row C (after online training), the policy already optimized, so no further human adaptation is needed. In both cases, we observe sign of human adaptation. In row B (using offline-learned, and fixed policy), this offline policy is not optimal and needs to be personalized. This is where we speculate that the participants may realize the mismatch and attempt to adapt. But due to a lack of clear or consistent direction, the participants' responses did not exhibit any discernible pattern or trend. For example, PI shows delayed timing in l. while others do not exhibit this behavior. Similarly, P5 demonstrated slightly carlier duration in de. which was not seen in other participants. In contrast, evidence of human- co-adaptation during
truth, especially for A. te, and de, where there are notable discrepancies between raw sensor data and the ground truth, and also, a rather significant delay in the actuator due to inflation/deelltion time.
disable the
Q2: Online learning effectively adapted the initial offline policy to provide personalized control for individual par- ticipants and enable robust performance in human-robot normative walking.
From Table 1, although the offline policy enables walking, does not achieve optimal performance in terms of cost, kinematic error, and EMG measures. As shown in Figure 3 and Table while the offline policy directly benefits par- ticipants 2 and 3 in terms of reduced EMG effort (below baseline shown by dashed line), it fails to do so for partic- ipants 1, 4 and 5. Through online training. Performance metrics improve for reduced cost and kinematic error, and most importantly, reduced EMG effort for all subjects. No- tice additionally that online training resulted in consistent and robust assistance to human walking. From Figure 3 and
7

---

AIP Assisted Human Walking Without a Simulator
2.5
1.11
Punticipant
2.0
1.2
Cost
1.5
0.0
EMG LEElec
1.0
0.6
0.5
0.2
0.4
10
20
25
30
10
20
25
30
10
20
25
30
Carit Cycles
Gast Cycles
Gait Cycles
Figure 3. Results of online training for all five participants where the shaded regions indicate the 95% confidence interval for the three online trials. The dashed lines are respectively the baseline human walking EMG effort without exosuit assistance. Participant I provided the offline policy.
Bettee Chiine Tokreg
Chatte Theerg
0.0
0.3 6:5
0.3 0.. 1.D
1.5
0.00
0.25
13.50
6.75
1.00
000
0.5
2.0
Figure 4. Timing and duration in state and control variables to demonstrate adaptation taking place during online learning. In the top panel above the bar charts, the blue line segment is d', the red is de, the purple is di and the green is dz. The bar plots show the mean differences in timing and duration between respective actual human walking measurements and those of the robot control. Specifialy, St = f1 tA. Ad d1 da, ötc t2 tc, and ddc d2- dc.
online learning clear, the benefit of which is shown in the discussion Q4 below.
they weigh the importance of reducing effort during this less effort demanding phase of walking. 5) The swing phase soft actuator onset timing 22, however, has adapted to be ahead of the human actual start of swing tc. Inflating actuators in this phase is critical to reduce human effort of lifting the leg and swing it forward. Note that the initial policy from offline learning also resulted in even earlier swing onset (2, an outcome that may be caused by out-of-distributio effect as there was no soft actuator deployment during offline training. Consequently, to accommodate soft actuator delays there has to be an early onset, but cannot be too early so that the soft actuator is in the way of a normal knee swing flexion (reduce the peak knee angle error). Additional details about addressing the OOD issue can be found at Appendix G.
Q4: Human and robot co-adapted to achieve normative walking with reduced human EMG Effort
1) Refer to Table 1, online training of robot control has led to normative walking, as measured by the peak knee angle approaching that during normative walking (small peak knee error), and reduced EMG effort for all participants. This is a result of online co-adaptation between the human and the robot. To see that, we show next how robot control has taken effect by looking into measurable human walking states. 2) Let's examine the duration of human stance phase (d,) and swing phase (dc) of offline policy and after online learning. Note that the respective duration has changed little (refer to the top row of Figure 4 above the bar charts with more details provided in (Figure 9), and Figure 8a & b). This is because the participants walk naturally and thus maintains their normative walking patterns. 3) In the meantime, note that the robot has reduced its stance duration (di) ) and swing duration (d2) accommodate soft actuator deployment delays (refer to the top row of Figure 4 above the bar charts). 4) Next, if we inspect the robot control onset timing 4 for stance, it varied around human stance timing t (refer to the bar plots in Figure 4 with more details shown in Figure 9 B). As the soft actuators are to provide leg support for stance, the human responses could vary depending on how
Q5 (Ablation Study) Both safe regulation of joint kine- matics and reduction of human EMG effort are neces- sary to achieve stable human-robot normative walking. We performed an ablation study on the cost objective func- tion. Our proposed performance index, which incorporates both EMG effort and kinematic eтror or state error, demon- strates superior performance. By balancing the reduction of EMG effort and adherence to state error tolerance and safety constraints, the RL controller optimizes both aspects of the user's 's walking behavior. Refer to Figure 3, this bal- anced approach leads to convergence, stability, and signifi- cant improvements in the user's mobility, as evidenced by lower stage cost, peak knee error, and decreased EMG effort.
8

---

AIP Assisted Human Walking Without Simulator
Without EMG Term
Without Kinematic Tearm
Full Objective
20
Ponk Kaee Error
0.5
1.5
Stage Cost
FMG Fiffort
A)
10
0.4
1.0
0.5
0.6
1.6
0.3
B)
1.4
0.4
16
0.3
0.2
1.2
0.2
0 10 20 30 10
20
30
0
10 20
30
0
10
20 30
Gait
Gait
Figure 5. Ablation study: Learning performance under different formulation of objective functions associated with ablation study 1) Without EMG effort in the control objective function 2) Without kinematic term the control objective function 3) Full version of objective function. Row A: Results of online training for three different objective functions. Row B: Results of robot controller timing for three different objective functions. The shaded regions represent the % confidence range of the three experiment trials. The x-axis the number of gaits. The black dash lines are the reference INSELINE from human normative walking profiles.
However, if only EMG effort Kinematic error was used in Equation 10, not only the EMG did not reduce but also it resulted in significant learning variance. The absence of state kinematic error in the cost function (Figure 5 Or- ange line) resulted in failure to maintain normative walking patterns, which led to increased EMG levels and overall less effective assistance. The absence of EMG effort in the performance index (Figure 5 Green line) leads to a lack of focus on reducing muscle activity. Consequently soft exosuit failed to provide the necessary support to reduce muscular strain, resulting in increased EMG levels.
under active development, they have been shown effec- tive accelerate convergence and reduce user-specific data needs. Additionally, our roadmap also includes extending the method beyond basic maddiil-simulated level-ground walking to encompass more complex and realistic tasks and scenarios, such as walking at variable speeds, incline and decline walking, among others. Another impactful future research could expand this approach consider a broader and more diverse population, integrate complex locomotion patterns, and explore long-term adaptation.
Q6 AIP is generalizable to incline walking. As a first step of extension, we developed a new incline walking platform to validate the feasibility of AIP in more complex locomo- tion tasks. As shown in Figure 10, which are results from 7-degree incline treadmill walking, increasing the incline significantly alters gait kinematics. However, as shown in Panel C, the AIP method remains robust, consistently reducing EMG effort below BASELINE levels across all participants.
6. Conclusion
In this study, we introduce the AIP method to address the challenges of soft exosuit-assisted human walking by bridg- ing offiine and online learning, rather than developing yet another RL method. By leveraging existing RL algorithms (dHDP), AIP enables personalized assistance to reduce hu- man physical effort during normative walking. AIP im- proves data quality through the RIIV method, integrating of- fline imitation learning with RIVV and online dHDP learning for data-efficte efff-eefficeet, personalized assistance. Our framework, implemented and demonstrated in physical environment, overcomes the challenge of optimizing human-robot inter- action without dedicated simulator or dynamic model. We validated our approach on five participants without selec- tively presenting results, demonstrating a simple yet robust solution for safe robot control. The co-adaptation between human and robot effectively mitigates actuator delays, re- ducing muscular effort and enhancing human-robot synergy. This work paves the way for personalized robotic assistance in rehabilitation and performance enhancement.
5. Limitation and Future Work
Firstly, the AIP method has been tested for single side con- trol of one leg. Extending it to bilateral control introduces challenges in coordinating and synchronizing assistance across limbs. Multi-agent RL offers a promising direction to address this open question. Secondly, RL's reliance on exten- sive data persistent limitation. Methods such as LNSS (Zhong et al., 2023a), ATD (Zhong et al., 2023b), and other techniques such as transfer learning are promising directions

---

AIP Assisted Human Walking Without a Simulator
Acknowledgment
needs of this population. These adjustments are crucial to effectively address the distinct challenges of impaired gait and to ensure the success of such an extension.
Zhong and Si's research was supported in part by the National Science Foundation under Grant No. 2211740. Yumbla, Soltanian, and Zhang's research was supported in part by the National Science Foundation under Grant No. 1944833 and in part by Arizona Biomedical Research Cen- tre under Grant No. RFGA2022-010-08. The authors thank Jahnav Rokalaboina and Nick Kirkby for their support in collecting data for the incline experiments.
References
Adamczyk, P. G. and Kuo, A. D. Mechanisms of gait asym- metry due to push-off deficiency in unilateral amputees. IEEE transactions on neural systems and rehabilitation engineering, 23(5)777--785, 2014.
Impact Statement
Ahn, J. and Hogan, N. Walking is not like reaching: evi- dence from periodic mechanical perturbations. PloS one, 7(3):33176,, 2012.
This paper is the first to demonstrate the application of re- inforcement learning (RL) for controlling a soft exosuit to assist human walking with reduced effort. Our study specif- ically focuses on providing walking assistance to healthy individuals walking on a flat treadmill. This study is not attended for walking rehabilitation applications, which will be considered in our future work. What we have shown in this study is an important first step toward future broader ap- plications of the technology. We chose this setting because implementing such technology without thoroughly testing on healthy participants under the most common condi- tion (such as level ground) could raise significant societal concerns. Notably, assistive walking technologies for unim- paired and impaired populations are typically approached separately, each with distinet objectives.
Azocar, A. F., Mooney, L. M., Duval, J.-F., Simon, A. M., Hargrove, L. J., and Rouse, E. J. Design and clinical implementation of an open-source bionic leg. Nature biomedical engineering, 4(10):941-553, 2020.
Bain, M. and Sammut, C. A framework for behavioural cloning, In Machine Intelligence 15, pp. 103-129, 1995.
Baker, B., Akkaya, L., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 15:24639-24655 2022.
In the case of unimpaired individuals, as addressed in our study, the goal is to reduce the effort required for walking while still achieving a normative caiit-cosssstem with the aim stated in our paper's title. In contrast, wearable robots designed for individuals with physical impairments have different priorities. For these populations, improving gait symmetry, balance, and walking speed takes precedence over reducing effort. This is due, in part, to compelling evidence that irregular gait patterns, such as those seen in individuals with unilateral lower limb amputations, can lead to secondary complications if not properly addressed. Asymmetrical gait, for example, commonly observed in amputees and is associated with secondary health issues like osteoarthritis in the non con--mmttttedd joints (Hof et al., 2007; Adamezyk& & Kuo, 2014) and lower back pain (Ehde al., 2001).
Belkhale, S., Cui, Y., and Sadigh, D. Data quality in imita- tion learning. Advances in Neural Information Processing Systems, 36, 2024.
Bıyık, E., Palan, M., Landolfi, N. C., Losey, D. P., and Sadigh, D. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint 1911::1910.04365,00 2019.
Bortole, M., Venkatakrishnan, A., Zhu, F., Moreno, J. C., Francisco, G. F.,, Pons, J. I.., and Contreras- Vidal, J. L. The h2 robotic exoskeleton for gait rehabilitation after stroke: early findings from a clinical study. Journal of neuroengineering and rehabilitation, 12:1-14, 2015.
Brenneis, D. J., Parker, A. S., Johanson, M. B., Butcher, A., Davoodi, E., Acker, L.., Botvinick, M. M., Modayil, J., White, A., and Pilarski, M. Assessing human interac- tion in virtual reality with continually learning prediction agents based on reinforcement learning algorithms: pilot study. arXiv preprint 11X 2021.
Given these considerations and the outcomes of this study, we believe that our current AIP framework has the potential to be further developed for studies focusing on assistive walking for individuals with physical impairments and have the potential to scale up such walking at varying speed, on different surfaces, ascending and descending stairs. How- ever, this would require redefining the control design objec- tives, rethinking offline data collection strategies, and devel- oping a new experimental protocol tailored to the unique
Bryan, G. M., Franks, P. W., Song. S., Reyes, R., O' O'Donovan, M. P., Gregorezyk, K. N., and Collins, S. H. Optimized hip-knee--nkke exoskeleton assistance reduces the metabolic cost of walking with worn loads. Journal of neuroengineering and rehabilitation, 18:1-12, 2021.
10

---

AIP Assisted Human Walking Without Simulator
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burch- fiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint MrXiv:2303.04137 2023.
Galashov, A., Merel, J. S., and Heess, N. Data augmentation for efficient learning from parametric experts. Advances in Neural Information Processing Systems, 35:3 1484 31496, 2022.
Collins, S. H., Wiggin, M. B., and Sawicki, G. S. Reducing the energy cost of human walking using an unpowered exoskeleton. Nature, 522(7555):122-215 2015.
Gandhi, K., Karamcheti, S., Liao, M., and Sadigh, D. Elicit- ing compatible demonstrations for multi-human imitation learning. In Conference on Robot Learning, pp. 1981- 1991. PMLR, 2023.
Cui, Y. and Niekum, S. Active reward learning from cri- tiques. In 2018 IEFE international conference on robotics and automation (ICRA), pp. 6907-6914, IEEE, 2018.
Gao, X., Si, J., and Huang, H. Reinforcement learning control with knowledge shaping. IEEE Transactions on Neural Networks and Learning Systems, 35(3):3156- 3167, 2024.
Cui, Y., Isele, D., Nickum, S., and Fujimura, K. Uncertainty- aware data aggregation for deep imitation learning. In 2019 International Conference on Robotics and Automa- tion (ICRA), pp. 761-767. IEFE, 2019.
Gao, Y. and Liu, Y.J. Adaptive fuzzy optimal control using direct heuristic dynamic programming for chaotic discrete-time system. Journal of Vibration and Control, 22(2):595-605: 2016.
Daftry, S., Bagnell, A., and Hebert, M. Learning transfer- able policies for monocular reactive mav control. In 2016 International Symposium on Experimental Robotics, pp. 3-11. Springer, 2017.
Granberry, R., Duvall, J., Dunne, L. E., and Holschuh, B. An analysis of anthropometric geometric variability of the lower leg for the fit & function of advanced functional garments. In Proceedings of the 2017 ACM international symposium on wearable computers, pp. 10-17, 2017.
Dani, A. P., Salehi, I., Rotithor, G., Trombetta, D., and Ravichandar, H. Hluman-in-the-loge robot control for human-robot collaboration: Human intention estimation and safe trajectory tracking control for collaborative tasks. IEEE Control Systems Magazine, 40(6):29-56 56, 2020.
Guhur, P.-L., Chen, S., Pinel, R. G., Tapaswi, M., Laptev, I., and Schmid, C. instruction-drivee history-aware poli- cies for robotic manipulations. In Conference on Robot Learning, pp. 175-187. PMLR, 2023.
Ding, Y. Kim, M., Kuindersma, S., and Walsh, C. nuuman-in-th--oee optimization of hip assistance with soft exosuit during walking. Science Robotics, 3(15):caar5438, 2018. doi: 10.1126/sciroboties. aar5438. URL htRp::tss://wwt http://wwwwiiceeeee.ee//mmm org/ doi. abs/10 10000000000 126 www.sccoriiiioo aar5438.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint
Duan, J., Wang, Y. R., Shridhar, M., Fox, D., and Krishna, R. Ar2-d2: Training a robot without robot. arXiv preprint arXiv:2306. 13818, 2023.
2018.
Hafner, R. and Riedmiller, M. Reinforcement learning in feedback control: Challenges and benchmarks from technical process control. Machine learning, 84:137-169, 2011.
Ehde, D. M., Smith, D. G., Czerniecki, J. M., Campbell, K. M., Malchow, D. M., and Robinson, L. R. Back pain as secondary disability in persons with lower limb ampu- tations. Archives of physical medicine and rehabilitation, 82(6):731-734, 2001.
Hasan, M., Yumbla, E. Q., and Zhang, W. Development of soft inflatable exosuit for knee flexion assistance. In 2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatromics (BioRob), pp- 1-6, 2022. doi: 10. 1099/1ooob52689 2222
El-Sousy, F., Amin, M. M., Soliman, A. S., and Mo- hammed, O. A. Optimal adaptive ultra-local model-free control based-extended state observer for pmsm driven single-axis servo mechanism system. IEEE Transactions on Industry Applications, 2024.
Hilleli, B. and Yaniv, R. Toward deep reinforcement learning without simulator: An autonomous steering example. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Englert, P., Paraschos, A., Deisenroth, M. P., and Peters, J. Probabilistic -bsel- imitation learning. Adaptive Behavior, 21(5):388-44) 403, 2013.
Hof, A. L., van Bockel, R. M.. Schoppen, T., and Postema, K. Control of lateral balance walking: experimental findings in normal subjects and above-knce amputees. Gait & posture, 25(2):250-25 258, 2007.
Enns, R. and Si, Helicopter trimming and tracking con- trol using direct neural dynamic programming. IEEE Transactions on Newral networks, 14(4):929-9)) 9399,999999999999 2003.

---

AIP Assisted Human Walking Without Simulator
Huang, H. H., Si, J., Brandt, A., and Li, M. Taking both sides: seeking symbiosis between intelligent prostheses and human motor control during locomotion. Current opinion in biomedical engineering, 20:100314, 2021.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives open problems. arXiv preprint wrXiv:205..01643 2020.
Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., and Zhang, Y.- Q. Proto: Iterative policy regularized offline-oo-nllin reinforcement learning. arXiv preprint arXiv:2305. 2023.
Inoue, T., De Magistris, G., Munawar, A., Yokoya, T., and Tachibana, R. Deep reinforcement learning for high pre- cision assembly tasks. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 819-825. IEEE, 2017.
Li, M., Wen, Y., Gao, X., Si, J., and Huang, H. Toward expedited impedance tuning of robotic prosthesis for personalized gait assistance by reinforcement learning control. IEEE Transactions on Robotics, 2021.
James, S. and Davison, J. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1612-10) 16101 2022.
Li, Z., Li, X., Li, Q., Su, H., Kan, Z., and He, W. Human-in- the-loop control of soft exosuits using impedance learning different terrains. IEEE Transactions on Robotics, 38 (5):2979 2993, 2022a. doi: 20..100//RO.2222.33.060
Joshi, S. and Paik, J. Pneumatic supply system parameter optimization for soft actuators. Soft Robotics, 8(2):152- 163, 2021. doi: (0w.1//9//oo.220000000
URL https: www.:..0.1000oo00 /tr00///oroooo 2019 0134. PMID:
//doi.org//// 1ttrg//1... wttpg//...188//// 32598232.
Li, Z., Li, x., Li, Q., Su, H.,, Kan, Z., and He, W. Human-in the-loop control of soft exosuits using impedance learning on different terrains. IEEE Transactions on Robotics, 38 (5):2979-299) 2022b.
Kelly, M., Sidrane, C., Draggs-Campbell, K., and Kochen- derfer, M. J. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference Robotics and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint WwXi::/50.002971 2015.
Kim, J., Lee, G., Heimgartner, R., Arumukhom Revi, D., Karavas, N., Nathanson, D., Galiana, I., Sckert-Erdheinn A., Murphy, P., Perry, D., et al. Reducing the metabolic rate of walking and running with versatile, portable exosuit. Science, 365(6454) 672, 2019a.
Liu, W., Zhong, J., Wu, R., Fylstra, B. L., Si, J., and Huang, H. H. Inferring human-robot performance objectives dur- ing locomotion using inverse reinforcement learning and inverse optimal control. IEEE Robotics and Automation Letters, 2022.
Kim, M., Liu, C., Kim, J., Lee, S., Meguid, A., Walsh, C. J., and Kuindersma, S. Bayesian optimization of soft exosuits using a metabolic estimator stopping pro- cess. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9173-9179, 2019b. doi: 10.1
Long, Y., Du, Z., Cong, L., Wang, W., Zhang, Z., and Dong, W. Active disturbance rejection control based human gait tracking for lower extremity rehabilitation exoskeleton. ISA transactions, 67:389-399 2017.
Koller, J. R., Gates, D. H., Ferris, D. P., and Remy, C. D. seddy-n-thhe-oopp optimization of assistive robotic de- vices: A validation study. In Robotics: Science and Systems, volume 2016, pp. 2016.
Mandlckar, A., Xu, D.. Martín-Martin R., Zhu, Y., Fei- Fei, L., and Savarese, S. Human-in-the tn-the-loop imita- tion learning using remote teleoperation. arXiv preprint arXiv:20/2.06733, 2020.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
Mandlekar, A., Xu, D.,, Wong. J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L.., Savarese, S., Zhu, Y., and Martin-Martin R. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 2021.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic gennseblle. In Conference on Robot Learn- ing, pp. 1702-1712, PMLR, 2022.
Mnih, V. Playing atari with deep reinforcement learning. arXiv preprint /312v: 2013.
Lei, K., He, Z., Lu, C., Hu, K., Gao, Y., and Xu, H. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on on-policy optimization. arXiv preprint arXiv:231 1,0331,, 2023.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. a., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D.,
12

---

AIP Assisted Human Walking Without Simulator
Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 51877550 Feb 2015, ISSN 0028-0836. doi: 10.1038/nature1422 12236.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction no-regret online learning, In Proceedings of the fourteenth international conference on artificial intelligence and statistics, PP. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Save, O., Zhong, J., Joglekar, S., Si, J., and Lee, H. Person- alizing human gait entrainment: A reinforcement learning approach optimizing magnitude of periodic mechanical perturbations. IEEE Robotics and Automation Letters, 2025.
Nesler, C. R., Swift, T. A., and Rouse, E. J. Initial design and experimental evaluation of a pneumatic interference actuator. Soft robotics, 5(2): 138-148,, 2018.
Schulman, J.. Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint (eXivv/707.0666 2017.
Nikovski, D., Zhong, J., and Yerazunis, W. Memory-based global iterative linear quadratic control. In 2024 10th International Conference on Control, Decision and In- formation Technologies (CoDIT), pp. 1056-1061. IEEE, 2024.
Shi, D., Zhang, W., Zhang, W., and Ding, X. A review on lower limb rehabilitation exoskeleton robots. Chinese Journal of Mechanical Engineering, 32(1):1-11, 2019.
Niu, H., Qiu, Y., Li, M., Zhou, G., Hu, J., Zhan, X., et al. When to trust your simulator: Dynamics-aware hybrid annloinean reinforcement learning. Ad- vances in Neural Information Processing Systems, 35: 16599-36611 2022.
Shridhar, M., Manuelli, L., and Fox, D. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799. PMLR, 2023.
Si, J. and Wang, Y.TT. Online learning control by associ- ation and reinforcement. IEEE Transactions on Neural networks, 12(2):264 276, 2001.
O' Neill, C. T., McCann, C. M., Hohimer, C. J., Bertoldi, K., and Walsh, C. J. Unfolding textile-based pneumatic actuators for wearable applications. Soft Robotics, 9(1): 163-172, 2022.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference machine learning, pp. 387- 395. Pmlr, 2014.
Polygerinos, P., Wang, Z., Overvelde, T. B., Galloway, K. C. Wood, R. J., Bertoldi, K., and Walsh, J. Mod- cling of soft fiber-reinforeed bending actuators. IEEE Transactions on Robotics, 31(3):778-789, 2015. doi: 10.1 111.0T0..000..2222222
Siviy, C., Bae, J., Baker, L.,, Porciuncula, F., Baker, T., Ellis, T. D., Awad, L. N., and Walsh, C. J. Offline assistance optimization of soft exosuit for augmenting ankle power of stroke survivors during walking. IEEE Robotics and Automation Letters, (12):828-835, 2020. doi: 10.1109/ LRA.2000.2965072
Qi, C., Abbeel, P.,, and Grover, A. Imitating, fast and slow: Robust learning from demonstrations via decision-time planning. arXiv preprint
2022.
Siviy, C., Baker, L. M., Quinlivan, B. T., Porciuncula, F., Swaminathan, K., Awad, L. N., and Walsh, C. J. Opportu- nities and challenges in the development of exoskeletons for locomotor assistance. Nature Biomedical Engineering, 7(4):456-472, 2023.
Rao, K., Harris, C., Irpan, A., Levine, S., Ibarz, J., and Khansari, M. Rl-cyclegan: Reinforcement learning aware simulation-to-real. In Proceedings of the IEEF/CVF Con- ference Computer Vision and Pattern Recognition, pp. 11157-11166, 2020.
Spencer, J., Choudhury, S., Venkatraman, A., Ziebart, B., and Bagnell, J. A. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872, 2021.
Rodriguez-Femááddez Fernndezz A., Lobo-Prat, J., and Font- Clag-Llaguues, M. Systematic review on wearable lower-limb exoskeletons for gait training in neuromuscular impairments. Journal of NeuroEngineering and Rehabilitation, 18:22, 12 2021. ISSN 1743-0003.
Sridar, S., Qiao, Z., Muthukrishnan, N., Zhang, W., and Polygerinos, P. soft-inflatable exosuit for knee rehabil- itation: Assisting swing phase during walking. Frontiers Robotics and AI, 5:44, 2018.
doi: 10. 10.1186/11984-02200088551. URL https:
//heuuroengeehab.bioomeeeeeeee #tticcee//o..116//1223333
13

---

AIP Assisted Human Walking Without a Simulator
Sridar, S., Poddar, S., Tong, Y., Polygerinos, P., and Zhang, W. Towards untethered soft pneumatic exosuits using low-volume inflatable actuator composites and portable pneumatic source. IEEE Robotics and Automation Let- ters, 5(3):4062-4069, 2020. doi: 10. 1 109/ 2986744.
Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 27306-27320, Curran Associates, Inc., 2022.
Wu, R., Zhong, J., and Si, J. Actor-critic reinforce- ment learning with phased actor. arXiv preprint arXiv:2404. 1834, 2024.
Sun, J., Liu, F., Si, J., and Mei, S. Direct heuristic dynamic programming based on an improved pid neural network. Journal of Control Theory and Applications, 10(4):497- 503, 2012.
Yang, Q.. Cao, W., Meng, W., and Si, J. Reinforcement- learning-basd tracking control of waste water treatment process under realistic system conditions and control per- formance requirements. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(8):5284-522 2021.
Taylor, M. E., Suay, H. B., and Chemova, S. Using human demonstrations to improve reinforcement learning. In 2011 AAAI Spring Symposium Series, 2011.
Yumbla, E. Q. Qiao, Z., Tao, W., and Zhang, W. Human assistance and augmentation with wearable soft robotics: a literature review and perspectives. Current Robotics Reports, pp. 1-15, 2021.
Torabi, F., Warnell G., and Stone, P. Behavioral cloning from observation. arXiv preprint arXiv: 2018.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offinne-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023.
Tu, X., Li, M., Liu, M., Si, J., and Huang, H. H. A data- driven reinforcement learning solution framework for optimal and adaptive personalization of a hip exoskeleton. In 2021 IEEE international conference on robotics and automation (ICRA), pp. 10610-10616. IEEE, 2021.
Zhang, J., Fiers, P., Witte, K. A., Jackson, R. W., Poggensee, K. L., Atkeson, C. G., and Collins, S. H. Human-in- the-loop optimization of exoskeleton assistance during walking. Science, 356(6344 2017.
Wen, Y., Liu, M., Si, J., and Huang, H. H. Adaptive control of powered transfemoral prostheses based on adaptive dy- namic programming. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5071-5074. IEEE, 2016a.
Zhang, L., Liu, G., Han, B., Wang. Z., Yan, Y., Ma, J., and Wei, P. Knee joint biomechanics in physiological conditions and how pathologies can affect it: a system- atic review. Applied bionics and biomechanics, 2020(1): 7451683 2020a.
Wen, Y., Si, J., Gao, X., Huang. S., and Huang, H. H. A new powered lower limb prosthesis control framework based on adaptive dynamic programming. IEFE transactions neural networks and learning systems, 28(9):2215-2220, 2016b.
Zhang, L.,, Liu, G., Han, B., Wang, Z., Yan, Y., Ma, J., and Wei, P. Knee joint biomechanics in physiological con- ditions and how pathologies can affect it: A systematic review. Applied Bionies and Biomechanics, 2020: 1-22, 4 2020b. ISSN 11 1176-2322 doi: 10.1 10.11555000077456633.
Wen, Y., Brandt, A., Si, J., and Huang, H. H. Automati- cally customizing powered knee prosthesis with human in the loop using adaptive dynamic programming. In 2017 International Symposium on Wearable Robotics and Rehabilitation (WeRob), pp. 1-2. IEFE, 2017a.
Zhao, K., Ma, Y., Liu, J., Jianye, H., Zheng, Y., and Meng, Z. Improving offline-to tonee-to-nnnneee reinforcement learning with geensmmbles. In ICML Workshop on New Frontiers in Learning Control, and Dynamical Systems, 2023a.
Wen, Y., Si, J., Gao, X., Huang. S., and Huang, H. H. A new powered lower limb prosthesis control framework based on adaptive dynamic programming. IEEE Transactions on Neural Networks and Learning Systems, 28(9):2115- 2220, 2017b.
Zhao, K., Ma, Y., Liu, J., Zheng, Y., and Meng, Z. Ensemble- based offline-too to-ne--t--nnneeee reinforcement learning: From pes- simistic learning to optimistic exploration. arXiv preprint arXiv:2306.000 0000000677.1...1 1, 2023b.
Wen, Y., Si, J., Brandt, A., Gao, X., and Huang, H. Online reinforcement learning control for the personalization of a robotic knee prosthesis. IEEE transactions on cybernet- ics, 2019.
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fineggrained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.1 13705, 2023c.
Zhong, J., Wu, R., and Si. J. A long n-step surrogate stage reward for deep reinforcement learning. Advances in Neu- ral Information Processing Systems, 11:12733-12745 2023a.
Wu, R., Zhong, J., Wallace, B., Gao, X., Huang, H., and Si, J. Human-robotic prosthesis as collaborating agents for symmetrical walking. In Koyejo, S., Mohamed, S.,
14

---

AIP Assisted Human Walking Without Simulator
Zhong, J., Wu, R., and Si, J. Mitigating estimation errors twin td-regularized actor and critic for deep reinforcement learning, arXiv preprint urXiv:2311.03777
2023b.
Zhong, J., Nikovski, D., Yerazunis, W. S., and Ando, T. Learning time-optimal control of gantry cranes. In 2024 International Conference on Machine Learning and Ap- plications (ICMLA), pp. 945-950, IEEE, 2024.
Zhu, M., Biswas, S., Dinulescu, S. I., Kastor, N., Hawkes, E. W., and Visell, Y. Soft, wearable robotics and hap- tics: Technologies, trends, and emerging applications. Proceedings of the IEEE, 110(2)2246-72) 2022.
15

---

AIP Assisted Human Walking Without Simulator
A. Participant Information and IRB approve
Five healthy individuals (3 male and 2 female, participated in the study under a protocol approved by the Institutional Review Board (IRB ID#: STUDY00011 10 for level ground walking) and [RB ID#: STUDY00019873 for incline walking) The average height, weight, and age of the recruited participants were 163 = em, 66.1 11.6 kg, and 28 1.9 years, respectively. The complete anthropometric data of the subjects can be seen in Table 3.
Figure 6 is summary of histograms of the range of gait patterns which directly illustrate gait diversity of the participants. These data show that the participants' gait patterns span a broad spectrum, which covers safe and normative walking ranges as reported in well-established human dynamic walking literature. We therefore consider our experimental data capture commonly observed gait variations.
Table 3. Subject participants' anthropometric data.
Subject
Gender
Age
Weight (kg)
Height (m)
SI
M
26
76
1.75
S2
27
52
1.54
S3
28
79
1.65
S4
57.5
1.58
S5
M
28
80
1.72
timing gail percent
dA timing in gail percent
tc timing gail percent
de timing gail percent
30
40
Count
40
Count
40
Count
Count
40
30
20
20
10
0.10
0.15
0.20
0.10
0.15
0.20
0.6
0.7
0.1
0.2
0.3
tA
dA
tc
dc
Figure 6. Histogram to show Diversity of Gait Patterns across all the participants. All recorded gaits show coverage of possibles region within the normative walking ranges ( tA E [0.1, da E 0.3), to E [0.6, 0.75).dc E (0.1,0.3)
B. Additional Related Work
RL successes with and without simulated environments. The most celebrated reinforcement learning (RL) achievements with superhuman performance are playing computer games (Silver et al., 2014; Mnih et al., 2015). These successes are largely attributable to the use of unbiased simulation environments, which provide extensive and repeatable training data. However, the simulator-based successes have rarely been duplicated in the real physical world. High-fidelity simulators are often prohibitively expensive or even impossible to construct due to the complex dynamics, limitations in assessing and representing inherently uncertain physical systems, such as sensor and actuator noise, communication delays and other factors (Rao et al., 2020; Niu et al., 2022; Nikovski et al., 2024). Nonetheless, RL has been directly applied to physical systems without the use of simulators. For example, (Inoue et al., 2017) presents a method to enable industrial robots to perform sege-preciiion assembly tasks (such the peg-in-hole) by training an LSTM using reinforcement learning directly on the physical device. There have been some other successful demonstrations of RL agents interacting with simulated raw environments instead of simulators providing directly accessible state-action iiee-aactio--ewaart data. For example, (Hilleli& & El- Yaniv, 2018) trains RI. agents for autonomous highway steering using raw image sequences from simulated environment. The VPT (Baker et al., 2022) is a semi- superviee imitation learning method, where an inverse dynamics model (IDM) is trained with labeled data to generate pseudo-label for vast amount of unlabeled online videos. This allows for training a behavioral prior that exhibits nontrivial zero-shot capabilities and can be fine-tuned using imitation learning and reinforcement learning perform complex tasks. The method shown achieve significant results in the Minecraft
16

---

AIP Assisted Human Walking Without a Simulator
Game, especially the crafting diamond tools, which were impossible for RL alone previously. The AR2-D2 (Duan et al., 2023) allows users record themselves manipulating objects, and the data is then used train real robot to perform similar tasks. Despite these efforts, these studies still depend on gathering large amounts of data, often requiring hundreds of hours of long sequences of video or episodes, for RL to effectively converge. There is a significant gap in research addressing RL applications that operate under limited data conditions, particularly those with fewer than a few hundred state transitions. Another compounding factor that significantly complicates the problem in data-scarce environments is the suman-in-the-lowe effect, which is difficult to model or build a simulator for the human-robot interacting dynamics. These issues remain largely unexplored.
Offline to Online RL Offline-to to-llee RL has been explored through various approaches. (Zhang et al., 2023) introduce policy regularization via expansion and implement a Boltzmann action selection strategy. Several studies, including (Lee et al., 2022), (Zhao et al., 2023a), and (Zhao et al., 2023b), utilize an ensemble of pessimistic value functions to mitigate distributional shift, which serves as an implicit form of conservatism. (Nakamoto et al., 2024) propose training an additional value function to address over-conservatism issues caused by the initialized value function in the offline phase. (Niu et al., 2022) develop a dynamics- aware policy evaluation scheme to bridge the dynamic gap between source and target domains. (Li et al., 2023) introduce a policy regularization term for trust-region-styy -tylle updates. (Lei al., 2023) use an on-policy optimization method that unifies both offline and online training without extra regularization. However nearly all of the current offline to online RL collects the offline data based on extensive simulations with millions of samples and none of them have demonstrated directly learning from physical environment using LIMITED data for an offline policy, Our AIP approach is one of the first realization that does not require simulator to provide offline data and We developed our RIIV-based, data centric approach not only reduces intra-person step length variations but also reduces inter- seter-person variation. Because of our innovative data-centric approach, plus our use of well-established methods (BC and dHDP), our AIP approach demonstrated its impressive performance as we have reported.
dHDP algorithm. The dHDP is one of the carliest deterministic policy gradient methods (Si & Wang. 2001). This class of algorithms is considered more effective than popular methods such as PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018) for real-tine real-time continuous control applications. The dHDP is based on a fundamental principle that the actor adjusts the policy in the direction of the action-value gradient, and the critic updates the action-value function. This idea is considered a predecessor in a better known algorithm, the NFQCA (Hafner & Riedmiller, 201 1), which in turn, is considered a predecessor of the even better known DPG algorithm (Silver et al., 2014), which, as we know, has been further developed into good algorithms, such as DDPG and SAC. In a nutshell, the dHDP a bare-bone policy gradient method. All these methods be made more stable by integrating experience replay and target networks, which were first introduced in the deep Q-networks (DQN) (Mnih, 2013). A more detailed discussion dHDP and its performance evaluation in comparison to DDPG, as well as its several significant applications in complex and realistic engineering systems, such as Apache helicopter stabilization, tracking, and reconfiguration control (Enns & Si, 2003), power grid (Sun et al., 2012), chemical processes (Yang et al., 2021), fuzzy control system (Gao & Liu, 2016), Single-Axis Servo Mechanism System (El-Sousy et al., 2024), and more can be found in (Wu et al., 2024).
PPO and SAC are widely used in robotic learning based on extensive simulations or under structured environment. Both algorithms have shown great ability to learn stochastic policies by parameterizing a probability distribution over actions. These distribution-based policies facilitate exploration by sampling actions during training (Haarnoja et al., 2018; Schulman et al.. 2017). However, applying PPO and SAC in exosuit robot that does not have a simulator or a dynamic model of human-robot interaction, presents true challenges.
In wearable robotics, particularly with humans in the loop, safety and comfort are of paramount importance. Human participants become frustrated if the robot operates erratically. An analogy is how people feel when they learn to ski. Falling to the ground in arbitrary ways make them feel frustrated, a mental state that adversely affect their natural behavior or even cause injuries. A stochastic policy can become a source of causing discomfort to human users. In other real time control applications such flight control, we cannot afford erratic control actions either due to safety concerns. Perhaps because of this, to the best of our knowledge, all existing applications of PPO and SAC rely on simulators, which enable unlimited data collection, and controls are conducted in structured environments. The reliance on huge amount of data and the needs of safety assurance have significantly narrowed the pool of potentially feasible methods.
However dHDP has been shown convergent theoretically and demonstrated convergence under limited data conditions empirically in several wearable robotics applications (Wen et al., 20;b;;b; 2017a; 2019; Wu et al., 2022). Additionally, dHDP has been shown its compatible performance to DDPG in many DMC robotic control problems (Wu et al., 2024).
17

---

AIP Assisted Human Walking Without a Simulator
These nice theoretical properties and practical successful studies have given us the confidence deploy dHDP in our eurrent study, which directly learn the exosuit control solution with 5-dim state space and 4-dim action space.
In summary, given the track record of dHDP in human-robot locomotion control applications, and also to build initial trust in human participants for this new soft robot control problem, we took the first step of using dHDP to learn in real time for soft robot assisted human walking with reduced effort. This has helped us overcome the critical issue to not rely on a mathematical model or simulation model, which are nearly impossible to obtain (Brenneis et al., 2021). While this approach has met our expectation in the current study, we recognize the limitations or potential challenges as summarized in the Limitations section, and we plans to systematically explore and evaluate different solution architectures and algorithms in future studies.
Modeling of soft exosuit A pertinent example of these challenges can be seen in the develo and application of soft inflatable exosuits. On one hand, modeling a simulator for these devices is highly complex (Polygerinos et al., 2015). This is mainly due to the compliant nature of soft robots, which introduce physical properties that are difficult to model. For instance, several studies have shown that even obtaining a quasi-static model of torque for a soft actuator not trivial (Nesler et al., 2018; O' O'Neill et al., 2022). On the other hand, the dynamic interaction between the human wearer and the robot creates a highly coupled and complex system, making it even more challenging to model accurately (Zhu et al., 2022).
Control of rigid lower limb exoskeleton and powered prosthesis. Rigid robotic lower-limb exoskeletons and prostheses and their controls are being actively researched or even commercially available (Huang et al., 2021; Siviy et al., 2023; Shi et al., 2019). A typical control strategy of these devices often focus on mimicking the kinematics of biological joints via position control (Bortole et al., 2015; Long et al., 2017). However, another control strategy, referred to as finite state machine impedance control, is often preferred especially for consideration of achieving compliant lower limb behaviors. This stratagy provides safe human-exxskeleton interactions, as biological systems are capable of in order to adapt to various environments (Azocar al., 2020). Unfortunately, neither of the above two strategies are applicable to soft exosuit. For control actuation in either control strategy, rigid device control torques are generated by mechanical joint motor actuators, which can be directly used as a control parameter. In soft inflatables, however, the torque is generated from both human and the exosuit. Differentiating the two sources is difficult or nearly impossible. The natural control parameters for exosuit instead is the timing of inflation and deflation, which introduces additional delays to actuation and thus reduced stability margins have to be considered in the control design.
Personalized controls for individual users are common to all wearable lower limb devices, and human-in- in-the-loop (HIL) optimization represents several important design approaches (Koller et al., 2016; Zhang et al., 2017; Ding et al., 2018; Kim et al., 2019a; Bryan et al., 2021). They are used in open or closed-loop force or torque control to operate individual joints. In those applications. Bayesian optimization plays key role to provide optimal controls. These methods search for an extremum on the system response surface to determine the optimal control parameters. Although these methods can customize the control strategies or parameters, they are time-consuming and lack of adaptation. A small change of the wearer requires a re-design of the control (Tu et al., 2021). is noted that a large class of wearable rigid device controls aims at achieving reducing metabolic cost reflected by oxygen intake. However, it usually takes long walking time to be able to extract reliable measurements. This prohibits online and real time requirement that is highly desired for wearable robot applications. To overcome the limitations due to control strategy or optimizaiton method, data-driven reinforcement learning (Rls)-aased optimal adaptive control methods have been developed and successfully demonstrated for robot control of exoskeletons and prostheses. (Wen et al., 2019; 2017b; Li et al., 2021). In these applications, robot control relies on optimal cumulative cost/reward related to producing normative walking using directly measurable human-robot walking variables. However, these methods have been principally implemented in rigid robot devices (Huang et al., 2021), not soft inflatable exosuits, the control problem formulation different and the solution presents unique challenges due to discussions in the above.
C. Safety Constraints
The problem under investigation requires human physical safety and control system stability of the human-robot system. In this study, physical safety refers to that the human participants do not fall or endure injury as a result of robot control. This is ensured by imposing safety bounds to limit the soft suit inflation and inflating duration timing. The control system stability is in the same classical control sense, and we set one of the RL design objective in Equation 1 that the state regulation errors approach 0 (or practically error tolerance bounded). Our assurance of stability and safety is embedded in learning quantitatively, and guaranteed by analysis qualitatively, Our systematic data have shown that these constraints are met and
18

---

AIP Assisted Human Walking Without Simulator
objectives achieved.
To ensure human participants walk continuously and safely, we consider several safety constrains: 1) the soft actuator pressure is limited to 206.8 kpa; 2) the control timings and infiation/deflatioon duration are constrained by taking reference of those during participant's 's normative walking as shown in Table 4, which are within realistic ranges (Zhang et al., 2020b), These physical constraints help prevent significant misalignment between controller timing and the respective gait phase during human walking. Without these constraints, it may trigger soft actuator deployment and cause discomfort to the user, and 3) the online training objective is set for the control timings to approach those during normative walking, and thus in a safe state. Equation (11) renders such state constraint where the target state variables 5 [. d tc, de, 0] (Table 4) obtained from normative walking profile without soft actuator deployment to assist human walking:
Table 4.
Safety constraint for the control timing and target state for control regulation to reach.
Control
fi
d
f2
d2
Target State
dA
8g
dc
Safety constraints (% gait phase)
[0,20]
[0,20]
[60,80]
[0,20]
normal walking (% gait phase)
14%
15%
68%
15%
60"
6, = (s 8)²
(11)
where the respective tolerance for each state variable is as shown in Table 5. They represent realistic sensing and actuation errors inherent in physical systems, and they physically meaningful, human physiologically realistic, and validated in studies of human biomechanics such (Zhang et al., 2020b).
Table 5. Ranges of state error tolerance that are used in learning for achieving normative walking.
Error Tolerance
ta A
dB dB
tc tc
dc dc
0- 0
Tolerance Range
[-5%,5%]
[-5%,5%]
[-5%,5%%
[-5%,5%]
[0, 40 deg]
Table 6.
Tolerance values that ensure human normative walking.
State
ta
dB
tc
dc
0
inf value
10 %
10%
60%
10%
53
sup value
20 %
30%
75%
30%
780
D. Hardware Details
Soft inflatable actuators are designed to generate extension torque, fabricated from nylon fabric and thermoplastic polyurethane to ensure a transparent interaction with the user. When the knee is flexed and the actuators are inflated, they apply extension torque the knee joint. These actuators are strategically positioned in the popliteal fossa to aid in knee extension. Further design details of the soft inflatable exosuit can be found in (Sridar al., 2018; Sridar al., 2020).
The plectro-pneumatic system that controls the real-time inflation of the exosuit includes:
microcontrolller (Raspberry Pi), solenoid valves (MHE3-MS1H, Festo, Hauppauge, NY) for switching between inflation and deflation, pressure sensors AASDXAVXIOPGGAS, Honeywell International Inc., Morris Plains, NJJ) for monitoring the internal pressure of the actuators.
Kinematic data were collected using:
camera-based motion capture system (T40s, VICON Inc., Los Angeles, CA) sampling at 100 Hz.
EMG sensors (Delsys Trigno, Delsys, Natick, MA) were used to capture muscle activity, sampled at 2000 Hz. An instrumented treadmill (Bertec Inc., Columbus, OH) was used as the platform for the walking trials. The treadmill is equipped with force plates that measure the user's 's ground reaction forces at a sample rate of 2000 kHz.
An IMU motion capture system (Ultium Motion, Noraxon Inc., Scottsdale, AZ) was used to detect the maximum knee joint angle.
19

---

AIP Assisted Human Walking Without a Simulator
EMG sensors were placed on both legs over three muscles of interest: vastus lateralis (VL), biceps femoris (BF) and reclus femoris (RF). The raw EMG data were first band-pass filtered (Butterworth,, 4th order, 20 Hz and 450 Hz cutoff frequencies). The profile of the signal was obtained by computing the root-mean-square envelope using moving window of 250 ms. The integral of the envelope was computed for each gait cycle to quantify the overall muscular effort.
E. Experimentation, Hyperparameters and Implementation Details
We use PyTorch for all implementations. All results were obtained using our desktop with Intel Core 19-12900K proces- sor ww.rxperimeetttion. The experiments consisted of two sets of walking: offline, normative walking session with the exosuit attached but not inflated, and an online walking session with RL-controlled exosuit inflation and deflation. Data of one participant was recorded during offline walking for one experiment session. The offline session lasted around 10 minutes of about 170 steps. For the online human-robot collaborative walking, three sessions were performed for each participant. Each online session began with the controller initialized to the learned offline policy from IL, and lasted 10 minutes of about 150 steps. All participants walked at a constant speed of m/s on the treadmill during all experimental sessions, Safety constraints were imposed as discussed Section 3.3. Data Collection for Offiine Training. After a two-minute warm-up period for participant I get accustomed to the experimental setup and walking speed, data collection began. MoCap video data of the state variables in Equation (1) were collected. MoCap data were synchronized with the time sequences of the state variables to provide control target for offline training. Data Collection for Online Training. After two minute warm up period, the learned offline policy (refer to Section 3.4) was used for initializing the online RL controller for all participants. To mitigate environment noise and intra-human variance, a consecutive 5 steps were used to obtain one gait sample, resulting in a total of 30 gait samples. The RL policy update was performed for every gait sample. Performance Evaluations. Evaluation sessions were performed after online learning convergence upon meeting criteria (Table 5). Each participant rested for 5 minutes after online learning prior to evaluation which involves walking of 100 steps in about 6 minutes. Evaluation data were processed using similar procedures those used in processing online learning data.
E.1. Offline Training Procedure
The offline training consist with 200 episode. An Episode start with the first offline data in the dataset D the end of the dataset with total data points of 150. For each training trial, we use an off- policy exploration strategy, adding Gaussian noise ((,, 0.05) to each control. The algorithm hyperparameter for offline training is Table 7.
Hyperparameter
Value
Exploration noise
(o0 0.05)
Noise clip
t 0.5
Policy update frequency
2
Batch size
32
Buffer size
200
7
0.95
T
0.1
Adam Learning rate
0.001
Table 7. Hyper Parameters used for offline training
E.2. Online Training Procedure
For the online human-robot collaborative walking, three sessions were performed for each participant. Each online session began with the controller initialized to the learned offline policy from IL, and lasted 10 minutes of about 150 steps. All participants walked at a constant speed of I m/s on the treadmill during all experimental sessions, Safety constraints were imposed as discussed in Section 3.3. The algorithm hyperparameter for offline training is as Table
E.3. Network Structure and optimizer
The actor-critic networks in DHDP are implemented by feedforward neural networks with two layers of weights. Each layer has 256 hidden nodes with rectified linear units (ReLU) for both the actor and critic. The input layer of actor has the same dimension as observation state. The output layer of the actor has the same dimension as action requirement with tanh unit.
20

---

AIP Assisted Human Walking Without Simulator
Hyperparameter
Value
Exploration noise
(0, 0.01)
Noise clip
+0.1
Policy update frequency
2
Batch size
5
Buffer size
20
0.95
T
0.4
Adam Learning rate
0.001
Table Hyper Parameters used for online training
Critic receives both state and action as input to THE first layer and the output layer critic has 1 linear unit to produce Q value. Network parameters are updated using Adam optimizer with learning rate 10-3
E.4. Code
For Code and data, please visit http:///ghhhboooommmmmmisss.llct
F. dHDP solution and properties
To find thd dHDP solution, let the critic value as Equation 8 be Qo where 0 denotes the critic weights that are to be learned by using dHDP. Specifically, weight updates were performed to minimize the loss as function of the weights (0):
L(0)= Eamp, [(V Qa(sk, ux))²
(12)
where in the above, y denotes the critic target. Accordingly, the actor weights (denoted by (Ø)) are updated by applying the chain rule to the total return from the start distribution J with respect to the policy parameter (d):
V [ = Bo-p-a
V VuQa us) uk)) (oo VOTT(SH))
(13)
The update rules for the critic and the actor, respectively are:
- (V)(
(14)
x 0 + aV 1())
where a is the learning rate.
Here analyze and characterize properties of the learning process and the control performance, specifically those related to learning convergence, solution optimality, and stability as result of online dHDP learning. In the following, we express the exosuit control system with the following general nonlinear dynamics for the ease of discussion although this model is unknow, and our offline to online learning is completely data-driven.
8 = f(sk, ux). k = 0, 1,...
(15)
where 8 E 5 and E IR4 are defined in Equations 1, 2, respectively, k denotes discrete time steps.
The objective optimal control is to find a control policy that can stabilize system (15) and minimize the cost-to-go in Equation (8).
According to the Bellman optimality principle, the optimal cost-to-go satisfies the following relationship.
Q" (Sk, ux) = e& (Sk, 4+) 72" (SK+1, ($x+1)),
(16)
and the optimal control law n² can be expressed as
nt" (SL)
arg Q" (sk. uk), M4
(17)
21

---

AIP Assisted Human Walking Without Simulator
where Q" (sk; (g) is the state-action value function corresponding to the optimal control policy x² (sk)
We need the following definition and assumption develop results.
Definition 1. (Stabilizable System) A nonlinear dynamical system is said to be stabilizable on a compact set S2 E R", if for all initial states 80 E S2 there exists a control sequence ug, M1,-.., ",,,,,,,,,.......,, Wi.... such that the state Sk: - 8 as k + 00 where g a equilibrium point.
Assumption F.1. System (15) is controllable and stabilizable. The system state są = s is an equilibrium of the system under the control u4 = ग (sx) = uf for 8 - sf, i..., f(s", u²) = 8 The feedback control sequence uy determined from control policy æ represented by the actor neural network, and in the most general case is bounded by actuator saturation. Assumption F.2. The stage cost function eę (sk, finite, continuous 8g and ukk and positive semi-definite with Ck (sk, ux) = 0 if and only Sk s and uk u²
Note that the above assumptions are reasonable and realistic, as they are under the presumptions that a person who can use a exosuit assist walking can reach equilibrium state that they achieve normative walking while their muscle actívities reduced level less than that without wearing assistance.
As actor-critic method, dHDP solve the Bellman's 's optimality by learning to approximate both policy and value functions where actor refers to the learned policy and critic refers to the learned value. An actor-critic algorithm starts with an initial value, e.g., Qo u) = 0 and an initial arbitrary policy To. Then for i = 0, 1, 2, ..., it iterates between policy update and policy evaluation steps.
Qit1 (sk, ux) ck (sk; ux) + 7Q (Sk+1; as (8k+1)),
(18)
and
ass (8k) = arg min Qi (8k, ux). we
(19)
Or by combining (18) and (19), we have
Qi+1 (5ks us) = Cj. (5ki up.) + min Qi (SK+1, uk+1).
(20)
uk+1
Theorem F33 Let Assumptions F.1 and F.2 hold. Let Qi be the sequence of estimated Q values starting from Qo 0 at ith update of RL agent. For policy Tú its actor network weights are updated based on the policy gradient estimator (14), and the controls are bounded by the output function of the action network. Then
(1) Bounded: there is an upper bound such that 0 S Qi(sk, un) S for 1,2,
(2) Qs a non-decreasing sequence satisfying Qs(sk, ux) S Q++1 (sk, ux); Vi.
(3) Convergence: the limit of the sequence, Qoe (sk, ux) Iim Qi (Sk: uz), satisfies
Qoo (Sks wx) C& (Sk, un) + min Qoo (8841, u++1). ux-1
(21)
(4) Optimality: the Q-value sequence Q, (sk, Ux) and the corresponding policy my (sk), with To (sk) = lim Ti (sx). converge to the optimal value Q" and optimal policy x², respectively:
Too (sk) = π² (sk)s
(22)
Qoo (SN, ux) = Q" (sk, uz)
(23)
Proof. (1) Let (7())) be a deterministic control policy represented by neural network which is a continuous mapping from sy in stochastic environment E. Let (-) = O, and Z; be updated by
Zi+1 (Sk, ux) en (sk, ux) + 72 (SK+1, (8)
(24)
Thus, Zı (Sk. ux) = Ck (Sk. uz).
According to Lemma in (Gao et al., 2024), we obtain
22

---

AIP Assisted Human Walking Without Simulator
Zi41 (sks (k)
20 pl (Sk+s: 77 (Sx+))).
(25)
Σ rew (sk+s: (sk+s)) S
Er'ce j=0
If Assumption F.I holds, ep (sk+ uk+s) is bounded, there exists an upper bound Y such that
Σ Nen (1 SY,
(26)
According to Lemma in (Gao et al., 2024), as Qi+1 is the result of minimizing the right-hand side of (20), we have
Qi+1 (8k, ux) S Zit1 (Sk, us) S V.
(27)
(2) Define a value sequence 4 as
P1+1 (8k, ux) = ck (sk, 4/) + t (sk+1 T+t1 (sk+1)),
(28)
and Po = Qo= = In the following, a shorthand notation is used for o ( T(+1) = (5(51+1: (5++1)).
Since Po (8k, ) and Q1 (sk, u&) cs (sk: uk), and Cg is positive semi-definite under Assumption F.2,
P (sk: ux) S Q1 (Sk: ug).
(29)
From (18) and (28), we get
Tb (5k, uk.) Qi-1 (sks ux)
(30)
= (Q: (SK+1, Te) T.-1 (5)+1, 0.
Therefore,
$; (Sk, ux) S Qi+1 up)
(31)
Further by using Lemma in (Gao et al., 2024)
Qi (Sk, ux) S p. (sk, ux) S Qu+i (8k, ux).
(32)
This completes the proof of Theorem F.3 (2).
(3) From parts (1) and (2) in the above, Q; a monotonically non-decreasing sequence with an upper bound. Therefore, its limit exists. Let the limit be lim;-roc Qi (sk, ux) Qoo ux).
Given and for any u1+1, according to (18), there is
Qi (Sk, wx) S ca (sk, ux) + 7 (SK+1, uk+1).
(33)
As Q; is monotonically non-decreasing, we have
Qi-1 (SR, un) S Qoo (Sk. uk),
(34)
the following then holds
Qi (Sk, ux) S cb (sk, ux) + min Qoo (SK+1, u4+1). uo41
(35)
As 0o, we have
Qxe (Sk, wx) I ca (Sk, ux) + min Qoo (S841, 1x+1).
(36)
On the other hand, since the cost-to-go function sequence satisfies
Qi+1 (Sk, u+) c₁. (sk, uk) y min Qi (Sk+1, uk+1),
(37)
uo+1
23

---

AIP Assisted Human Walking Without Simulator
applying inequality (34) oc, Qoo (sk, ux) 2 ck (sk, ux) + min Qoo (SK+1, ux+1).
(38)
Mk-1
Based on (36) and (38), (21) is true. This completes the proof of Theorem E.3 (3). (4) According to Theorem E.3 (3) and by using Equations (18) and (19), we have
Pe (8k, u&) = ca (sk, u&) + min Qao ($K+1, ux+1)
(39)
ck ($ks ux) + +Q (Sk+1, Too (sk+1))
and
Too (su) = arg min Qoe (sks uk).
(40)
Wk
Observing (39) and (40), and then (16) and (17), we can find that (22) and (23) are true. This completes the proof of Theorem F.3 (4).
Theorem F4. Let Assumptions F.1 and F.2 hold, and Qi be the sequence of estimated Q values starting from Qo 0. For policy ne its actor network weights updated based the policy gradient estimator (14). If Q, converges to Q as n1 + Toc. then To is a stabilizing policy.
Proof. If Assumption F.1 holds, let ((sx) be stabilizing control policy, and let its cost-to-go A; be updated by the following equation from Ao() 0,
Ast1 (sk, ux) cb (sk, uz) + 7A: (86+1, (8841)),
(41)
We have
A (sk, ug) = Σ R R (sk+)))
(42)
j=0
Because u(sE) is stabilizing policy, if Assumption F.1 and F.2 holds, have st S" and ea (5k, ux) 0 as k co. Therefore, (sks, ux) + 0 as k + co.
Next, from Lemma in (Gao et al., 2024), n, minimizes Qi we have
Qi (Sk, up) S A; (Sk, up)
(43)
Since A,(sk, ux) + 0 as h + co, we have (sk, ux) + 0 as k - ac.
From Theorem F.3 (3), we obtain ex (8s, ux) = 0 as k + co. Further, under Assumption F.2, ce(8R, ux) = 0 if and only 8g = s", we have $; + sf as k + co. This completes the proof.
G. Addressing OOD Problem
A primary challenge during online training caused by significant actuator delay associated with soft actuator inflation and deflation. Specifically, there is an approximately 0.2-second delay to fully inflate and 0.25-second delay to deflate. These delays could not be adequately captured during the offline imitation learning phase as offline policy was obtained without exosuit control. As shown in Figure 4, to compensate for the inflation delay, the control variable f1 was significantly shifted an earlier onset, allowing the system to anticipate the slower actuator response time. Similarly, to mitigate impact of deflation delays. the duration variables and de were substantially shortened, ensuring that the system could maintain synchronization with the human walking pattern. These adjustments were critical aligning the actuator responses with the real-time dynamics of human movement, thereby enhancing the overall effectiveness of AIP by achieving normative walking with reduced effort while all safety constraints are met.
H. Human Adaptation
24

---

AIP Assisted Human Walking Without Simulator
030
0.30
08
10
10
0.55
2.3
8.35
14
3MO 3ttm
0.15
8:5
10
0.10
60
01/55
005
Figure 7. EMG profile and Characteristic timings and durations of gait trials Baseline: without actuating the exosuit. Under the offiine policy obtained from offline training. C) Under trained online policy obtained after online learning convergence. The results row B) illustrate how participants responded and adapted to offline-trainee policy. However, their adaptations show no clear or consistent trend toward reducing EMG effort. In contrast, the human-robo co-adaptation process during online lcarning led to consistent reduction EMG effort, as demonstrated Figures 3 and 4. shaded regions represent the 95 confidence range of the three experiment trials. The x-axis is the number of gaits. The results in row A) included one trial since there is no control assistance.
610
075
13
15
29
2779
6.22
1175
K15
510
Figure 8. Characteristic timings and durations of gait trials: a) Raw gait data in seconds during training. b) Respective RIV-processeed data. c) RL policy during training. shaded regions represent % confidence range three experiment trials. The x-axis the number of gaits.
25

---

AIP Assisted Human Walking Without Simulator
Actual Timing Experiment Data Between Trained Offline Policy And Trained Online Policy Evaluation
Trained Online Policy Evaluation
Human Stance Gait Kinematics (Equ 1).
Human Swing Gait Kinematies (Equ 1)
Entire Gait Duration
6
0
de
Entire Gait Duration
Robotic Stance Control (Equ 2)
Robotic Swing Control (Equ 2)
Trained Offline Policy Evaluation
Oilline
5
5 de
Eg
Ey 6x
66
4
1:
dx
Er
05
6.
47
d
4.
1)
d5
11
0.5
1.0
1.5
08
0.5
1.0
0.0
1.0
0.00
0.25
0.50
0.75
1.00
0.5
188
20
line 64)
lime())
lime (4)
(inz())
time (6)
Explore the Roboaic Co-adaptation
From the Perspective of Timing Differnce Between RL Policy and Human Gait Kinematic
Positive:
Difference Between RL Control and Gait Kinematics Timing
0.20
Trained Offline Policy Evaluation
Robot Control More behind
Difiorenco (1)
Difference Between RL Control and Gait Kinematics Timing
Actual Human Gait Kinematic Timing
Trained Online Policy Evaluation
Negative:
0.00
-0.05
Robot Control More Ahead
-0.10
66x
6d4
6tc
6dc
Actual Human Gait Kinematic Timing
P1
P2
P5
0.2
0.15
02
Differened
Ditiernned
Time Differenor ()
Time Diflermor ())
0.10
Diffeence G3
0.1
6.05
0.1
0.00
00.05
-0.65
-0.1
-011
8ts
6d,
8tc
8dc
80c
6d.
6dc
8ta
8d/
6tc
8dc
86s
Eds
8tc
0d:
80.
5dl.
0Cc
6dc
Figure 9. Row A represents actual experimental data from evaluations of both the offline-trained policy and the online-trained policy. Row B: explores human-robot co-adaptation by analyzing the timing differences between the controller and human gait. specifically: Specifialy, A duration after online training increases to accommodate inflation/deflation (time takes to actuate the airbag) as row A) shows that dashed orange line is longer than solid blue line. 2) Preserved Typical Human Gait: The duration of human stance d de changed very little indicating that participants continue walking naturally with their typical gait patterns. 3) Stance Onset Timing fı: as Row B) shows, that for majority of the participants (4 out of 5), RL adjusted the control to align more elosely with the human stance onset timing t And one participant, the control onset shifted slightly earlier. 4) Swing Onset Timing l2: consistently shifts earlier than the human's 's actual swing starting time lc. ensuring actuators inflate in time assist lifting the leg. The offline policy had attached actuator but the absence of actuator deployment during training. However, if this onset is too carly, could hinder natural knee hence, online learning automatically adjusted strike balance between assistance and preserving gait mechanics.
fA, 5d. di d.A. ôtc to te, and 5dc de de. On Human Robot Co-adaptation: 1) The gait
earlier lz, likely due the
26

---

AIP Assisted Human Walking Without Simulator
I. Incline Task
Cycle
11e0 Xin (0m0)
15%
20%
33%
45%
50%
00%
75%
83%
90%
100%
Peeteet Golt Cydle
1.0
Pariccppnt
1.2
3.0
Poak
0.4
0.2
0.4
15 Galt
25
()
Figure 10. A) Degree of Incline Walking B) Walking Profiles vary over incline angles C) Complete Online Training Results 7 Degree Incline Walking with 2 Participants. The shaded regions represent the 95 % confidence range of the three experiment trials. The x-axis is the number of gaits. The dashed line in the EMG effort panel represent the BASELINE EMG effort when there is no control of the device.
27


Reviews:
[{"review_id": "iVlYpFOIKP", "reviewer": "Reviewer_tqYD", "summary": "This paper presents AIP (Adaptation from an offline Imitating expert Policy) for controlling a soft inflatable exosuit to assist human walking without relying on a simulator. The approach first learns from human walking demonstrations (offline phase), then adapts this policy online to personalize assistance. The method is validated on five participants, demonstrating reduced muscle effort while maintaining normative walking patterns.\n\n## update after rebuttal\nI acknowledge the authors' rebuttal and maintain my original assessment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for thier thoughtful feedback, please check our [new results](https://www.dropbox.com/scl/fo/rgonc4oohtzgf87jqlq3y/AKOxgA5jW9PHt3NKRFLAPcw?rlkey=04igqadzdmyojb9y48zlds5gf&st=g4pizvbd&dl=0) \n>Q1 Isolating the benefit of co-adaptation\n\nTo address the issue of isolating the benefits of co-adaptation, please refer to Fig 4, 7, and the newly added results (Fig 9) from the same previous experiment data.\n1. human-robot co-adaptation is coupled as the human brain sensorimotor system governs their behavior during robot assisted walking. Isolating human adaptation is not feasible for the current study. But human adaptation reflected by human walking behavior naturally emerges in response to robot learning control.  \n2. To isolate the effect of co-adaptation, the only thing we could do was to disable the robot’s adaptation by freezing the policy updates—this setup is shown in Fig 9.\n3. For Fig 9, in row A (baseline), participants walked naturally and showed their natural gait pattern; in row C (after online training), the policy is already optimized, so no further human adaptation is needed. In both cases, we observe no sign of human adaptation.\n4. In row B (learned offline policy), the Offline policy is not optimal and needs to be personalized. This is where we speculate that the participants may recognize the mismatch and attempt to adapt, but since there is no clear or consistent direction, the participants’ responses appeared without clear pattern or trend. For example, P1 shows delayed timing in $t_A$, while others do not exhibit this behavior. Similarly, P5 demonstrated a slightly earlier duration in  $d_C$, which was not seen in other participants.\n5. Evidence of co-adaptation during online learning was clear. Fig 4 and 7 further support this: humans slightly adjust gait duration to accommodate actuator inflation/deflation, while the RL controller adapts its onset timing. This coordinated adaptation is evident in the bar plots of Fig 4 (revised presentation is now Fig 12), showing significant changes in timing and duration between human gait kinematics and robot control.\n\n>Q2 paper’s true contribution is in the practical implementation rather than novel algorithms.\n\nYes, the offline-to-online learning framework is well-established in the robotics community. Additionally, our proposed RIIV is not limited to the specific locomotion task presented in this study. The core principles of RIIV are broadly applicable to other human–robot locomotion scenarios (For example, our newly added experimental data involves walking on an inclined treadmill at 7 degrees, please refer to Fig 8).\n\n>Q3 when simulator-free learning becomes advantageous VS simulatior-based learning\n\n1. We thank the reviewer for this excellent question. In our case, a simulator is not available and therefore, it is not feasible for us to  quantitatively compare simulator-based and simulator-free approaches in this study.\n2. However, to shed some light, we may consider minimum fidelity requirements for simulations. A simulator can be beneficial if it meets key fidelity criteria—such as accurately replicating system dynamics and kinematics. The feasibility and usefulness of simulation, however, depend heavily on the application. For example, a cart-pole task requires much simpler modeling than a human-in-the-loop soft exosuit system.\n3. Even if a model were available to simulate our system’s dynamics, deploying it for real-time control would remain highly challenging as in the well-known sim-to-real gap in reinforcement learning\n4. In contrast, our simulator-free approach directly engages with the real-world complexities without relying on approximations. However, it comes with its own challenges—particularly the need for careful problem formulation and data-centric design (e.g., our RIIV method) to ensure efficient and effective learning while capturing the invariant dynamics of the human-exosuit interaction.\n5. Ultimately, the advantage of simulator-free learning lies in its realism and adaptability, while the benefit of simulation\ndepends on the availability of a high-fidelity model. In our context, simulator-free learning is a practical and effective choice, and perhaps, the only choice now.\n\n>Q4 Roadmap for more complex scenarios\n1. We agree and our current roadmap focuses on generalizing the method beyond basic treadmill simulated level-ground walking  to include more complex and realistic tasks and scenarios, including walking at variable speeds, inclined and declined walking, among others.\n2. As a first step of extension, we have begun developing/validating the approach on inclined walking. Please refer to the newly added Figure 8, which shows results from a 7-degree incline treadmill walking. As shown,  increasing the incline significantly alters gait kinematics. However, as shown in Panel C, our AIP method remains robust, consistently reducing EMG effort below BASELINE levels across all participants.", "claims_and_evidence": "*Key Claim:* AIP enables effective control of soft exosuits without a simulator\n\n*Evidence:* The authors demonstrate successful implementation on five participants with consistent performance improvement. Their approach successfully adapts to individual differences without requiring a complex simulator - which would be difficult to develop for this application due to the unpredictable human-soft robot interaction dynamics.", "methods_and_evaluation_criteria": "The offline-to-online approach with RIIV data processing addresses the key challenge of working without simulation in human-robot learning. Performance metrics comprehensively capture both technical performance (stage cost, convergence time) and human factors (EMG reduction, knee error). The participant pool of five individuals provides reasonable diversity for initial validation.\n\nThis work would be a good reference for practical reinforcement learning for physical human-robot interaction, especially in scenarios where accurate simulation and large expert data collection is infeasible.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "*Offline-to-online transition methodology:* The authors' approach of using a single participant's data for offline learning and then testing generalization to other participants is valid for demonstrating adaptability. The experimental protocol included appropriate warm-up periods and rest periods between sessions to minimize fatigue effects.\n\n*Baseline comparisons:* The comparison against previous reported EMG reductions (7.37%) provides context for their 20% reduction. However, a within-study baseline comparison showing human adaptation alone would have strengthened the experimental design by better isolating the benefits of the co-adaptation process.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**\n- Addresses the critical challenge of developing controllers for soft exosuits without simulators, which are nearly impossible to create for these complex human-robot systems.\n- Demonstrates successful direct learning in a physical environment with human subjects, a significant achievement in reinforcement learning.\n- Shows robust personalization across participants with measurable benefits (20% EMG reduction), outperforming previous methods (~7.37% reduction reported in comparable studies).\n- The participant diversity (varying height, weight, gender) is appropriate and exceeds similar studies in the field.\nIncludes supplementary videos demonstrating walking with the exosuit, strengthening the empirical validation.\n\n**Weaknesses**\n- The method relies on established RL algorithms (BC and dHDP) rather than introducing algorithmic innovations, though the paper's true contribution is in the practical implementation rather than novel algorithms.\n- The ablation study could provide deeper insight into the specific conditions under which simulator-free learning becomes advantageous versus when simulations might still be beneficial.\n- While the authors address population diversity adequately for this study, a more explicit roadmap for extending to more complex scenarios (variable walking speeds, different terrains) would strengthen future applications.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfiEK4HpuP", "reviewer": "Reviewer_Q99a", "summary": "This paper presents an RL-based control framework for a soft exosuit that assists human walking without a simulator. The proposed Adaptation from an offline Imitating Expert Policy (AIP) approach learns from human walking demonstrations and refines control using dHDP. AIP prioritizes data quality over large-scale simulations, enabling real-time personalized assistance. Tested on five participants, the method reduces muscle effort while maintaining normative walking patterns, demonstrating learning convergence, stability, and optimality.\n\n## update after rebuttal\nThank you for the thoughtful rebuttal. I appreciate the effort to address my concerns. I will maintain my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for thier thoughtful feedback, please check our [new results](https://www.dropbox.com/scl/fo/rgonc4oohtzgf87jqlq3y/AKOxgA5jW9PHt3NKRFLAPcw?rlkey=04igqadzdmyojb9y48zlds5gf&st=g4pizvbd&dl=0) \n>Q1 Compare with other methods\n\nWe thank the reviewer for the suggestion. In response, we have added a comparison Table 9 summarizing related work that\nuses EMG as a performance metric:\n1. There is no simple intuitive heuristic control and as shown in the Table 9, rule-based methods consistently underperform compared to automatic control approaches. This is because they rely on expert-defined heuristics and lack adaptability to individual variations. The only known method for knee assistance, proposed by Sridar et al., requires manual tuning and achieves only a modest 7.37% EMG reduction. In contrast, our AIP method achieves an average 20% reduction in EMG, requires significantly less time, and learns automatically through reinforcement learning.\n2. Some related approaches, such as Bayesian optimization (BO), rely on probabilistic models or approximations of the objective function. These introduce early approximation errors and struggle to extrapolate effective policies from the limited state-action data available in our study. Furthermore, model-based methods are generally impractical for soft exosuits due to the difficulty of accurately modeling their highly compliant and nonlinear dynamics.\n3. This further motivates our use of RL, as it enables us to collect human data for offline learning and extract an initial policy that can be used to effectively initialize online adaptation.\n\n>Q2 Extended Use Issue\n\n1. As this is the first application of its kind, long-term testing across multiple days or in uncontrolled, real-world environments falls outside the scope of this study—consistent with prior work in the field (see Tables 9 and 10).\n2. However our experimental design offers insight into stability over extended use. Each participant completed six cycles of 10 minutes walking followed by 5 minutes rest, totaling 7.5 hours for all participants. Refer to newly added Fig 9C, the AIP policy consistently reduced EMG effort with minimal variation, suggesting stable performance post-training.\n3. We expect AIP to maintain performance once converged. Potential timing drift due to actuator wear is inherently accounted for, as all five participants used the same hardware. Any degradation would have been captured during training. Moreover, our pneumatic actuators are easily replaceable, supporting routine maintenance and long-term deployment.\n4. Generalization to real-world conditions aligns more with industrial-scale evaluation. While outside the current scope, we believe AIP’s model-free, data-driven approach can generalize effectively, provided additional training data is collected in those settings.\n\n>Q3 Participant and Gait Diversity\n1. Our participant pool included both males (3) and females (2), with a weight distribution of 69 ± 11.6 kg and height of 1.63 ± 0.09 m, demonstrating equal or greater variability in key physical characteristics compared to related studies in Table 10. \n2. As shown in the newly added Fig 10, we present a histogram illustrating the range of gait patterns observed in our study. Based on established tolerance bounds from prior bio literature, our participants’ gait patterns span a broad spectrum within normative walking ranges. This indicates that our study captures a wide range of commonly observed gait variations.\n3. Additionally, since this is the first study of its kind—with no prior precedent for the proposed method—IRB approval was essential to ensure participant safety. Including individuals with extremely diverse body types could introduce safety concerns, such as improper fit or reduced actuator effectiveness. \n4. To further examine gait diversity and robustness, we introduced new results involving treadmill walking on a 7-degree incline, where the AIP method continues to perform robustly. For details, please refer to our response to Reviewer tqYD, Q4.\n\n>Q4 Computational Requirements\n\nYes, our method runs effectively on low-power edge devices. For real-time deployment, we use a Raspberry Pi 5 microcontroller, with power consumption ranging from 2.7 to 12 watts. For details, please refer to our response to Reviewer prNK, Q7.\n\n>Q5 Subject Feedback\n\n1. During training, we asked participants every five steps, “Do you understand the soft suit’s behavior?” \n2. Participants’ awareness improved gradually. Within the first 20 steps, they commonly responded, “I don’t understand” By 50 steps, they reported, “I kind of know” and by 100 steps, most expressed, “I am fully aware”.\n3. There are two directions to improve this process. First, enhancing the sample efficiency could reduce the initial \"confusion\" phase. Second, adding real-time visual feedback on when and how assistance is being applied—could help participants understand suit’s behavior more quickly.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. Section F. dHDP solution and properties in the supplementary material.", "experimental_designs_or_analyses": "Yes. The experimental designs are reasonable, however, it would be stronger if direct comparisons to alternative methods were provided, such as human-in-the-loop optimization, traditional rule-based control strategies, or a simpler heuristic baseline. Also, the experiment setup is simple now, it would be beneficial to explore how AIP adapts over extended use.", "relation_to_broader_scientific_literature": "This paper builds on prior work in wearable robotics, RL, and human-in-the-loop optimization, addressing the unique control challenges of soft exosuits. Prior RL-based exosuit control relied on sim-to-real transfer, whereas this paper eliminates the need for a simulator by combining offline imitation learning with online adaptation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. Simulator-free RL for exosuit control and offline-to-online. The proposed AIP framework eliminates the need for a simulator, addressing a major data limitation in prior RL-based exosuit control methods. The combination of offline imitation learning and online RL adaptation enables personalized exosuit assistance.\n2. Data-centric approach. This paper introduces Reducing Intra- and Inter-Person Variation (RIIV) to improve data quality for imitation learning.\n3. Convincing results. The experimental results demonstrate reduced muscle effort for human participants and fast adaptation.\n\nWeaknesses:\n1. Unclear long-term adaptation**.** The paper does not explore whether AIP remains stable over prolonged use or how well it generalizes to real-world walking conditions beyond a treadmill setup.\n2. Limited participant diversity – The study only includes five participants, primarily young, fit females and heavier males, which limits the generalizability of the findings. Older individuals, lighter males, and a broader range of body compositions could exhibit different gait patterns and adaptation responses, affecting the model’s robustness.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What are the computational requirements for the real-time deployment? Could the method work on low-power edge devices?\n2. How stable is the method over multiple days of use? Does the learned policy continue improving over time, or does it plateau?\n3. What are the biggest challenges in scaling your method to a larger, more diverse population?\n4. Did you collect any subjective feedback from participants regarding their comfort, ease of movement, or perceived assistance from the exosuit? If so, what were their suggestions for improvement?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PkfcbspZbA", "reviewer": "Reviewer_prNk", "summary": "This paper presents an innovative approach to controlling soft exosuits for assisted human walking using reinforcement learning (RL) without relying on a simulator. The authors propose an online Adaptation from an offline Imitating Expert Policy (AIP) approach that addresses key challenges in RL-based control of physical devices, including limited data, absence of simulators for human-robot interaction, computational constraints, and the need for personalized control while ensuring safety. The method first uses offline learning to mimic human expert actions through real walking demonstrations without robot assistance, then initializes online actor-critic learning to optimize personalized robot assistance. The authors demonstrate their framework on five human participants, showing consistent performance improvements in terms of reduced human effort while maintaining normative walking patterns. The paper also provides qualitative performance guarantees for their online RL method, including learning convergence, dynamic stability, and solution optimality.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for thier thoughtful feedback, please check our [new results](https://www.dropbox.com/scl/fo/rgonc4oohtzgf87jqlq3y/AKOxgA5jW9PHt3NKRFLAPcw?rlkey=04igqadzdmyojb9y48zlds5gf&st=g4pizvbd&dl=0) \n\n>Q1 Compare with sota method\n\nWe thank the reviewer for the suggestion:\n1. We have added Table 9, which summarizes related work that uses EMG as a performance metric. Our RL-based method achieves an average EMG reduction of 20%, significantly outperforming rule-based methods such as the one reported by Sridar, which achieved only 7.37% reduction.\n2. The model based method and Bayesian optimization required probabilistic models or approxiamation of the objective which\nis not suitable for our problem. \n3. Please refer to Reviewer Q99a Q1 for further detailed explainations.\n\n>Q2 Long Term effects\n\nAs the application is the first study of its kind, long term study falls outside the scope of this study. However:\n1. A potential challenge for long-term deployment is actuator timing drift due to wear and tear. In our study, all five participants used the\nsame exosuit hardware across sessions, so any natural degradation—such as material fatigue or delay shifts—was implicitly\ncaptured during learning. In such effects occurred, AIP would have adapted to them as part of the online training process\n2.  As shown in the newly added Fig 9 C), the AIP policy consistently maintains reduced EMG effort across all evaluation sessions with minimal variation. \n3. Please refer to Reviewer Q99a Q2 for related comments.\n\n>Q3 Generalize and challenges of more complex tasks\n1. As the first study of its kind, our work focuses on level-ground walking at a constant speed—a standard and widely adopted practice in the wearable robotics community. As shown in Table 9 and 10, related studies have also primarily evaluated their methods under similar conditions. This controlled and repeatable environment serves as a foundational step for developing and validating the proposed AIP method before extending it to more complex locomotion tasks.\n2. Regarding generalization of AIP, we introduced new results involving treadmill walking on a 7-degree incline, where the AIP method continues to perform robustly. Please refer to Reviewer tqYD, Q4 for related comments.\n3. Real-world locomotion includes diverse tasks such as stair climbing and uneven terrain. While the core AIP principle remains applicable, these scenarios introduce challenges such as greater variability in human motion, the need for terrain-aware sensing, and potentially longer training times due to increased dynamic complexity.\n\n>Q4 Apply to other applications\n\n1. The formulation of the state, action, and cost need to be adapted for different tasks.\n2. For RIIV, captures the invariant feature of application and can potential inprove sample efficency. In other human-robot interactive systems, the principles of RIIV should still apply to identifying invariant task-related characteristics.\n3. To extract the offline policy, we used a motion capture system commonly adopted in human-robot locomotion studies. \n\n>Q5 Gait pattern varibility\n\nPlease refer to Reviewer Q99a Q3 for related comments. \n\n>Q6 Limitation and Transfer learning\n\n1. Our method currently controls a single leg. Extending to bilateral control introduces challenges in coordinating and synchronizing assistance across limbs. Multi-agent RL offers a promising direction to address this open question.\n2. Data efficiency is a known limitation of RL. We agree that transfer learning is a promising direction and we are actively exploring this approach, as it can accelerate convergence and reduce user-specific data needs. \n\n>Q7 Computational Complexity\n\n1. The computational complexity of our online learning method is similar to that of DDPG, as both use the vanilla deterministic policy gradient framework. Our actor and critic networks have two fully connected layers with 256 hidden neurons, and a small batch size of 5 is used during online training. Based on the complexity estimate $O( Batch * layer * neuron^2)$, the method remains lightweight and efficient.\n\n2. RIIV is a data-centric method with low computational complexity of $O(state)$, designed to minimize unnecessary input variance. This alignment improves data efficiency (Fig 2) and supports stable learning with compact networks and small batch sizes—without compromising performance.\n\n3. For real-time deployment, we use a Raspberry Pi 5 microcontroller, with power consumption ranging from 2.7 to 12 watts.  It has consistently supported both control and inference in real time across all five participants, with no observed performance issues.", "claims_and_evidence": "The claims made in the submission are well-supported by evidence. The authors claim that their AIP approach can effectively control a soft exosuit to assist human walking without a simulator, which is demonstrated through experiments with five human participants. The claim that online adaptation can personalize the control for individual users is supported by the consistent reduction in EMG effort (muscle activity) across all participants after online training, despite initial variations. The paper also claims that their data-centric approach is more effective than algorithm-centric approaches for this application, which is supported by comparing their RIIV (reducing intra-person and inter-person variation) method with a benchmark direct normalization approach, showing improved action divergence and better alignment with ground truth data.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand. The authors use a combination of offline imitation learning and online reinforcement learning, specifically direct heuristic dynamic programming (dHDP), which is well-suited for the constraints of real-time control with physical devices. The evaluation criteria include stage cost, peak knee error, and EMG effort, which effectively capture both the quality of the walking assistance (normative walking) and the reduction in human effort. The authors also consider important practical aspects such as time to convergence of RL online learning and action divergence to measure offline policy optimality. The experimental design involving five human participants provides a reasonable sample size for demonstrating the effectiveness and generalizability of the approach.", "theoretical_claims": "The paper makes several theoretical claims about the properties of their learning process and control performance, including learning convergence, solution optimality, and control system stability. While the main paper provides a high-level overview of these claims, the authors reference detailed proofs in the appendices. The theoretical framework appears sound, building on established principles from reinforcement learning and control theory. The authors acknowledge the challenges of controlling soft inflatable actuators due to their nonlinear nature, material properties, and manufacturing variations, and provide a theoretical analysis that accounts for these factors.", "experimental_designs_or_analyses": "The experimental design is robust and well-executed. The authors test their approach on five human participants walking on a treadmill at a constant speed, with clear safety constraints to ensure participant comfort and safety. The experiments include both offline data collection using a Motion Capture system for ground truth and online learning using IMU sensors for real-time control. The analyses include quantitative comparisons of performance metrics before and after online learning, as well as ablation studies to understand the contribution of different components of the cost function. The results are presented clearly with appropriate statistical measures, and the authors do not selectively present results, which enhances the credibility of their findings.", "relation_to_broader_scientific_literature": "The paper effectively situates its contributions within the broader scientific literature on soft exosuit control, reinforcement learning for robot control, and offline-to-online RL approaches. The authors acknowledge prior work on human-in-the-loop optimization methods for exosuit control and highlight the limitations of these approaches in terms of requiring robust dynamical models and lacking online tuning capabilities. They also discuss recent advances in offline-to-online RL and note that most existing approaches rely on extensive simulations or vast amounts of offline data, which are not available in their application. The paper's data-centric approach represents a novel contribution to the field, addressing a gap in the literature on RL control of physical devices without simulators.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The paper addresses a significant practical challenge in controlling soft exosuits for human assistance, with potential real-world impact in rehabilitation and assistive technologies.\n- The approach is pragmatic, focusing on data quality rather than algorithm complexity, which is particularly valuable for applications with limited data and computational resources.\n- The experimental validation on five human participants demonstrates the robustness and generalizability of the approach.\n- The paper provides both theoretical guarantees and empirical evidence for the effectiveness of the proposed method.\n\nWeaknesses:\n- The paper could benefit from a more detailed comparison with state-of-the-art methods in exosuit control, beyond the ablation studies presented.\n- The long-term effects of the proposed control approach on human adaptation and learning are not explored, which would be valuable for understanding the practical implications of the technology.\n- The paper focuses primarily on level ground walking, and it's unclear how well the approach would generalize to more complex locomotion tasks such as stair climbing or uneven terrain navigation.\n- The sample size of five participants, while reasonable for a proof-of-concept, may not be sufficient to capture the full range of human variability in gait patterns and responses to exosuit assistance.", "other_comments_or_suggestions": "Other Comments Or Suggestions\n- The paper would benefit from a more detailed discussion of the limitations of the approach and potential directions for future work.\n- A more comprehensive analysis of the computational requirements and real-time performance of the proposed method would be valuable for understanding its practical applicability.\n- The authors could consider exploring the potential of transfer learning between participants to further reduce the online learning time for new users.\n- The paper mentions that the approach could be applied to other wearable robotics applications, but does not elaborate on how the method might be adapted for different types of devices or assistance tasks.", "questions_for_authors": "- How does the computational complexity of your online learning approach compare to other RL methods, and what are the implications for real-time control on embedded systems with limited resources?\n- Your experiments focus on level ground walking at a constant speed. How do you envision extending your approach to more complex locomotion tasks or varying walking speeds, and what additional challenges might arise?\n- The paper demonstrates co-adaptation between the human and the robot during online learning. Have you observed any longer-term adaptation effects in human participants after extended use of the exosuit with your control approach, and how might these effects influence the design of adaptive controllers for wearable robots?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "RBIU1uBjrV", "reviewer": "Reviewer_QtzX", "summary": "This paper develops an RL application for controlling soft wearable exosuit for human normative walk. Grounded in the motivation that this type of system lacks robust simulators or dynamic models, the paper approaches it from a model-free RL perspective. Furthermore, given the natural lack of data from this problem, the paper leverages the traditional approach of initializing a policy via Imitation Learning from pre-collected trajectories and then applies a RL fine-tune to optimize a notion of costs that encourages safe normative walk and reduced EMG efforts. Based on experiments conducted with five healthy patients, the paper claims that the proposed system improves personalized walking assistance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for thier thoughtful feedback, please check our [new results](https://www.dropbox.com/scl/fo/rgonc4oohtzgf87jqlq3y/AKOxgA5jW9PHt3NKRFLAPcw?rlkey=04igqadzdmyojb9y48zlds5gf&st=g4pizvbd&dl=0) \n\n>Q1 Fig.2 concern\n\n1)  DIRECT & RIIV are different methods resulting in different physical scales. Therefore, Fig2-left (the learning curve) should only be used to monitor learning convergence, but not for quantitative compassion of performance. For example,  consider only 1 non-zero error in eqn(6) be the swing timing $t_c =0.1$, then in physical time, RIIV error=0.0198s and DIRECT error=0.0946s.  Please note that, the other 4 plots to the right provides more details on compassions of different methods.   \n2)  We have included a revised Fig 4 (now Fig 11). Confidence interval is presented for Fig4-left. Note that, the other plots in Fig. 4 are shown in log scale, which does not reflect physical values of std. We therefore have included a new Table 8 to report actual STDs for each variable.\n\n>Baseline and Isolate co-adapation\n\n1) We kindly ask the reviewer to refer to Fig 3 and its caption where each participant has a dashed line indicating the baseline EMG effort during natural human walking without exosuit assistance.\n2) For details on isolating effects of robot and robot adaptation, please refer to Reviewer tqYD Q1.\n\n>Fig 4 concern\n\nWe have a new Fig 12. Plz review carefully, especially the caption on coadaptation.\n\n>Fig 5,6 concern\n\nAs reviewer suggested, we include a new Fig 13. Since each term in the overall cost differ physically, so are their scales. The results reported here are based on actual values without any pre-/post- process. The purpose of Fig 13 is to show how learning with only a partial objective (EMG or kinematic error alone) fails to guide effective adaptation to achieve normative walking with reduced EMG effort. This is what ablation study is designed for. \n\n> Challenge of the study and Gait veriation\n\nOur problem presents unique and significant challenges that distinguish it from standard simulated tasks:\n1. Unlike benchmark env based on rigid-body systems with reliable simulators, our system involves a soft exosuit interacting with a human, introducing complex, and time-delayed dynamics. No existing simulator can accurately model this interaction, making our simulator-free, real-world learning approach both necessary and significantly more challenging.\n2. Benchmark tasks offer well-defined state/action spaces and objective functions. In contrast, our study tackles a first-of-its-kind real-time control problem, where defining meaningful states, actions, and cost functions requires substantial innovation—making problem formulation itself a core contribution of our work.\n3. Regarding the gait variation, please refer to our detailed response in reviewer Q99a Q3.\n4. We appreciate the reviewer’s recognition that expert demonstrations help address the data challenge and note that, to our knowledge, no prior work has applied RIIV or this offline-to-online pipeline in soft wearable robotics—making it a novel contribution of our study.\n\n>Complex tasks and real world application\n\nPlease refer to our detailed response to reviewer prNk Q3\n\n>Appendix I, centering issue and typos\n\nFigs in Appendix I are discussed in the main paper. Fig 5 and 6 in the ablation study, and Fig 7 in the co-adaptation. Typos will be corrected, thanks.\n\n>Safety Constraints selected\n\n1. Table 5 presents biomechanics data that define the realistic range of normative walking patterns [Zhang et al., 2020b]. \n2. Based on this reference, the control safety constraints in Table 3 were established. In particular, we account for the actuator’s deflation delay that too long duration may raise safety concern so we cap it to 20% gait phases. The target states used in our controller correspond to the most commonly observed values within normative walking.\n3. Table 4 outlines constraints in the RL framework to ensure participants maintain normative walking and are not exposed to unsafe or misleading actuator behavior.\n\n>human, manual intervention of OOD\n\nThere was no human intervention or cherry-picking throughout the study—all results reflect true experimental data from all five participants. The adaptations were learned automatically by the RL controller during online training. In the offline phase, the exosuit was inactive, and data were collected from unassisted walking, meaning no actuator-induced effects were present. As shown in Figure 9A–B, activating the actuators led all participants to adjust their gait, introducing a distribution shift and potential OOD issue. Through online learning, the RL policy adapted to this shift as part of a natural human-exosuit co-adaptation process, without manual tweaking. As demonstrated in Fig 12, this adaptation was crucial for real actuator behavior and achieving normative gait patterns with reduced muscular effort.", "claims_and_evidence": "- The RIIV “method” improves data quality for learning the exosuit walk controller.\n\t\nEvidence: The work brings as evidence the prediction error of the controllers learned with the proposed method and two other baselines (the “Direct” method and directly using sensor data). From the bar plots, it is clear that, for some of the control variables, the RIIV method presents better results. Nonetheless, there are some points of concern: first, the presented final training cost (Fig. 2, left) is roughly the same for both cases, which makes unclear why RIIV presents better prediction errors in most variables. Furthermore, none of the results present confidence intervals, so it is unclear if they are statistically significant here.\n\n\n- Online learning effectively adapted the initial offline policy and provided a personalized control for human-robot normative walk.\n\t\nEvidence: The presented evidence shows the effect of online RL on top of the offline learned policy, presenting gains in terms of reduced EMG effort and peak knee error across the five participants. While this supports the hypothesis of offline-to-online adaptation, it is still unclear if this controller really brings benefits to humans. To clarify that, I would expect a baseline with no control signal (like a “placebo” controller) and evaluate the performance measures for the patients. The goal is to understand what is the impact of the whole exosuit in the patients. Another important point is: humans also adapt themselves to improve those performance measures, so it is unclear if the raised gains are due to the adaptation of the RL controller OR if they are due to the humans adapting their locomotion. To explain away the latter hypothesis, I would expect a baseline that runs the offline policy for as long as the online phase takes but without any RL adaptation, and after that evaluates the performance measures. \n\n- Human and robot co-adapted to achieve normative walking with reduced EMG effort.\n\nEvidence: The paper provides additional plots presenting the timing and duration of the state/control variables and describes how this evidence supports this co-adaptation hypothesis. Honestly, the text was really unclear and hard to follow, so I believe the paper failed to communicate the findings for this claim. Furthermore, the plots in Figure 4 are hard to read. Lastly, the suggest baseline from the previous point is also important here: it is unclear if the reduced EMG effort is due to the adaptation of the human or the RL controller, or both.\n\n- Both cost terms in the objective are required to achieve stable human-robot normative walk.\n\t\nEvidence: The paper conduct experiments ablating each term of the objective function to support this claim. While I believe this is the right evidence, the paper struggles to correctly communicate the findings again:it brings to separate figures, one of each representing the ablation of one component. But the figures are separated, and the plot scales are different, which makes really hard to analyze them. Therefore, it is hard to evaluate such a claim. I would suggest to unify Figures 5 and 6, and for each plot, bring 3 different curves: the one with the full objective, and other two ablating each component of the objective.", "methods_and_evaluation_criteria": "Yes. Since this is an application-driven paper, the evaluation happens directly in the final target application, and the evaluation criteria follows directly from the performance criteria expected from a walk assistance gadget.", "theoretical_claims": "The work is mostly driven by the application, so the work is more on the empirical side. The work does bring some theoretical analysis, but to the best of my knowledge this is largely inherited from previous work that analyses the RL algorithm properties and not exactly a direct contribution of this work.", "experimental_designs_or_analyses": "The general experimental design makes sense, although I believe the two baselines mentioned before are key to validate the impact of the proposed system. Besides that, It is not clear how challenging the presented setup is. The experiments are conducted in five healthy patients, in a stable walk of 1 m/s. The problem presents a state-space of 5 dimensions and action-space of 4 dimensions. Thus: \n\n1) It is unclear if the locomotion gaits from these five patients would indeed require a meaningful adaptation (in other words, how different should we expect their locomotion patterns to be?);\n \n2) The state/action spaces are comparable to small benchmarks in RL; while I understand the major challenge comes from the lack of dynamics model and limited data, the presence of expert demonstrations also considerably mitigates that and provides a good base policy for downstream RL.", "relation_to_broader_scientific_literature": "I understand the main contribution of the paper as providing a practical RL system for exosuit control under a limited setup (stable 1 m/s walk for healthy humans), which seems to be the first of its kind. The work does bring a very informative Related Work on the area, which helps understand why this is different from exoskeleton control. However, I am not expert on this area and I could not really evaluate if there are other potential methods for exosuit control that should be contextualized to this work.", "essential_references_not_discussed": "Please see my previous comment.", "other_strengths_and_weaknesses": "Strengths: I believe the overall problem setting (exosuit control) is really interesting and the proposed methodology fits the problem very well given the lack of a simulator/dynamics model.\n\nConcerns: Besides the concerns previously highlighted, I struggled to find what is the practical impact of the proposed controller. In line with what was described in the Impact Statement, the method (as it is developed and evaluated) is limited to a very restricted locomotion pattern and focuses particularly on healthy patients. While I understand the work provides a proof of concept, I believe it is still very far from a real-world controller, as it would require more extensive evaluation in different locomotion patterns, comprising different speeds, terrains, etc. As an application paper, it would be interesting to describe which applications are the developed controller purposed for, given the content presented in this paper.", "other_comments_or_suggestions": "Typos: The work consistently mistakes open quotes.", "questions_for_authors": "- How are the safety constraints and tolerance parameters selected? Tables 3, 4, and 5.\n\n- In Appendix G, are the described adjustments to address the OOD issue learned by the controller or is there a human, manual intervention?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a resubmission improved based on ICLR reviews. The AC agrees with the majority of reviewers that the quality is ok after substantial improvements. Some concerns still exist, e.g. whether the topic is a broad fit for ML community or assistive robotics community.
Decision: Accept (poster)