Reinforcement Learning Enables Real-Time Planning and Control of Agile Maneuvers for Soft Robot Arms
Authors: Rianna Jitosho
Tyler Ga Wei Lum
Allison Okamura
Karen Liu
Venue: CoRL 2023 Poster (2023)
Keywords: Soft Robotics
Reinforcement Learning
Sim-to-Real Transfer
Dynamics and Control

Abstract: Control policies for soft robot arms typically assume quasi-static motion or require a hand-designed motion plan. To achieve real-time planning and control for tasks requiring highly dynamic maneuvers, we apply deep reinforcement learning to train a policy entirely in simulation, and we identify strategies and insights that bridge the gap between simulation and reality. In particular, we strengthen the policy’s tolerance for inaccuracies with domain randomization and implement crucial simulator modifications that improve actuation and sensor modeling, enabling zero-shot sim-to-real transfer without requiring high-fidelity soft robot dynamics. We demonstrate the effectiveness of this approach with experiments on physical hardware and show that our soft robot can reach target positions that require dynamic swinging motions. This is the first work to achieve such agile maneuvers on a physical soft robot, advancing the field of soft robot arm planning and control. Our code and videos are publicly available at https://sites.google.com/view/rl-soft-robot.

Reinforcement Learning Enables Real-Time Planning and Control of Agile Maneuvers for Soft Robot Arms
Rianna Jitosho",
Tyler Ga Wei Lum", Allison M. Okamura,
C. Karen Liu
Stanford University (rjitosho, tylerlum, aokamura, karenliuj@stanffrddee
Abstract: Control policies for soft robot arms typically assume quasi-static mo- tion or require hand-designed motion plan. To achieve real-time planning and control for tasks requiring highly dynamic maneuvers, we apply deep reinforce- ment learning train policy entirely in simulation, and we identify strategies and insights that bridge the gap between simulation and reality. In particular, we strengthen the policy's tolerance for inaccuracies with domain randomiza- tion and implement crucial simulator modifications that improve actuation and sensor modeling, enabling zero-shot sim-to-real transfer without requiring high- fidelity soft robot dynamics. We demonstrate the effectiveness of this approach with experiments on physical hardware and show that our soft robot can reach target positions that require dynamic swinging motions. This is the first work achieve such agile maneuvers robot arm planning and control. Our code and videos are publicly available at https://sites. http://ossssss.ggeleeeeee hoom/vie//llsoftteeee
physical soft robot, advancing the field of soft
Keywords: Soft Robotics, Reinforcement Learning, Sim-to-Real Transfer, Dynamics and Controls
Introduction
Compared to traditional rigid robot manipu- lators, soft robot arms with inherent mate- rial compliance offer improved adaptability and safety in tasks requiring contact with the envi- ronment 2, 3]. However, soft robots have yet to achieve the high-speed capabilities of rigid robots, which can leverage dynamics to achieve difficult tasks [4, 5, 6, 7, 8). We aim enable dynamic control for class of soft robot arms made from inflated fabric tube with fab- ric Pneumatic Artificial Muscle (IPAM), which has near-linear dynamics and high-bandwidth dynamic response with minimal hysteresis [9].
Preumatic
Muscle
Motion Cacture Markers
Tright Surface
Figure I: We achieve soft mbot control tasks that require leverag- ing robot dynamics. Our inflated-beam robot uses a pneumatic bend- ing actuator swinging onto target surface. We use a motion capture system mea- the rohot's state compute control actions using reinforcement learning.
mobile to perform agile maneuvers
policy trained
We mount this inexpensive and lightweight inflated-beam soft robot arm on a mobile platform that allows ID linear actuation of the arm's 's base (Fig. 1). Integrating the soft arm with a sliding base increases the robot's workspace and allows additional dynamics that can be leveraged by a controller.
Most prior work in soft robot arm control assumes quasi-static motion [10, 11, 12, 13). This as- sumption is enticing because soft robots have complex dynamics resulting from their compliance and high dimensionality, so using quasi-static models for soft robot motion makes them easier to
Equal Contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Step 1: Perform System Identification
Step 3: Deploy Learned Policy on Soft Robot Hardware
Slate
M
Loop Commands
Tooal Rit B0- Mocal Faramitiors
Experiment Data
Soft Rahot Hardwarn
Mation Capture
Step 2: Train Policy in Simulation using Reinforcement Learning
Action
Obsarvation
Trajectories Policy Updates
Simulation Loop
Policy
Jant Toqpi Tseid
Osservatio
Simuiztiin
Policy Architscture
Old allc
Observation
MLP
pllit
Onutit
Action
3o Model Parameters
LITM
Layer Norm
Ocar
Pollcy
Action 3
Figure 2:
In our proposed framework, we first fit parameters for physics-based dynamic model using about 30 scconds of real experimental we control policy using deep reinforcement learning entirely simulation using model Finally, we deploy our learned control policy directly on real hardware using motion capture system measure the robot's 's state.
previous step.
control. However, this approach is unable to exploit dynamic capabilities, such as leveraging swing- ing motions to generate momentum, More recent works have incorporated dynamic models [14, 15), but limit their applications to tasks that are simple enough to use hand-designed motion plans, such as tip trajectory following. In contrast, our learned policies eliminate the need for a pre-defined motion plan because they are goal-conditioned and handle planning and control jointly.
In our work, we use dynamic model that approximates our soft robot arm as multi-link rigid body system. This allows us to leverage robot dynamics, as well as existing rigid-body simulators and control frameworks, to achieve tasks requiring high-speed motion. With this model, we employ reinforcement learning (RL) to train control policy and deploy this policy directly onto real soft robot system. However, naive transfer to the real world is unsuccessful, which
common challenge encountered when deploying control policies learned in simulation [16, 17J. We overcome this challenge by implementing key adjustments to the simulator's actuator and sensor models as well improving the policy's robustness modeling inaccuracies using domain randomization. Fig. 2 illustrates the three-step process of our proposed framework. To our knowledge, this is the first work that achieves real-time motion planning and control for high-speed behaviors on physical soft robot arm. This work demonstrates that a simple physics-based model can used to build simulator that enables learning of agile maneuvers for soft robots with zero-shot sim-to-real transfer.
To evaluate system's performance, we optimize trajectories for each goal in test set as an approximate upper bound of performance. We find that the policy from our RL framework performs similarly to the approximate upper bound while achieving real-time planning and control. In con- trast, trajectory optimization requires an offline planning stage that takes several minutes to perform, which is impractical for real-world use.
2 Related Work
Here discuss the most relevant prior work (see Supplementary Material (SM) A for more detail).
Soft Robot Arm Modeling. Our work requires motion that is impossible under quasi-static assump- tions, preventing the use of kinematic models such as the piecewise constant curvature model [13]. However, we find that we do not require the higher model fidelity offered by the augmented rigid body formulation [18], Cosserat rod theory [19], Koopman [20] models, or Long-Short Term Mem- ory models [14]. Instead, choose model similar the n-link pendulum model used by Jitosho et al. [21] since it is cheap to compute, simple to implement, and enables fast simulation.

---

Soft Robot Arm Control. Recent works have demonstrated dynamic behaviors on soft robot arms. Centurelli et al. [14] train a control policy using reinforcement learning to perform tip path follow- ing under light payloads. Fischer et al. [15] use an operational space controller and proportional- derivative control to perform tip path following for pick and place, throwing, and drawing. Bruder al. [20]. Haggerty et al. [22] use Koopman-ber peassdd--baedd system identification to construct explicit lin- dynamical models of soft robots, and they use modee-pred control and linear quadratic regulator, respectively, to perform trajectory following tasks. Dynamic models improve the speed and accuracy of tip path following, but these works require hand-designed tip trajectories, limiting the capabilities of their robots. In our work, we achieve autonomous motion planning. This enables agile maneuvers to emerge during training, such as high-speed swinging motions, that would be difficult to design by hand.
Reinforcement Learning for Controlling Physical Robots. A common training paradigm when learning policies for physical robots is to train control policies entirely in simulation and then deploy them in the real world [23]. However, policies trained in simulation often fail in the real world due to model discrepancies between the simulated and real systems [16, 17]. One approach to over- coming this sim-to-real gap is training the policy under domain randomization so that the resulting policy will be robust to simulation inaccuracies [24]. This approach has been successfully applied to robotics domains such as legged locomotion [25, 26, 27, 28, 29], in-hand manipulation [30, 31, 32], and robot grasping [24, 33, 34]. In addition to this randomization, Tan al. [35] overcome the sim- to-real gap by improving their simulator with system identification and actuator modeling for their quadruped robot. Our approach to actuator modeling achieves sim-to-real transfer despite lacking access to the underlying dynamics of our actuators.
3 Method
3.1 Soft Robot Modeling
We consider class of inflated-beam soft robots that utilize iPAMs as their bending actua- tors [9]. Our hardware implementation features one beam as the main body and second in- flated beam with smaller radius (the (PAM) attached along the length of the main body, Pres- surizing the fPAM causes one side of the main body to shorten, forming bend (Fig, 3a,b). We model the soft robot arm as an n a) link pendulum with hinge joints between links and between the base link and the sliding cart (Fig. 3c). This allows us to model the soft robot arm with finite number of state variables and use rigid- body simulators. The cart is attached to fixed rails with a prismatic joint. We model each hinge joint as a rotational spring and damper to capture the stiffness and damping of the inflated beam. We as- sume linear mapping between the pres- sure inside the bending actuator and the torque applied at each hinge joint. This results in the following definition for the hinge joint torques:
b)
Man Body
L a
481
04
IPM
05
Figure a) Illustration the fabric Pncumatic Artificial Musele (IPAM) actu- stion. inflating the IPAM causes shorten by AL. This enforces an asym- metric shortening of the robot's main body. causing to bend. b) Image of our physical soft rohot mounted to the mobile cart of linear actuator, c) Illss- tration of the 5-link pendulum model. Rigid links (bluc) are connected via hinge joints (red) each other and
cant (black). This of the linear actuator (gray) with prismatic joint. We define the cart position Blom and relative joint angles 0, We also model rotational springs and dampers each binge join. (not visualized).
attached
Tmoddl == K0 - DD + bp,
(I)
where Tmodel E Rn are the joint torques, 0 E R2" are the joint angles, Í E IR" are the joint velocities,
p E IR is the fPAM pressure, K E IR= is a diagonal matrix for stiffness, D E R is a diag- onal matrix for damping, and b E IRn a linear mapping from bending actuator pressure to joint torques. We set n 5 and fit K, D, and b using data from experiments with open-loop commands. We use measurements from the physical hardware for mass and inertia values. We describe these measurements as well as our system identification method, results, and validation in SM-B.
3

---

3.2 Learning Dynamic Soft Robot Control Policies
In this work, we focus on two soft robot arm control tasks:
1. Free Space Target Reaching: Reaching tip position targets in free space. We are particularly interested in targets that are high enough that the soft arm requires swinging to achieve the task.
2. Shelf Target Reaching: Swinging onto the surface of a shelf (Fig. 1). This task is difficult because the robot needs to coordinate the swinging of the arm with the sliding of the base such that 1) the tip swings above the shelf surface while avoiding a collision, and 2) the tip moves laterally over the shelf edge before the tip swings back down. This task also demonstrates that we can readily apply our framework tasks involving contact.
We formulate these as RL problems, in which an agent sequentially chooses actions to interact with an environment with the goal of maximizing cumulative reward [36]. For each task, we learn single policy that can reach diverse range of target positions, enabling real-time planning. We describe our RL framework here, and provide problem formulation and policy training details in SM-C.
Action Space. The RL agent's action a E IR² includes the commands for our robot's two actuators. The first actuator is a pressure regulator that controls the IPAM pressure. We define the commanded fPAM pressure in kPag as ap E R. which is different from the fPAM pressure p in Eq. 1 because takes time for the regulator inflate or deflate the fPAM. The second actuator is the linear actuator controlling cart motion. The linear actuator has internal controller which operates in "Velocity Mode", which requires the user to send velocity commands over time. We define the commanded cart velocity in m/s as Gean-vel E R, which is different from the actual cart velocity Ueart E R because it takes time to accelerate the cart to new velocity. Section 3.3 provides a comprehensive descrip- tion of the actuator dynamics in our simulator, emphasizing the crucial additional details concerning the real actuator dynamics that needed to be accurately modeled in the simulation.
Observation Space. The RL agent's observation consists of two parts: Orobor and Otask Orobot R includes the cart position Meart E I, the cart velocity Veart E IR, the tip position Xtip E R2, the tip velocity Vup E IR?, the previous cart velocity command Gprev-cart-vel R, and the current fPAM pressure p E IR. We found that including aprev-cart-vel and improves system performance, and we hypothesize these observations encode critical information for understanding actuator dynamics and generating coordinated actions over time. Orask defined specifically for each task. For the free space target reaching task, we define Oask = Xarget, where Xtagett E R is the target position. For the shelf target reaching task, we define Ousk (Xurget; durger), where Xurget R.2 is the target position and drargot E R is the difference between the y-coordinates of the target and edge of the shelf (Fig 4).
Reward Function and Reset Conditions We describe the reward func- tion and reset conditions in SM D.
Architecture Our control policy consists of multi-layer perceptron (MLP) followed by Long Short-Term Memory (LSTM) layer. The MLP has 3 layers (256, 128, 64 hidden units) connected with Exponen- tial Linear Unit (ELU) layers. The output of this MLP is concatenated with the original input, which is then fed into an LSTM layer with 256 hidden units followed by a Layer Normalization layer. This recurrent ar- chitecture allows the policy leverage temporal information improve performance and handle the partial observability of the robot state.
duarget
Figure Shelf target reaching task in simulation. Xusei is the target position, which visualized a green sphere with radius diccen 0.04 m. charpa the difference between the conrdinatss Xarge and the edge of the shelf.
3.3 Sim-to-Real Policy Transfer
One of the main contributions of this work is overcoming the discrepancy between the robot's simulated training environment and the real robot hardware. We find that matching the actuation and sensing in simulation to the behaviors of their real-world counterparts is crucial for achieving zero-shot sim-to-real transfer, while high-fidelity dynamic model of the soft robot is not necessary. In addition, we concur with previous work that

---

modeling latency in actuators as well as domain randomization also play important roles in the success of sim-to-real transfer [35].
Pressure Regulator Dynamics. Physical pressure regulators have finite flow rates, resulting in reduced tracking performance for high-frequency signals. As an example, the frequency response of the pressure regulator developed by Gao et al. [37] shows decreasing gain with increasing frequency past threshold. This issue is also apparent in our hardware, where the pressure regulator struggles to track abrupt pressure command changes (ex. black and red signals in Fig. 5). As in [37], we find our pressure regulator to behave like a low-pass filter, so we model this behavior in simulation by applying an infinite impulse response (IIR) filter to the pressure command. We implement this using the update rule:
p k+1 f- oa + (1 (1-o)p";
where a E is an empirically-chosen parameter. This model closely approximates reality (Fig. 5) and serves as a general approach for modeling pressure regulators in soft robot RL frameworks.
Linear Actuator Dynamics. Many actuators to use (e.g. position control for motors), but introduce modeling challenges for sim-to-real transfer. For example, our linear actuator's "Velocity Mode" controller hides the actual cart physics from the user. This controller first requires the user to specify a fixed acceleration Üfieed E IR before running the control policy. Then, the user sends velocity commands over time, and the actuator accelcrates the cart at the fixed acceleration until the velocity reaches the latest commanded velocity. This results in piece-wiss wise linear velocity profile (e.g. red signal in Fig. 6) consisting of constant slopes (constant acceleration) of either 0 or =iixed (e.g.
with internal controllers that make them easier
d'ixed m/s²).
In simulation, we can directly manipulate the cart's kinematic state. However, the real cart cannot instantaneously reach any velocity due to physical limitations. Additionally, directly imposing a new cart state could result in significant simulation inaccuracy because the soft robot arm is attached to the cart (could lead to very unrealistic forces on the soft robot arm). To bridge simulation and reality, design a simulated cart controller that operates by applying forces in manner that closely approximates the behavior of the real controller. Importantly, this approach does not require a precise understanding of the physical linear actuator controller's implementation. Because there are forces on the cart from the rest of the robot, we cannot simply apply a constant force to the cart to achieve constant acceleration. We address this challenge by implementing a low-level cart controller in simulation that performs acceleration tracking when the velocity error is high and velocity tracking when the velocity error low. We implement this by applying a cart force (in Newtons) Fean E R defined as:
[ + Ki (Uxxed Veat),
ley| > 0.1 (track acc)
Fean =
ley < 0.1 (track vel)
Gcart-vel
(Note: a is for action, not acceleration)
where 6, E IR is the velocity tracking error, IR is the mass of the robot, Uixed E R is the fixed cart acceleration, Veart E IR is the actual cart acceleration, and KG R and ky E IR are the
Pressura (kPag)
Command Actual
Simulated
Cart Velocity (mls
Command
Actual
20
Simulated
Simulated, No Latency
IPAM
10
0.2
0.4
0.G
0.8
1.5
Time
Time
Figure 5: Pressure in the IPAM in response pressure command. The IPAM pressure regulator is bandwidil-limited so we simulate this beluvise by applying inlinite impulse respomse (IIR) filier the RI. command. The RMS error between the actual and simulated pressure kPag. sure communds (.07 to 20 kPag).
Figure 6: Cart velocity in response velocity command. The lincar actuator cannot respond immediately to changes in commanded ve- locity,
simulate the linear large velocity errors and a velocity tracker for small emorr. The RMS between small compared to the range velacity commands (-1.0 to 1.0 m/s). The simulated velocity trajectory more closely approximates the velocity trajectory when actuator latency simulated.
acceleration tracker for
small compared to the runge pres-
simulated velocity 0.08
5

---

acceleration tracking gain and velocity tracking gain, respectively. In the acceleration tracking case, moxxed is a feedforward term based on F = ma, and ky (éixed rean) is a feedback term to improve tracking on this fixed acceleration. In the velocity tracking case, the velocity is already close to the commanded velocity, so only a small feedback term is required. Our simulated cart motion closely approximates the real cart's piece-wise linear velocity (Fig. 6). This strategy can be adapted to overcome sim-to-real difficulties for broad range of actuators.
Linear Actuator Latency. Physical actuators exhibit latency, which can hinder him-to-real transfer when left unmodeled in simulation [35]. We model the linear actuator latency in our simulator by adding a time delay between the time the policy outputs a velocity command and the time that the command is sent to the simulator's 's lower-level controller. Fig. 6 shows that modeling the delay helps our simulated cart's velocity profile closely match the real cart's velocity profile.
Velocity Measurements. During physical experiments, we compute tip velocity with first-order finite-differencnng, so we replicate this in our simulated observations.
Domain Randomization. We use domain randomization while training our policy in simulation so that it can handle the remaining sim-to-real gap not captured by our other adjustments. We apply additive Gaussian noise of (0, Jobs) and N (0, aaa)) for all elements of the observation and the action respectively, where dbbs E IR and act E IR are the observation and action noise parameters. We also apply uniform scaling of U(1 - Edyns 1 + Edya) to all elements of the dynamics parameters K, D,D b. where 6dyn E R is the dynamics noise parameter. We tune these parameters so that the policy takes longer to train, but the policy is still able to achieve nearly the same reward when fully trained (additional domain randomization details in SM F).
4 Results
We evaluate the capabilities of our RL framework by performing demonstrations and experiments on real soft robot arm hardware (hardware details in SM-E). Our learned policies achieve real-time planning and control of agile maneuvers not previously demonstrated on soft robot arms.
4.1 Characterizing Performance for Free Space Target Reaching
In this section, we assess the quality of behaviors generated by our RL framework by showing that its performance is close that of an approximate upper bound method. The upper bound method uses trajectory optimization (TO) for motion planning and real-time tracking controller for control (details in SM-H). We emphasize that trajectory optimization is not practical choice for our task since it requires time-consuming offline planning stage.
We compare our RL. method with the TO method on the free space target reaching task. When train- ing our RL method, we sample target positions Xtarges-train E [-0.4 m, 0.0 m] [0.55 m, 0.7 m). We test our RL method on array of 54 target positions Xargetttest [-0.4 m, 0.0 m] x [0.55 m, 0.8 m] (Fig. 7). Targets with z-coordinates above 0.7 m are outside the training distribution. To compare with the approximate upper bound, TO, we precompute 54 trajectories in an offline planning stage, each specifically optimized to reach its respective target position (~04.7 hours to precompute, as solving each trajectory took between 1-13 minutes, depending on the target position). All targets are on the left side of the soft robot because the fPAM is designed bend towards this side.
Fig. 7 shows that our learned policy reaches wide range of target positions, missing only the most difficult ones. Specifically, targets towards the upper left are difficult because they require the most aggressive swinging to reach, and targets with = O are difficult because the robot needs to swing aggressively to the sides in order to swing the tip towards the rail center (y = 0) while avoiding the rail limits. Despite this, our policy achieved a success rate of 94% for targets within the training distribution, 50% for targets outside the training distribution, and 80% overall. The TO method achieved success rates of 94%, 83%, and 91%, respectively. This suggests that our RL method matches the performance of the TO method within its training distribution, and also has some level of generalization to targets outside of this range, though at a much lower success rate.

---

Targets
RL
TO
Time (5)
reached w/ RL
Targets
reached
(apprax, upper bound performance)
0.8
2(im)
0.8
E
0.6
Targets
missed wi both
0.4
0.6
Left rail limit
-0.25
0
0.25
-0.25
0.25
(m)
(m)
Y (m)
Figure Comparison learned policy (RL) vs. trajectory optimization (TO) for reaching lurgel tip position in free space. The black circle is
Figure 7: Free space larget reaching on hardware for our learned
policy (RI.) trujectory optimization (TO). Targets above 0.7
centered the position with radius of 4 Both methods
are outside our learned policy's 's training distribution. We overlay the
perform high-speed swinging to reach the target position, showing that able produce motions comparable
starting position soft
Fig. compares tip trajectories achieved on hardware with our RL method and the TO method. These plots show similar trajectories for both methods, suggesting that our method approaches the approximate upper bound of performance. We see that the robot reaches the target tip position by building up momentum over multiple swings to increase its tip height. This demonstrates that the free space target reaching task is not possible without well-timed actuation that leverages swinging motion. Such motion plan would be difficult to design by hand, but our RL method enables real- time planning and control of these types of agile maneuvers for soft robot arms.
We further demonstrate that leveraging swinging is required for task success by comparing our policy PID controller, which does not reason about swinging (method and results in SM-I).
4.2 Ablation Study on Shelf Target Reaching
We assess the significance of individual components of our framework through ablation study with the shelf-reaching task. During training, we sample target positions Xurget-ttrai train [-0.48 m, -0.4 m] x [0.58 m, 0.67 m] and position the shelf such that duaget E [-0.05 m, 0.2 m] (Fig 4). To evaluate each ablation, we deploy the policy on physical hardware and run 3 tri- als for each the following shelf target positions (y, 2): (-0.48 m, 0.6 m), (-0.48 m, 0.63 m), (0.48 m, 0.66 m), and (-0.53 m, 0.63 m). We then compute the success rate across these trials and compare the results of these policies with our best-performing policy (Table 1).
Actuator and Sensor Models. We find that our actuator and sensor modeling improvements in our simulator (Sec. 3.3) were crucial for success. We verify this with the following ablations:
Table I: Shell Target Reaching Task Ablation Study Policy
No fPMM Smoothing: We remove the IIR filter that models the fPAM pressure regulator's dynamics, so
Success
pressure commands instantly update the pressure.
Ours
92%
Actuator and Sensor Model Ablations
No Cart Acceleration Tracking: We remove the
IPAM Smoothing
acceleration tracking mode in our low-level con-
No Cart Acceleration Tracking
8%
Cart Action Delay
0%
troller, leaving only the velocity tracking mode.
Ground Truth Velocilies
0%
No Cart Action Delay: We remove the delay in
Domain Randomization Ablations
Rundomization Observations
sending the RL command to the simulated low- level cart controller.
Domain Randomization on Dynamics
25%
Domain Randomization
17%
Architectore Ablations
Ground Truth Velocities: We use ground truth ve-
MLP Only
0%
locities in our observations instead of velocities computed using first-order finite differencing.
LSTM
JPAM Actuation Ablation
Carl Actions Only (No (PAM)
0%
While these policies obtain high success rates in simulation, these ablations result in poor perfor- mance on the physical robot due to the larger sim-to-real gap. The failures appear similar to those saw during initial development before we added these sim-to-real adjustments. We found the underlying causes of these failures by methodically evaluating the failures and comparing the sim and real observations, actions, and dynamics. More details about this process can be found in SM G.
7

---

Domain Randomization. We find that the best performance on hardware is achieved by training policy with domain randomization on policy actions. We consider ablations with different domain randomization schemes to verify this:
Domain Randomization Observations: We remove randomization the actions and add
Gaussian noise to the observations.
Domain Randomization on Dynamics: We remove randomization on the actions and scale
dynamic parameters (K, b. by value from a uniform distribution.
No Domain Randomization: We remove randomization on the actions.
Within this ablation subset, removing all domain randomization performed the worst, verifying the benefit of domain randomization in crossing the sim-to-real gap. The ablations with observation and dynamics randomization each show small improvement in success rate compared to the ablation without any randomization, suggesting there a small sim-to-real gap in our dynamic parameters and our observation measurements. However, there is a large difference success rate between our rest-performing policy and these ablations, which suggests that action randomization is important to address the remaining sim-to-real gap captured by our adjustments described in Sec. 3.3.
Architecture. Our best-working policy architecture is a multi-layer perceptron (MLP) followed by a Long- Short Term Memory (LSTM) network. We consider ablations with different architectures:
MLLP only: We remove the LSTT layer so that the policy does not have memory. LSTM before MLP: We modify the architecture that the LSTM is before the MLP.
Both ablations perform poorly on physical hardware. We expected the MLP without the LSTM to perform worse because memory has been shown to be useful in control tasks with partial observ- ability [38, 39]. The higher success rate for the architecture with the LSTM after the MI.P suggests that the MLP provides a useful representation for the LSTM to model temporal dependencies.
IPAM Actuation. We verify that the fPAM is necessary for task success by running ablation in which we turn off fPAM actuation in both the simulated and real systems (i.e. the robot is limited to cart actions only). The robot had low success rates in both simulation and on physical hardware, demonstrating that both actuators (fPAM and cart) are required for the shelf-reaching task.
Limitations
While our work demonstrates significant progress in enabling real-time planning and control for soft robot arms, there limitations that should be addressed in future work. One limitation is the use of motion capture, which is costly estimation would allow broader use of our system. Another limitation is that we do not fine-tune the policy in the real world, which could allow adaptation to changes in the environment or robot hardware. Finally, while our work demonstrates that using simple physics-based models for this class of soft robots an effective approach, is important to acknowledge that the implementation this work has fewer actuators than many soft robot systems. More complex soft robots may benefit from higher-fideliee or learning-basse based models. In future work, we are interested in exploring how to expand this framework to other soft robot platforms.
up and not suitable for outdoor field use. Onboard state
6 Conclusion
In this study, we demonstrate that deep reinforcement learning enables real-time planning and con- trol of dynamic maneuvers for soft robot arms. We identify key insights needed to overcome sim- to-real challenges for zero-shot sim-to rimalo-real transfer, including domain randomization and improve- ments to actuator and sensor modeling. Our work provides evidence that accurate actuator and sensor models crucial for task success, and that agile behaviors can be learned without requiring high-fidelity model of soft robot dynamics. We demonstrate the effectiveness of our framework through physical experiments that show soft robot arm executing agile maneuvers in real time.

---

Acknowledgments
If paper is accepted, the final camera-ready version will (and probably should) include acknowl- edgments. All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.
References
[1] W. Burgard, O. Brock, and C. Stachniss. Safety evaluation of physical human-robot interaction via crash-testing. In Robotics: Science and Systems III, pages 217-224, 2008.
[2] H. Jiang, Z. Wang, Y. Jin, X. Chen, P. Li, Y. Gan, S. Lin, and X. Chen. Design, control, and applications of soft robotic arm. arXiv preprini arXi::2007.04047, 2020.
[3] D. Rus and M. T. Tolley. Design, fabrication and control of soft robots. Nature, 521(7553): 467-475, 2015.
[4] F. S. M. Alkhafaji. Modeling and control high speed robotic arm for industrial applications. In Global Congress on Electrical Engineering, pages 121-130, 2021.
[5] S. Mori, K. Tanaka, Nishikawa, R. Niiyama, and Kuniyoshi. High-speed and lightweight humanoid robot arm for skillful badminton robot. IEEE Robotics and Automation Letters, (3)1127-1734, 2018.
[6] Nakanishi, R. Cory, M. Mistry, J. Peters, and S. Schaal. Operational space control: A theoret- ical and empirical comparison. The International Journal of Roboties Research, 27(6)77377757, 2008.
[7] A. Takahashi, M. Sato, and A. Namiki. Dynamic compensation in throwing motion with high- speed robot hand-arm. In JEEE International Conference on Robotics and Automation, pages 6287-6292, 2021.
[8] Y. Yamakawa, A. Namiki, and M. Ishikawa. Motion planning for dynamic knotting a flexible rope with high-speed robot arm. In IEEE/RSI International Conference on Intelligent Robots and Systems, pages 49-54, 2010.
[9] N. D. Naclerio and W. Hawkes. Simple, Low-Hysteresis,, Foldable, Fabric Pneumatic Artifi- cial Muscle, IEEE Robotics and Automation Letters, 5(2):3406-3413, 2020.
[10] J. M. Bern, Y. Schnider, P. Banzet, N. Kumar, and S. Coros. Soft robot control with a learned differentiable model. In IEEE International Conference on Soft Robotics, pages 417-423 2020.
[11] T. George Thuruthel, F. Renda, and F. lida. First-order dynamic modeling and control of soft robots. Frontiers in Robotics and AI, 7, 2020.
[12] M. Grube, J. C. Wieck, and R. Seifried. Comparison of modern control methods for soft robots. Sensors, 22(23):9464, 2022.
[13] I. Robert J. Webster and B. A. Jones. Design and kinematic modeling of constant curvature continuum robots: A review. The International Journal of Robotics Research, 29(1))1661- 1683, 2010.
[14] A. Centurelli, Arleo, Rizzo, S. Tolu, C. Laschi, and E. Falotico. Closed-loop dynamic control of soft manipulator using deep reinforcement learning. IEEE Robotics and Automation Letters, 7()):744144788, 2022.
[15] O. Fischer, Y. Toshimitsu, A. Kazemipour, and K. Katzschmann. Dynamic task space control enables soft manipulators to perform real-world tasks. Advanced Intelligent Systems, page 2200024, 2022.

---

[16] A. Boeing and T. Bräunl. Leveraging multiple simulators for crossing the reality gap. In International Conference on Control Automation Robotics & Vision, pages 1113-1119, 2012.
[17] S. Koos, J.-B. Mouret, and S. Doncieux. Crossing the reality gap in evolutionary robotics by promoting transferable controllers. In Conference on Genetic and Evolutionary Com- putation, GECCO '10, page 119-126. Association for Computing Machinery, 2010. ISBN 9781450300728.
[18] R. K. Katzschmann, C. Santina, Y. Toshimitsu, A. Bicchi, and D. Rus. Dynamic motion control of multi-segment soft robots using piecewise constant curvature matched with an aug- mented rigid body model. In IEEE International Conference on Soft Robotics, pages 454-461, 2019.
[19] N. Naughton, J. Sun, A. Tekinalp. T. Parthasarathy, G. Chowdhary, and M. Gazzola. Elastica: A compliant mechanics environment for soft robotic control. IEEE Robotics and Automation Letters, 6(2):3389 3339999,,,, 2021.
[20] D. Bruder, B. Gillespie, C. D. Remy, and R. Vasudevan. Modeling and control of soft robots using the koopman operator and model predictive control. Robotics: Science and Systems, 2019.
[21] R. Jitosho, N. Agharese, A. M. Okamura, and Manchester. dynamics simulator for soft growing robots. In IEEE International Conference on Robotics and Automation, pages 11775- 11781, 2021.
[22] D. A. Haggerty, M. Banks, Kamenar, A. B. Cao, P. C. Curtis, I. Mezić, and B. W. Hawkes. Control of soft robots with inertial dynamics. Science Robotics, 8(8 x(8)))ea,,,6 2023.
[23] P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 2226-2240, 14-18 Dec 2023.
[24] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Au- tomation, pages 3803-3810. IEEE, 2018
[25] Dao, H. Duan, K. R. Green, J. W. Hurst, and Fern. Learning to walk without dynamics randomization. 2nd Workshop on Closing the Reality Gap in Sim2Real Transfer for Robotics (Robotics: Science and Systems), 2020.
[26] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. In Proceedings of Robotics: Science and Systems, 2021.
[27] Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust percep- tive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62):eabk2822, 2022.
[28] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, volume 164 of Pro- ceedings of Machine Learning Research, pages 91-100, 2022.
[29] Siekmann, K. Green, Warila, A. Fern, and Hurst. Blind bipedal stair traversal via sim- te-real reinforcement learning. In Proceedings of Robotics: Science and Systems, 2021.
[30] O. M. Andrychowicz, B. Baker, M. Chociej, Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng. and W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
10

---

[31] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving Rubik's s cube with a robot hand. arXiv 2019.
preprint
[32] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk,
K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J.-F. Lafleche, D. Fox, and G. State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv preprint 0rXiv:2210.1370,, 2022.
[33] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari. RI-cyclegan: Reinforcement learning aware simulation-to-real. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[34] E. Tzeng, C. Devin, J. Hoffman, C. Finn, P. Abbeel, S. Levine, K. Saenko, and T. Darrell. Adapting Deep Visuomotor Representations with Weak Pairwise Constraints, pages 688-703. Springer International Publishing, 2020. ISBN 978-3-030-43089-4
[35] Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. In Proceedings of Robotics: Science and Systems, 2018.
[36] R. S. Sutton and A. G. Barto. Reinforcement Learning An Introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-5
[37] R. Z. Gao, M. Hébert, J. Huissoon, and C.. Ren. μPump: An open-source pressure pump for precision fluid handling in microfluidics. HardwareX, 7, Apr. 2020.
[38] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi,
R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre,
T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, Molloy, T. L. Paine, C. Gulcehre, Z. Wang. T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, Schaul, T. Lilficrap, K. Kavukcuoglu,
D. Hassabis, Apps, and Silver. Grandmaster level in StarCraft II using multi-agent rein- forcement learning. Nature, pages 1-5, 2019.
[39] Zhu, X. Li, P. Poupart, and G. Miao. On improving deep reinforcement learning for pomdps. arXiv preprint (XXxx:/004.000 07977, 2017

---

Supplementary Material
A Additional Related Work
Soft Robot Arm Modeling, While the pose of any point on a traditional, rigid-link robot can be fully defined by its link lengths and joint angles, the kinematics of soft robots more complex due to their elasticity and continuum nature [1]. While Finite Element Methods (FEM) provide high-accuracy models of soft robots, their computational cost and high dimensionality make them difficult to use in control methods [2]. Instead, researchers opt for more tractable approximations for practical implementation [1]. The most common modeling approximation is the Piecewise Con- stant Curvature (PCC) model, a kinematic model in which the robot is approximated as a series of constant-curvature arcs. Other works use models that offer a middle ground in computational com- plexity. Katzschmann et al. [3] introduce augmented rigid body formulation that models soft robot rigid-bodied robot with parallel elastic actuation, which enables dynamic model that respects PCC assumptions. Naughton et al. [2] draws from Cosserat rod theory to model soft arms as continuum elements that can bend, twist, shear, and stretch.
Another approach to dynamics modeling through data-driven modeling methods. Bruder et al. [4] model a soft robot using the Koopman Operator, in which projected linear state space model is fit with collected data. Jitosho al. [5] create soft robot simulator that models soft growing robots
as N-link pendulums with linear springs and dampers between links, and prismatic joints to model lengthening and retracting, and these parameters are fit with collected data. Bern et al. [6] train neural network to find mapping from motor angles to quasi-static tip position. Centurelli et al. [7] model soft robot forward dynamics with a Long Short- Term Memory (LSTM) network to find a mapping from an actuation vector to a tip position.
Soft Robot Sim-to-Real Addressing the sim-to-real gap is important challenge in the domain of soft robots. This challenge arises from the complex kinematics and dynamics inherent to these systems, rendering their accurate simulation notably more difficult compared to conventional rigid robots. Many works use FEM to accurately simulate the behavior of real soft robot hardware. Du- bied et al. [8] model soft robot arm with pneumatic actuation using differentiable FEM, which provides gradients to efficiently calibrate the Young's 's modulus, fit damping, and tune actuation pa- rameters to match the real system. Hagiwara al. [9] develop a soft robotic gripper with both soft deformation and high stiffness, and they model it with finite element analysis support simple grasping with visual servo control. Zhang et al. [10] use differentiable simulator to learn the mate- rial parameters of soft robotic fish from quasi-static data and use these parameters to create robotic fish tail designs that can be used in the real world. These soft robot modeling methods can be very accurate and useful for sim-to-real transfer, but are too slow to be used for training a policy using reinforcement learning.
Sim-to-real transfer of control policies for soft robots is promising direction of research. Li et al. [11] use deep deterministic policy gradient (DDPG)-based control system for soft robot arm to perform path following, which is first trained in a kinematics-only simulator and then fine-tuned on real-world data. Although they successfully transfer policies to the real world, they only perform slow-moving path following, and their kinematics-only simulator has a large sim-to-real gap, so fine-tuning (deploy policy, collect data, retrain, repeat) to
they need multiple rounds of overcome the sim-to-real gap. Similarly, Zhang al. [12] use Q-Learning to train control policy for mlownnoving reaching task, and their policy needs to be trained both in simulation and in the real world. In contrast, our control policies demonstrate highly dynamic swinging behavior, are trained entirely in simulation, and can be deployed on a real soft robot zero-shot.
There are many other aspects of soft robot sim-to-real that are being studied. Yoo al. [13] propose a neural network to predict a soft robot's current shape configuration from point cloud data, and they demonstrate strong performance in the real world when trained entirely simulated data. An exciting direction for future work is to replace the motion capture system used in our work

---

with this neural network, as it would allow our control policies to be deployed in environments that do not support motion capture hardware. Kriegman et al. [14] introduce a soft robot design and construction kit that can be used to simulate, create, and measure the simulation-reality gap of simple soft robots. While this approach can identify soft robot designs with smaller sim-to-real gaps, our work is focused on overcoming the sim-to-real gap when deploying learning- based control policies real soft robot hardware.
Soft Robot Arm Control. Many previous works leverage traditional, physics- methods to control soft robot arms. Santina [15] achieves closed-loop stabilization with feedback linearization on a simulated soft inverted pendulum. Weerakoon and Chopra [16] use an energy-based controller for soft robot swing-up task and then use a cneer--uudratic regulator controller for stabilizing the soft robot in the upright position. Bruder et al. [4] model predictive control with a Koopman model to perform tip trajectory tracking for a soft robot. Grube et al. [17] perform soft robot arm trajectory following with a kinematic controller and dynamic controller, and they find that the dynamic controller achieves higher accuracy and robustness than the kinematic controller but also requires more computational resources. Haggerty et al. [22] model and control soft robots using a data-driven learning method based on Koopman operator theory, allowing them to generate fast motion with high-deflection shapes.
Another method for soft robot arm control is training a neural network with large amounts of data. Bern et al. [6] train a neural network to approximate the forward dynamics of a soft robot arm, and then perform quasi-static trajectory following by using gradient-basee optimization with this learned model. Thuruthel et al. [19] train a recurrent neural network to model the forward dynamics of a soft robot and then use the learned model and trajectory optimization to create open-loop trajectories. Next, they test these open-loop trajectories on the real robot and use this data to train neural network in supervised fashion to be used closed-loop predictive controller. Qiuxuan al. [20] fit soft robot dynamics model with multi-layer perceptron and then train control policy with deep Q-learning.
We would like to note the distinction between soft robot arms and rigid robot arms with soft ac- tuation. The actuation choice (soft vs. traditional) has minimal impact on the robot kinematics, whereas the limb choice (soft vs. rigid) greatly affects robot kinematics due to the compliance of the soft body. This results in additional modeling and control challenges for soft robot arms.
Reinforcement Learning for Controlling Physical Robots. An alternative paradigm for training robot control policies using reinforcement learning involves learning from both simulation and real- world data. Bousmalis et al. [21] first train a control policy in simulation and then fine-tune the policy on real robot. Rusu al. [22] use real-world data to train a generator network that transforms simulated images into real images so that the policy can learn from more realistic observations. While this paradigm can reduce the sim-to-real gap, do not use this approach for problem because collecting large amounts of real-world data would be time-intensive and result in significant degradation to the hardware.
There is a growing interest in applying reinforcement learning to soft robot arm control, with ex- isting works primarily focusing on trajectory following at relatively slow speeds [2. 7, 19, 23, 24]. Naughton et al. [2] focus on additional tasks that require maneuvering between structured obstacles. Similarly, we achieve tasks that require reasoning about objects in the robot's 's environment. Bianchi et al. [25] develop open-loop controller for throwing objects to target positions by learning model between the sequence of actuations and the resulting landing position. Our work differs from previous implementations of RL for soft robot arms because we focus on achieving tasks that require high-speed motion and do not need a predefined motion plan.
B System Identification Method, Results, and Verification
We measurements from the physical hardware to compute mass and inertia values for our robot model. Our physical soft robot arm contains internal hardware inside the tip, but otherwise is

---

hollow, inflated beam. Based on this, we approximate the most distal link solid cylinder since it contains internal hardware, and we approximate the proximal links as cylindrical shells. We measure the mass of the full robot arm and the internal hardware, and use this to compute the weight of distal cylinder and each of the four cylindrical shells (0.1 kg and 0.05 kg respectively). With the measured masses and the measured cylinder radius (0.038 m), we can compute the inertia for each geometry accordingly.
We fit K. D, and b using data from fixed- base experiment (no use of the cart's linear actuator). In this experiment, we send sequence of fPAM pressure commands and measure the resulting sequence of pressure values (p²:²) as well as the resulting robot trajectory (0I:N). From these
values, we can compute velocities (él:N) and accelerations (0 (?!:N via 4th-order finite differencing.
Using these values, we compute the joint torques - at timestep via inverse dynamics [26]:
M(0*)0t + C(O*; 0 + g(0"),
(2)
where M, C, and g, are functions that compute the mass matrix, Coriolis terms, and torques due to gravity, respectively.
To model parameters K, D, and b, we pre-compute (T (:)) using Eq. 2, then we fit model parameters using least squares on our definition of joint torques in Eq. 1:
N
min K.D.b
Ke D + bp - =12
(3)
20
pressure (kPag)
15
IPAM
5
b)
Data Model
Torque at Base
Pin Joint (Nm)
0.2
-0.2
10
14
16
1.8
20
22
244
26
28
Figure 9: Results the system identitication litting procedure. a) Input control signal into the system Comparison of joint torques computed from experiment data (red) versus our model with fitted parameters (blue). The RMS error 0.02 Nm,, which is small compared to the torque values plot.
Fig. 9 shows the results of the system identification procedure. Fig. 9a shows the sequence of IPAM commands sent the physical robot. includes slow sweep of pressures capture quasi-static behavior, square
to capture the step-response, and series of high- high-frequency sine waves to capture swinging dynamics. The fPAM commands were sent at 60 Hz for roughly 28 s, resulting in about 1600 timesteps of measurements. Fig. 9b shows the results of the least squares fitting. We overlay the joint torques derived from measured data (Eq. 2) versus the one computed using esti- mated parameters (Eq. 1). The RMS error is 0.02 Nm, which is small compared to the actual torque values (-0.28 to 0.25 Nm), showing good agreement between our model and the measurements.
The numerical values for stiffness, damping, and our control mapping are below:
diag (K) (0.8385, 1.5400, 1.5109, 1.2887, 0.4347) Nm/rad
diag C) (0.0178, 0.0304, 0.0528, 0.0367, 0.0223)
b (0.0247, (0002,7000666,, 0.0779, 0.0498, 0.0268) Nm/psi

---

To demonstrate that our dynamic model captures our soft robot arm'ss behaviors, we compare the tip trajectories of the real robot with simulation that uses our fit model. The fPAM command sequence sent to each system is sinusoid with a frequency not used in the system identification procedure. Fig. shows the comparison tip position over time. The tip y-coordinate ranges from -10 to 7 em with an RMS eπror of 2 em. The tip z-coordinate ranges from 51 to 52 em with an RMS error of 0.3 em. The average distance between the simulated and real robot tip positions across all time steps is 2 cm. We find that this model fidelity is sufficient for the tasks demonstrated in our work.
a)
IPAM pressure (kPag)
20
2.8
5)
y y-cordinaae (m)
0.1
Data
Model
Tlp
0
-0.1
Tip e z-coordinate (m)
Data
Model
0.53
0.52
0.51
0.6
1.6
2.2
2.8
Time (8)
Figure 10: Evaluation of fit model. a) Input control signal into the system that is different than the original system identification experiment. b) Comparison tip y-coordinate between experiment data versus a simulation with the fit model. The RMS error cm. e) Comparison cooddinatte between experiment data versus sinulation with the lit model. The RMS error 0.3
C RL Framework Details
Problem Definition for Reinforcement Learning. We formulate the soft robot arm control task as a reinforcement learning problem. This is commonly modeled as Markov Decision Process (MDP) given by (S, A, P, R), where S is the state space, A the action space, P S A x S - R is A
+ the reward function. P(s 18,, ae) gives the probability of the agent transitioning from state s; $1+1 when takes the action ae. R(st, ag $t+1) gives the reward r, the agent receives when it transitions from state 8 81+1 when it takes the action a, The agent's s goal is to maximize the return Rg -m= > which is the total discounted reward from timestep t onwards, where E [0, 1] is the discount factor that defines how much the agent favors near-term rewards over far-term rewards.
the transition function, and R
This formulation assumes a fully observable MDP, but in many real-world robotics control prob- lems, the full state of the robot cannot be captured. These problems be modeled Partially- Observable Markov Decision Process (POMDP), where the agent receives observations from ob- servation model (o,, St, d.). In this setting, the agent cannot observe the full state at each timestep. Common solutions to this problem include stacking history of observations [27] or compressing the history into a hidden state through the use of recurrent neural networks [28, 29]. These solutions have the added benefit access to additional temporal information.
D Reward Function and Reset Conditions
We define reward at each timestep, E R, to be weighted of rewards from individual reward functions Σ, wors, where r; R is the reward from the i-th reward function and

---

Wi E IR is weight associated with the i-th reward function. Specifically, we use:
10000,
X Xtarger
dauccess
Wpps 1.0,
Poss
[0,
otherwise
Wyel
0.1,
Ivel
|/vupll
-
100,
Blean > y
Wiimtt 1.0,
Pimii
0,
otherwise
where diuccess 0.04 m is the maximum tip-to-target distance to count success and 0.25 is the rail limit. The small velocity reward improves training speed because it encourages swinging, which is required for higher target positions. An episode terminates when the cart exceeds the rail limits, the tip position reaches the target position, or the episode exceeds 100 timesteps (3 seconds).
E Experimental Setup
To demonstrate our RL framework on physical hardware, we design and build an experimental setup featuring soft, inflated-beam robot with a mobile base (Fig. 1). The central component is a computer running the control policy. It sends pressure commands to the soft robot, sends velocity commands to the linear actuator, and receives state measurements from a motion capture system. Despite not having a GPU, policy inference takes under 10 ms running on this computer, which enables real-time planning and control.
-nla Robot. The total length and weight of the soft robot arm is 44 cm and 0.12 kg, respectively. The main body has a radius of 3.8 em. As described by Naclerio and Hawkes [30], the beams forming the main body and the IPAM are constructed by forming tubes with bias-cut, woven fabric. The bias-cut orients the fabric fibers such that the tubes become shorter and wider when pressurized. This results in the shortening during pressurization. We use two pressure regu- lators (Festo VPPI) control the pressure in the main body and fPAM, and each regulator also has built-in pressure sensor. The main body is held at constant pressure (0.4 kPag), and the fPAM is commanded to varying pressures (-0.7 to 20 kPag). The lower body pressure reduces opposition to fPAM actuation while having enough pressure to maintain its cylindrical shape. The fPAM com- mand bounds were determined empirically with the following principles: (I) the minimum pressure must show the fPAM visibly deflated (2) the maximum pressure must cause maximum contraction of the fPAM (3) the range of pressure commands should be small to increase tracking performance of the pressure regulator. Our central computer sends pressure commands and receives pressure measurements via serial communication with microcontroller (Teensy 3.6).
Mobile Base We utilize cart on a belt-driven linear actuator (Igus ZLW-1040B) that acts as a mobile base for our soft robot arm. The base of the soft robot arm is directly attached to the cart, which slides along the actuator rails within a 0.6 m range. We use the linear actuator's "Velocity Mode", which requires setting acceleration value followed by sending velocity targets over time. Our central computer communicates with the linear actuator over Transmission Control Protocol (TCP).
Motion Capture Sensing, To simplify the hardware and sensing scheme for our robot, we use a mo- tion capture system (OptiTrack with Flex 13 cameras) to measure the robot's current configuration. We place five sets of markers equally spaced along the soft robot arm, and a sixth set on the sliding cart. For each set of markers, the motion capture system provides the position and orientation in the global frame. We also use motion capture to measure task-specific observations Orask-
Simulation and Policy Learning Details. We simulate the soft robot arm using Isaac Gym [31], a perforaefor simulator that leverages GPU parallelization to simulate thousands of robots simultaneously. Using one NVIDIA RTX 3090 GPU, our simulation runs at 18,000 FPS (each frame is one action step with control timestep of 33 ms) by running 4,096 environments in parallel. The simulation timestep and control frequency are two important parameters determine. Our real, physical system runs at 30 Hz control frequency (most of this time is spent communicating with the
5

---

sensors to measure the current state), so we run the control policy at 30 Hz in simulation accordingly. However, simulation often requires smaller timesteps to ensure numerical stability. We found that simulating the vine robot at 1200 Hz (0.833 ms timestep) is sufficiently stable.
For modeling the fPAM pressure, we found that the filtering parameter a was different for inflation (ap p) and deflation (ap < p). Thus, we use Gindate 0.86 and O'dettate 0.81 for modeling inflation and deflation, respectively. For modeling the cart dynamics, we use K2 = 0.3, ky = 30, and an action delay of control timestep (33 ms). For domain randomization, we use Jebs 0.001, Jxt = 0.001, and Edyn = 0.001.
We train all learned policies with a learning rate of 3e-4, discount factor 7 of 0.99, and PPO clipping interval Catip of 0.2. We also normalize the observations, values, and advantages, and train the policy with 4 epochs per policy update. Using horizon length of 16 (number of timesteps between updates for each robot, with all robots running in parallel), 4096 simulated robots, and a maximum of 500 update iterations, the approximate number of training timesteps is 32M steps (16 x 4096 x 500). Training takes about 40-80 minutes on an NVIDIA RTX 3090 GPU, which is substantially less time than Elastica's 1 hour RL training [2].
The policy is fed normalized observations (subtract the mean and divide by the standard deviation of each dimension). It outputs actions in [-1.0, 1.0), which are then scaled to be in the appropriate range for each action dimension. We train the policy with the Proximal Policy Optimization (PPO) algorithm [32], using highly-optimized GPU implementation called rl. gl-games [33], which uses vectorized observations and actions for faster training.
F Domain Randomization Details
In our experiments, we use Jabs 0.01, net = 0.005, and edyn = 0.05. Note that the observation noise and action noise are applied to the normalized observation and the unscaled action (actions in the range [-1.0, 1.0).. This ensures that the scale of the noise relative to the original value is consistent, so all components are affected similarly.
G Evaluating Failures
When evaluating our learned policy without fPAM smoothing, we noticed that the real fPAM actu- ations were not well-coordinated with the cart's motion. From there, we compared the commanded and actual fPAM pressures and found that the actual pressure did not immediately track the com- manded pressure. When evaluating our learned policy without the linear actuator adjustments, we noticed that the real cart would not be able to change direction quickly enough to avoid the rail limits. We compared the sim and real cart velocity tracking profiles, which helped us model the cart controller more accurately. Finally, when evaluating our learned policy with ground truth ve- locities, comparing real and sim observations revealed that the velocities computed with first-order fiiie-diiffereciin were noticeably different from the simulator's ground truth velocities.
H Trajectory Optimization Planning and Control Method
We use trajectory optimization to determine a reference trajectory (control trajectory u 1:N-1 and resulting state trajectory xI:N) that brings the tip of the soft robot arm to the goal position within a fixed time horizon N. We define the state vector x (Yeat Meant 01, 0i.... .,. 0s, 0s) and control input u = (Femn ap) where Fean is the force applied to the cart and ap is the fPAM pressure. We solve the following optimization problem to compute the reference trajectories.

---

N
N-1
|x* x11 + Σ
|u" 11
min
I:N
1-N-1
k=
k=1
s.t.
x++ = f(x"; " ,
k = 1, 1.... 1,
Umin S u < Umaxx
k
N - 1,
(4)
Veart-min S Blart S Veart-max:
k 1,
N - 1,
Beart-min
S Veam S Veart-max:
k I....,,, N-1,
g(x" N =
where Venn and Vian are the first and second elements of x* respectively. The objective function is a quadratic cost on deviation from a nominal state trajectory x with weight matrices Q and quadratic cost on control effort with weight matrices R1:N-1 The first constraint is for dynamic feasibility, the second is for control limits, the third is for cart position and velocity limits, and the fourth is for the tip position to reach the goal position at the final timestep. We set each x of the nominal trajectory to be a pose that bends to the left (with zero velocity) since all goal positions require bending to the left. Because we use a multi-link rigid body approximation as our dynamic model, we have a broader array of options for our simulator and optimizer choice. For this work, we chose Dojo [34] for numerical stability (since we have stiff equations of motion and would like to take larger time steps), and we used its associated trajectory optimization package which implements iterative Linear Quadratic Regulator with Augmented Lagrangian methods.
The numerical values for Qh and R for k I....,N N - are:
diag(Q") = (10, 1, 10, 1, 10, 1, 10, 1,
diag(R)) (1,
The numerical values for QN are:
diag(Q") = (100, 10, 100, 10, 100, 10, 100, 10, 100, 10, 100, 10)
Dojo is a powerful simulator in that it is able simulate robots with stiff dynamics while taking relatively large time steps, while remaining numerically stable (in contrast to IsaacGym, which needed to be simulated 1200 Hz to avoid numerical instability). This is critical for reducing the number of timesteps in the optimal trajectory (and in turn the number of variables in the optimization problem). However, it has yet to be optimized for runtime performance. Solving the trajectory optimization problem took between 1-13 minutes, depending on the target position. We expect the solve time to decrease as the simulator is further optimized for speed. Re-planning in real-time, (c.g. with model predictive control), was not possible due to the optimization solve time, so instead we used simple but fast tracking controller that adds feedback term to the reference action trajectory based deviation from the reference tip trajectory:
k cart-vel vel t vel + (u viip)
(5a)
k = ox Opre + kg (tt =(pp).
(5b)
Reference action trajectories vel ref as well as a rer are extracted from the optimal solution for x II:N-1 and I::N-1. The current tip position =k (Blip: =(sp)) is measured with motion capture, and ky and k2 are controller gains. The adjusted actions are clamped to be within the action limits for the cart and fPAM and then sent to the physical hardware. During trajectory optimization, we use conservative constraints on cart velocity and fPAM pressure so that the tracking controller has margin to exceed the reference control before needing to be clamped within actuator limits. We empirically chose gains that improved performance for one of the more difficult target positions, and then used this for all other experiment runs. The gains used were k, 0.1 and kg 5.0.
11:-1
1::-1 and reference tip trajectories w ver and
7

---

As with our policy trained with RL, we require careful but simple strategies to achieve sim-to-real transfer. First, the trajectory constraints allow us to respect state and control limits (i.e. cart position, velocity, and acceleration as well as fPAM pressure). We note that we indirectly enforce constraints on cart acceleration by constraining Ferrt. Second, the use of a tracking controller allows us to overcome minor model errors, similar how domain randomization during policy training produces robustness to model errors. Finally, did not model actuator dynamics when solving the reference trajectory for simplicity, but this caused delay between the reference and actual tip trajectory that could not be overcome by the tracking controller alone. We address this by extending the state and control reference trajectories for few extra timesteps and filling the new elements with æ and w/ 1 respectively. We hypothesize that this adjustment handles the sim-to-real gap introduced by actuator latency and response times.
We visualize an example state and control trajectory in Figs. 11-14 (same trajectory as the one shown in Fig. 8). The tip position plots show the extension of the nominal trajectory by about 0.1 s. The plot of tip y-position shows how the additional time is necessary for the robot to reach the final position. In addition, our trajectory optimization formulation includes a quadratic cost on control effort, resulting in relatively smooth control trajectories as seen in Figs. 13-14. This aids in making both cart velocity and fPAM pressure reference trajectories easier to track. We also note that the actual tip position trajectories closely match the nominal trajectories, except that there delay that shifts the actual trajectories slightly to the right. We also that the actual fPAM and cart velocity trajectories closely match their commanded trajectories but with slight shift to the right due to actuator latency. We find that this planning and control method still performs well, despite this delay from actuator latency.
slight
Tip Position Y (m)
Nominal
Tip Position Z.(m (m)
0.7
Nominal
0.2
Actual
Actual
0.6
-0.2
0.5
0
0.5
1
1.5
2
0
0.5
1.5
2
Time (5)
Time (s)
Figure 1I: Tip p-positionnn time for reaching target free space. The nominal trajectory (blue) was computed with trajectory optimiza- lion. motion plan deployed on the physical hardware traking controller and the actual trajectory is visualized in red.
Figure 12: z-position over time for reaching target free space.
The nominal trajoctory (bluc) was computed with trajectory optimiza- This motion plan deployed on the physical hardware with tracking controller and the actual trajectory visualized in red.
IPAM Pressure (kPa)
20
Command
Cart Velccity (m/s)
Command
Actual
Actual
10
00.5
0.5
1.5
2
0.5
1.5
2
Time (s)
Time (s)
Figure velocity time for reaching target space. The blue trajectory shows the actuator commands when tracking the reference trajectory hardware. trajectory the actual cart velocity.
Figure pressure over reaching target free space. The blue trajectory shows the actuator commands when track- ing the reference trajectory on hardware. The red trajectory shows the actual (PAM pressure.
I Comparison to PID Control for Free Space Target Reaching Task
We compare our learned policy to a ceoportional-ineggaal-deeeeeeeee space target reaching task. This serves as a baseline controller that does not reason about leveraging swinging, and we show that it is largely unsuccessful in this task.
(PID) controller on the free
We use two separate PID controllers to achieve tip-position control. We use cart actuation to drive the y-coordinate error towards zero, and use fPAM actuation to drive the z-coordinate error
8

---

towards zero. We acknowledge that the fPAM actuation also affects the tip y-coordinate, but find that the y-coordinate PID controller able to account for this disturbance. Below are the equations used to compute the PID control commands (actions) for cart velocity and fPAM pressure:
Ceart-vel -Kp,y Kav Kiy/ / eydlt
0p = Kp,222 - Ka,tzz - Ki,z f (eadt
where By Mhip Vhorget and E2 2tip Zurget are the y-error and z-error of the tip po- sition, respectively. K. the PID controller gains. We took manual approach to tuning our PID gains that is similar to the Ziegler-Nichols method. Our final gains were Kpy and Kp Kp.= = 20. We found that derivative and integral gains had little effect on overall per- formance, and that increasing these gains led to instability, so we ultimately set these to zero
We run this PID controller for the same 54 tar- get positions discussed in Sec. 4, and the robot achieves success rate of 17% across the 54 larget positions. With this control method, the robot is not able to reach any of the target po- sitions with a 7-coordinate of 0.6 or greater; the maximum tip z-coordinate reached across 0.56 The
AL
PID
0.8
0.6
0.4
all PID control experiments PID controller limits the robot's workspace be- cause it aims to greedily reduce tip position error and does not incorporate any reasoning about swinging, significantly reducing its abil- ity to reach higher target positions. We illus- trate this in Fig. 15. Using our learned policy, the robot is able to reach the target tip position by building up momentum over multiple swings increase its tip height. In contrast, using the PID controller, the robot is unable to reach the target tip position, it simply moves directly towards the target and fails to bring the tip high enough. This demonstrates that the free space target reaching task is not possible without sell-timed actuation that leverages swinging motion.
0.25
0
0.25
0.255
0.25
(m)
(m)
Figure 15: Comparison of learned policy (RL) vs. PID control (PID) for reaching target tip position
space. black
at the larget position with rudius 4 cm. The RL control policy is able to perform high-speed swinging behavior to reach target position. The PID comtrol is unable track the targel position because greedily appronches the target position directly.

---

References
[1] I. Robert J. Webster and B. A. Jones. Design and kinematic modeling of constant curvature continuum robots: A review. The International Journal of Robotics Research, 29(13): 1661- 1683, 2010.
[2] N. Naughton, Sun, A. Tekinalp, T. Parthasarathy, G. Chowdhary, and M. Gazzola. Elastica: compliant mechanies environment for soft robotic control. IEEE Robotics and Automation Letters, 6(2):338--3366, 2021.
[3] R. K. Katzschmann, C. D. Santina, Y. Toshimitsu, A. Bicchi, and D. Rus. Dynamic motion control of multi-segment soft robots using piecewise constant curvature matched with an aug- mented rigid body model. In IEEE International Conference on Soft Robotics, pages 454-461, 2019.
[4] D. Bruder, Gillespie, C. Remy, and Vasudevan. Modeling and control of soft robots using the koopman operator and model predictive control. Robotics: Science and Systems, 2019.
[5] R. Jitosho, N. Agharese, A. M. Okamura, and Z. Manchester. A dynamics simulator for soft growing robots. In IEEE International Conference on Robotics and Automation, pages 1 11775- 11781, 2021.
[6] M. Bern, Y. Schnider, Banzet, N. Kumar, and S. Coros. Soft robot control with learned differentiable model. In IEEE International Conference Soft Robotics, pages 417-423, 2020.
[7] A. Centurelli, L. Arleo, A. Rizzo, S. Tolu, C. Laschi, and E. Falotico. Closed-loop dynamic control of a soft manipulator using deep reinforcement learning. IEEE Robotics and Automa- tion Letters, 7(2):4741-4744) 2022.
[8] M. Dubied, M. Y. Michelis, Spielberg, and R. K. Katzschmann. Sim-to-real for soft robots using differentiable FEM: Recipes for meshing, damping, and actuation. IEEE Robotics and Automation Letters, 7(2):5015-5022, 2022.
[9] K. Hagiwara, K. Yamamoto, Y. Shibata, M. Komagata, and Y. Nakamura. On high stiffness of soft robots for compatibility of deformation and function. Advanced Robotics, 36(19):995- 1010, 2022.
[10] Z. Zhang. Zhang, P. Ma, Nava, Du, Arm, W. Matusik, and Katzschmann. Sim2real for soft robotic fish via differentiable simulation. In JEEE/RSI International Confer- ence Intelligent Robots and Systems. IEEE, 2022.
[11] Y. Li, X. Wang, and K.-W. Kwok. Towards adaptive continuous control of soft robotic ma- nipulator using reinforcement learning. In JEFE/RSJ International Conference on Intelligent Robots and Systems, pages 7074-7081, 2022.
[12] H. Zhang, R. Cao, S. Zilberstein, F. Wu, and X. Chen. Toward effective soft robot control via reinforcement learning. In Intelligent Robotics and Applications, pages 1 173-184, Cham, 2017. Springer International Publishing. ISBN 978-3-319-6528944
[13] U. Yoo, H. Zhao, A. Altamirano, W. Yuan, and C. Feng. Toward zero-shot sim-to-real transfer learning for pneumatic soft robot 3d propriaceptive sensing. arXiv preprint arXiv:2303.04307, 2023.
[14] S. Kriegman, A. Mohammadi Nasab, D. Shah, H. Steele, G. Branin, M. Levin, Bongard, and R. Cramer-Bottiglic Scalable sim-to-real transfer of soft robot designs. In The 3rd IEEE International Conference Soft Robotics, pages 359-366, 2020.
10

---

[15] C. D. Santina. The soft inverted pendulum with affine curvature. In JEEE Conference on Decision and Control, pages 4135-4142, 2020.
[16] L. Weerakoon and N. Chopra. Swing up control of a soft inverted pendulum with revolute base In IEEE Conference Decision and Control, pages 685-690, 2021.
[17] M. Grube, J.C. Wieck, and R. Seifried. Comparison of modern control methods for soft robots. Sensors, 22(23)94464, 2022.
[22] D. A. Haggerty, M. J. Banks, E. Kamenar, A. B. Cao, P. C. Curtis, I. Mezić, and E. W. Hawkes. Control of soft robots with inertial dynamics. Science Robotics, ((81))edd66866 2023.
[19] G. Thuruthel, E. Falotico, Renda, and C. Laschi. Model-based reinforcement learning for closed-loop dynamic control soft robotic manipulators. IEEE Transactions on Robotics, 35 11):124-134, 2019.
[20] W. Qiuxuan, Y. Gu, Y. Li, B. Zhang, S. Chepinskiy, J. Wang, A. Zhilenkov, A. Krasnov, and Chernyi. Position control of cable-driven robotic soft arm based on deep reinforcement learning. Information, 11(6)331, 2020.
[21] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine, and V. Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In IEEE International Conference on Robotics and Automation, pages 4243-4250, 2018.
[22] A. A. Rusu, M. Večerík, T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell. Sim-to-real robot learning from pixels with progressive nets. In Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 262-270, 2017.
[23] S. Satheeshbabu, N. K. Uppalapati, T. Fu, and G. Krishnan. Continuous control of soft continuum using deep reinforcement learning. In IEEE International Conference Soft Robotics, pages 497-503, 2020.
[24] N. K. Uppalapati, B. Walt, A. J. Havens, A. Mahdian, G. Chowdhary, and G. Krishnan. A berry picking robot with a hybrid soft-rigid arm: Design and task space control. In Robotics: Science and Systems, page 95, 2020.
[25] D. Bianchi, M. G. Antonelli, C. Laschi, A. M. Sabatini, and E. Falotico. Softoss: Learning to throw objects with a soft robot. IEEE Robotics & Automation Magazine, 2023.
[26] K. M. Lynch and F, C. Park. Modern Robotics: Mechanics, Planning, and Control. Cambridge University Press, USA, 1st edition, 2017. ISBN 1107156300.
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[28] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi,
R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre. Z. Wang, T. Pfaff,
Y. Wu, R. Ring, Yogatama, D. Wünsch, K. McKinney, O. Smith, Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and Silver. Grandmaster level in StarCraft using multi-agent reinforcement learning. Nature, pages 1-5, 2019.
[29] P. Zhu, X. Li, P. Poupart, and G. Miao. On improving deep reinforcement learning for pomdps. arXiv preprint arXiv: 1004::7770777, 2017.
[30] N. D. Naclerio and E. W. Hawkes. Simple. Low- How-Hysteresis, Foldable, Fabric Pneumatic Arti- ficial Muscle. IEEE Robotics and Automation Letters, 5(2):3406-3413, 2020.
11

---

[31] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning. arXiv preprint a0X
2021.
[32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1rXivv1707.0000 1707.06377, 2017.
[33] D. Makoviichuk and V. Makoviychuk. rl-games: high-performance framework for rein- forcement learning. http:s////g.ttuub.om///////////
_gamess, 2022.
[34] T. Howell, S. Cleac' h, J. Kolter, M. Schwager, and Z. Manchester. Dojo: A differentiable physics engine for robotics, arXiv preprint arXiv:2203. 00806, 2022.
12


Reviews:
[{"review_id": "pe5RhXNQqv", "reviewer": "Reviewer_sYng", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to learn a reinforcement learning policy for a mixed actuator soft robot. The policy is trained in simulation and several adjustments to the simulation dynamics in combination with a system identification approach and a recurrent policy architecture are proposed in order to increase the performance of the controllers after transfer to the real system. The final policy is able to achieve many goal positions for which a combination of linear cart and soft actuator control is required. Several ablations are performed which demonstrate that the proposed extensions to the simulation engine are pertinent for performance. A PID controller baseline does not perform well, while a trajectory-optimization baseline is used to establish an upper performance bound---at strongly increased computational cost. ", "summary_of_recommendation": "This paper is clearly written and well executed. The problem of sim2real transfer is important and while the study only addresses a simple soft system, the proposed results are well motivated and convincing. Good paper. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Q1: Section 3.3: It is interesting that so few of the proposed components deal directly with the soft part of the robot, most seem to increase simulation accuracy for the cart actuation. Was the dynamics estimation for the soft part already good enough?\n\nQ2: Why did the planning stage take 4.7 hours? I understand that real-time planning is quite difficult to achieve with TO, but this planning time seems excessive. Could you explain further?\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am impressed by this paper. The idea is formulated in a clear way, the results are convincing and the ablations show the effects of almost all components.\nI enjoyed watching the video, which presented the experiments very clearly.\nWhile the \"soft\" control aspect of the arm is quite limited, due to the low-dimensional nature of the setup, and there remain minor issues with framing and one or two missing explanations in the text, I think this paper is a very well executed step towards better control methods for soft actuators.\n\nFinally, the limitations section should put stronger emphasis on the fact that a low-dimensional system was employed. It is very likely that more complex soft robots will require more components that specifically deal with the sim2real gap in the soft structures.\n\nDetailed comments:\nFigure 2 is not mentioned in the text. I think it would also further understanding if the policy architecture was explained in a more detailed way in text somewhere. By the time I read about the ablations, I had almost forgotten about the recurrent architecture of the policy. These choices should be stated more clearly, similar to the other components.\n\nRelated Work: Potentially add some explanations on how soft robot control is to be distinguished from soft actuation. Robots like the one in [A] also use soft/muscle-like actuators, but are otherwise rigid. These robots have already achieved highly dynamic behaviors, so it would be good to discuss how the employed soft robot is different. \n\nLine 169: Add citations about IIR use of pressure modelin for soft robots\n\nLine 178: We design a a cart controller [...]\nThis paragraph is written in a confusing way. You mention that a constant force cannot be applied to the cart. Is this meant to apply to simulation or to the real system?. In simulation one can set any acceleration directly. Perhaps it would be better to argue that the real car cannot reach any velocity instantly and therefore the controller in simulation has to be adapted in order to match the real system better. One way of doing that is the velocity tracking/acceleration tracking controller.\n\nFigure 6: Maybe also plot the simulated cart velocity when no delay is added. That would make the magnitude of the effect clearer.\n\nSection 4.2: Consider marking the goals that were used in the ablations in Figure 7. \n\nLine 275: What exactly is \"domain randomization on policy actions\"? I cannot find the exact explanation of this.\n\nFinally, consider adding a disclaimer to the video: \"Turn sound on\", I missed it the first time.\n\n\n\n[A] Learning to Play Table Tennis From Scratch using Muscular Robots. Büchler et al.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "doDOcgqWluP", "reviewer": "Reviewer_UnPS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper aims training control policies for soft robots using reinforcement learning and simulation. The soft robots are very hard to develop a high fidelity simulator, so RL is not easy to use with. The authors show that they can solve the problem for a single joint simple soft robot arm attached to a linear actuator. The authors model the arm with N-joint arm simulation, use domain randomization and careful tuning of simulation parameters such as latency. The results show that the trained policy can reach to different tip points, and the author show results on hardware as well. The ablations also show that different components are crucial for success.", "summary_of_recommendation": "I recommend the acceptence of the paper because it shows interesting results, but the evaluation could be much better if the authors used more complex problems or form factors.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Can do authors show some behaviors on more complex problems (i.e. flipping a piece of paper, dragging a cloth, etc) or form factors (quadruped with soft legs)? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem of using RL with soft robotics is important for robotics. Soft robots are desirable for many reasons but they are hard to control. The authors introduce these benefits and challenges clearly. \n\nThe proposed method is clearly described and tested on the hardware. The results are promising and ablation studies are also very useful to identify what is crucial for the policy.  My only concern about the paper is the simplicity of the experimental setup. If the authors used a more complex robot with a different form factor, the paper could be much more impressive. As an example, \nI would be very interested in to see the results if the problem studied was very sensitive to the force applied to the tip (which would cause deformation of the robot due to external forces). In any case, the results are well analyzed and the authors show that RL can be used on a soft robot without an accurate simulation model. The paper is definitely has some nice contributions and opens interesting future resarch directions.\n\nThe plots are not possible to understand when printed in Black and White.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "FHSep_X98Ee", "reviewer": "Reviewer_y2wh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a sim-to-real transfer approach for dynamic control of soft robot arms. It prepares a linear dynamics model for fabricated Pneumatic Artificial Muscles (fPAM) where the dynamics parameters are identified with the 30s of the real-world data. Then, domain randomization is applied to the dynamics parameters of the linear dynamics model during RL training.  The real robot experiments showed that the proposed method achieved higher success rates than the PID and trajectory optimization-based approaches. The ablation studies revealed that the combination of all proposed components improved the performance.", "summary_of_recommendation": "I respect the engineering efforts for the sim-to-real transfer; however, this study would lack the technical contribution for machine learning parts. For this reason, I would like to recommend a weak rejection of the current manuscript.\n\nAfter the rebuttal comment: I raise my recommendation from weak reject to weak accept. Although the technical contribution of the learning part is weak, the simple simulator enabling soft robot sim-to-real is good news and benefits the robot learning community. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Could you discuss the novelty of the learning part by referring to the other citations about sim-to-real transfer for soft robots? \n- The filter parameter of the regulator and latency of the linear actuator is designed experimentally. Thanks to them, it is good to see the simulator behavior is close to the real one. What happens if these parameters are randomized during training? Will the policy become more robust?\n- It will be better to show what parameters were randomized in not the supplemental material but the manuscript.\n- Did this study consider the filtering of the regulator and latency of the linear actuator for trajectory optimization? If not, testing trajectory optimization with them will be a more fair comparison.\n- Could the proposed approach perform trajectory tracking tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strength\n- Overall, this paper has good clarity, is well-written, and is easy to follow. The figures are beautiful. The demo video is impressive and helps the readers' understanding.\n- Great engineering effort to make the sim-to-real transfer of soft robots tractable by considering the soft robot, linear actuator, and pressure regulator dynamics. Demonstrating dynamic control for soft robots using sim-to-real approaches should be highly respected.\n- Good quality: comparisons and ablation studies are explored in detail.\n\n# Weakness\n- My primary concern is weak originality and the lack of technical contribution to learning parts. It seems just to apply domain randomization techniques, which have been widely used in previous studies. Considering actuator dynamics for sim-to-real may not be a substantial contribution. For example, [r1] considers actuator dynamics for sim-to-real transfer. It would be natural to consider the actuator dynamics even in soft robots.\n   - [r1] Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2022). Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62), eabk2822.\n- This study has weak significance and relevance. Although it poses an interesting problem, dynamic soft robot manipulation, it may not address the problem of robot learning.\nBesides, it solves this problem with the existing approaches mentioned above. This study applies a learning technic but does not improve it. CoRL audiences may not be interested because this study mainly introduces actuator modeling.\n- This study needs to add more citations about sim-to-real transfer for soft robots and discuss why the existing sim-to-real approach is difficult. For example, the following literature [r2-4] may be close to this study.\n   - [r2] Dubied, M., Michelis, M. Y., Spielberg, A., & Katzschmann, R. K. (2022). Sim-to-real for soft robots using differentiable fem: Recipes for meshing, damping, and actuation. IEEE Robotics and Automation Letters, 7(2), 5015-5022.\n  -  [r3] Hagiwara, K., Yamamoto, K., Shibata, Y., Komagata, M., & Nakamura, Y. (2022). On high stiffness of soft robots for compatibility of deformation and function. Advanced Robotics, 36(19), 995-1010.\n  -  [r4] Li, Yingqi, Xiaomei Wang, and Ka-Wai Kwok. \"Towards Adaptive Continuous Control of Soft Robotic Manipulator using Reinforcement Learning.\" 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.\n- I respect the engineering efforts. Also, the combination of DR and actuator dynamics showed better performance.  However, it also implies that this study provides many assumptions to achieve the sim-to-real transfer. The ablation studies revealed that actuator and sensor models were more impactful than domain randomization.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a reinforcement-learning- and sim-to-real-based approach to control a soft robot.

Reviewers appreciated the clear formulation, strong motivation, tests of physical hardware, and video. Achieving sim-to-real transfer for soft robots with a simple simulator is an excellent contribution. Reviewers also praised the papers writing quality, the video, and the thoroughness of ablation studies.

Reviewers raised concerns that the simplicity of the robot in experiments may suggest limitations on the proposed approach that should be more clearly stated in the limitations. A reviewer also provided a detailed related work that the authors should consider including.

Reviewers also appreciated the author's rebuttal, raising at least one reviewer's rating. As a result, all reviewers agree that the paper is above-bar and should be accepted.
Decision: Accept (Poster)