## Combine classical learning and prior knowledge

Optimisation-based algorithms can only solve a task if the goal belongs to the same convex space as the effector's one. The CartStemContact shows that we can continue to use the advantage of optimisation-based algorithms by learning the transition between two configuration spaces, even if they correspond to different convex spaces. This has two benefits: it can improve general sample efficiency and overcome the limitations of a convex approach. However, this study should be extended to include simultaneously learned and not learned internal agents. This would be particularly interesting in the case of the RodManipulator, where two opposite contact configurations can be handled using convex optimisation, whereas the other configurations are learned.

Another prior knowledge is provided by the expert trajectories. Unlike Behavior Cloning methods or other approaches that use entire trajectories, the definition of the knowledge graph and sub-part of the trajectories guide the learning by defining the passage point necessary to achieve the desired behavior. In the RodManipulator example, the use of expert trajectories to reconfigure contacts is easier to obtain than Behavior Cloning because it is independent of the task to be solved and does not correspond to an entire episode. Although this trajectory is not optimal, the algorithm can use it to solve the task and improve the efficiency of its own strategy.

## Overall method

We propose to use the contact configurations to create the configuration spaces, motivated by the importance of optimisation approaches in soft robotics. However, other approaches could also be considered, such as automatically defining the configuration spaces based on the robot's workspace in a specific contact configuration. This method is an intuitive way to incorporate prior knowledge into learning algorithms. Even though it can be applied to other fields than soft robotics, it is particularly relevant for the control of soft robots as traditional methods for path planning and predictive control with contact are limited.

The complexity of the general structure of the network has some limits. Each agent corresponds to one network, increasing the number of hyperparameters and update at each time step. Moreover, the definition of the configuration spaces can influence the convergence speed. In fact, agents share information, and an under-exploited agent takes more time to learn and thus to provide reliable information. There are several ways to improve this method. First, it would be interesting to extend the application domain of this technique, for example, by using multi-task and
dynamic graph. With a method for dynamically learning configuration spaces, these
spaces will be useful for the task and will be built to solve the task. In addition, the
more multi-task objectives the robot has to solve, the more general the configuration
spaces will be. Modifications of the learning conditions could also be considered,
such as local updates of the networks when the number of configuration spaces
becomes large or the reward of the external agents depending on the length of the
sequence of action.

4.3 Conclusion

In this Chapter, we investigate two methods for solving high-level tasks using
learning while maintaining the efficiency of inverse models and optimisation. Both
methods describe the state of a robot in a specific manner. The proxy approach
involves describing the robot in a reduced configuration space by using a fast model
extracted from the FEM model. This proxy can be used to generate a high-level
trajectory for solving a task using RL algorithms, which can then be corrected
and transferred to the FEM model using an inverse model and optimisation. This
motion planning method can be applied to complex soft robotics systems, and has
successfully solved manipulation and locomotion tasks. The graph approach classifies
the robot state in different configuration spaces based on the observation of contact
configurations. These spaces provide prior knowledge for learning and can be used
to combine learning and optimisation approaches. If the configuration of the robot
is in a convex space, convex optimisation can be used to control it efficiently. RL
algorithms can be used if the robot has to move to another space, for example, to
reconfigure the contacts.

However, these methods have limitations. There is a loss of expressiveness in
learning. The configuration spaces graph method requires to construct the graph,
which can be difficult when there are many contacts and these contacts are difficult to
localize. The proxy model is a simplified version of the FEM, and learning algorithms
can find biased solutions by exploiting the differences between the models. The
discussion sections suggest some ideas for dealing with these limitations. The
methods proposed in this chapter are therefore promising for control in soft robotics.
Another limitation is that all the tests were carried out in simulation. In order to
fully validate these methods, they must also be transferred to physical robots. This
transfer has its own difficulties, such as the quality of the FEM model for the proxy
method. In the graph method, detection of contact configurations from the sensors
is necessary to use it on a physical prototype.
Condensed FEM model for
control and design in soft robotic
applications

Contents

<table><tr><td>5.1</td><td>Learned condensed FEM model</td><td>94</td></tr><tr><td>5.1.1</td><td>Condensed FEM Modelling</td><td>95</td></tr><tr><td>5.1.2</td><td>Learned Condensed FEM modelling</td><td>97</td></tr><tr><td>5.1.3</td><td>Soft robot control using a condensed FEM modelling without contacts</td><td>101</td></tr><tr><td>5.1.4</td><td>Soft robot control using a condensed FEM modelling with contacts</td><td>105</td></tr><tr><td>5.1.5</td><td>Embedded control of soft robot using a condensed FEM modelling</td><td>111</td></tr><tr><td>5.1.6</td><td>Condensed FEM modelling extended to calibration and design</td><td>114</td></tr><tr><td>5.1.7</td><td>Discussion</td><td>124</td></tr><tr><td>5.2</td><td>Condensed dynamics for optimal control in soft robotics</td><td>125</td></tr><tr><td>5.2.1</td><td>Condensed dynamics for trajectory optimisation</td><td>125</td></tr><tr><td>5.2.2</td><td>Trajectory optimisation</td><td>128</td></tr><tr><td>5.2.3</td><td>Scenario 1: Use both derivative of the linear and non linear part of the dynamic in the backward step.</td><td>129</td></tr><tr><td>5.2.4</td><td>Scenario 2: Constraint actuation with the PneumaticTrunk.</td><td>130</td></tr><tr><td>5.2.5</td><td>Scenario 3: Multiple goals with the Lianford robot.</td><td>131</td></tr><tr><td>5.2.6</td><td>Discussion</td><td>133</td></tr><tr><td>5.3</td><td>Conclusion</td><td>134</td></tr></table>
<figure><img src="image_48.png" /><figcaption>“6. Communion.</figcaption></figure>

— Nemo

We live together, and we all resemble one another. We develop a collective identity based on shared ideas. Although our individuality is distinct, it is intertwined with the larger whole. This individuality is expressed through words and ideas that resonate with others. We often return to the same ideas and words. This is a matter of doing something different from them.
In previous chapters, various methods for describing the state of a soft robot
are explored. On one hand, FEM is used to describe the state of the robot (Chapter
3), while on the other hand, simplified descriptions of the state are proposed using a
proxy model or by dividing the observation space into several configuration spaces
(Chapter 4). All these methods can be used to solve high-level tasks, with respect to
the underlying modelling assumptions. The objective of this Chapter is to propose an
intermediate method based on a reduced description that retains the expressiveness
of the FEM model. We propose to use the condensed FEM model presented for
one-step control (see Section 2.2.3) to implement this method. The condensed FEM
model represents the kinematics or dynamics of the robot in the space of constraints
 actuators, effectors, and contacts). It can be used in several ways.

First, the condensed model can be learned. FEM modelling is a powerful
tool for modelling the behavior of soft robots; however, its use in real time can be
difficult for non-specialists in numerical computation. Machine learning approaches
enable rapid prediction of robot behavior but use the robot's mechanics as a "black
box", particularly for generating training data. The approach developed in the
first section is a "white box" approach, in which the mechanical structure of the
model is used to perform targeted learning (5.1.2). This approach preserves the
mechanical properties of the predicted quantities for contact (5.1.4) and non-contact
(5.1.3) control and for design applications (5.1.6). It provides real-time prediction
capability for embedded control (5.1.5). In the second section, the condensed model
is used for trajectory optimisation with soft robots (5.2). Trajectory optimisation and
multi-time-step control of soft robots using FEM models and optimisation methods is
challenging because of the high dimension of the manipulated mathematical systems.
For instance, using the optimal control formalism presented in Section 2.2.3 with the
robot's FEM model involves solving QP sequences with matrices of sizes proportional
to the size of the simulation mesh. The condensed model resolves this dimensional
problem by using small matrices in the backward phase of DDP.

5.1 Learned condensed FEM model

In this section, we present a framework to learn a compact mechanical rep-
resentation of a soft robot based on the condensed FEM model. This model can be
used in a quasi-static inverse control of soft robots and handle contact management.
We highlight its modularity by showing that the control of a complete gripper on
a manipulation task can be performed using the condensed FEM model learned
on a single soft finger. Then, we evaluate the low memory consumption of this
model. We show it is suitable for embedded control on the case of the inverse
control of a physical pneumatically actuated soft Trunk robot. Finally, we extend the
condensed FEM modelling to manage design parameters and show how we can use its inherent differentiability in both calibration and design optimization applications. The proposed framework and contributions are summarized in Figure 5.1.

<figure><img src="image_49.png" /><figcaption>Fig. 5.1: Illustration of the proposed framework and its applications. A physical soft robot is modeled using the FEM. The condensed modeling is obtained by projecting the FEM characteristic matrices in the constraint space. A neural network is trained offline for predicting the condensed FEM modeling from both the initial robot condensed state and applied actuation. For a given robot, a single learned condensed FEM modeling may be used in several applications i.e. for real-time embedded control, for inverse control involving a set of predefined contact points, for controlling several coupled similar robots altogether as well as for calibration and design optimization.</figcaption></figure>

### 5.1.1 Condensed FEM Modelling

As we explained in Section 2.1.3, the mechanical behavior of soft robot de- formations can be described through continuum mechanics. In this section the quasi-static formulation is considered:

$$K(q)\Delta q = f_{ext} - f_{int}(q) + H_a^T \lambda_a + H_c^T \lambda_c \quad (5.1)$$

where $\Delta q$ are small displacement of nodes around their current position $q$. We consider elastic behaviours expressed from a Poisson ratio $\nu$ and a Young Modulus $E$ though in principle the condensed FEM model could be learned for any elastic model. In addition, the definition of $\delta_c$ changes slightly from that given in Section 2.1.3, because we are now considering probable contact zones: at the learning stage, we do not yet have an obstacle, or a force applied by another object on the contact zone. $\delta_c$ can not be defined as the signed distance between two objects. Instead, a
displacement $\delta_c$ of the contact points relative to their rest position is imposed, in order to reproduce the effect of an obstacle on the contact zone.

A robot can be modelled using its inputs and outputs. The one-timstep control proposed in the Section 2.2.3 introduces the projection of the dynamics into the constraint space. The resulting mechanical quantities gather the mechanical coupling between passive constraints (effectors $e$) and active constraints (actuators $a$ and contacts $c$). This is a method of representing the inputs and outputs of the system while keeping the expressiveness of the underlying FEM model. It is possible to write these projected equations in the quasi-static context. The integrated impedance matrix $A$ is then replaced by the stiffness matrix $K$. In this case, $W_{ij} = H_i K^{-1}(x) H_j^T$ for $i, j \in \{e, a, c\}$ is the compliance (inverse of the stiffness) projected into the effector, actuator and/or contact spaces. Both the $W_{ij} = H_i K^{-1}(x) H_j^T$ for $i, j \in e, a, c$ and $\delta_i^{free}$ for $i \in \{a, c, e\}$ could be seen as a reduced model of the robot as illustrated in Figure 5.2.

<figure><img src="image_50.png" /><figcaption>Fig. 5.2: Illustration of the condensed FEM model for both the Diamond and Soft Finger robots. They are respectively actuated by 4 and 2 cables. In both cases, a FEM model of the robot (left) is used to learn a condensed model (right). The reduced model is based on the projection of the compliance matrix into the constraint space, resulting in drastically reducing the number of characteristic dimensions describing a robot state.</figcaption></figure>

In an inverse problem, this reduced model can be used to control soft robots (see Section 2.2.3). In the direct problem, when the actuator forces intensity $\Delta\lambda_a$ is varying, then the corresponding motion of the effectors $\Delta\delta_e = W_{ea}\Delta\lambda_a$ can be computed. As actuators are also controlled in displacement, varying the position of the actuators $\Delta\delta_a$ lead to obtain the motion created on the effectors through $\Delta\delta_e = W_{ea}W_{aa}^{-1}\Delta\delta_a$. This equation provides the Jacobian $J = W_{ea}W_{aa}^{-1}$ of the soft robot direct kinematics. It shows that both direct and inverse models of soft robots
can be derived as long as an approximation of equations 2.14 giving the effective state of the actuation $\delta_a$ and the contact $\delta_c$ are available.

## 5.1.2 Learned Condensed FEM modelling

In this section, the learned condensed FEM model is introduced. The different applications tackled with the learned condensed FEM model are also presented, namely modelling and control with and without contacts, embedded control and design optimization.

### Mechanical Representation Learning

We suppose that a high dimensional state of the robot is available through simulation. For instance, when using tetrahedral FEM with linear interpolation, the state of the robot is the position of all the vertices of its mesh. For reducing the dimension of the robot state, the global compliance matrix projected in the constraint space $W$ (see section 5.1.1) is a good candidate because it is a small dimensional matrix, independent of the size of the FEM mesh and it expresses a direct relationship between constraints. In combination with the free displacement vector $\delta^{free}$ (see Section 2.2.3), it is relevant enough for modelling and controlling a soft robot. In theory, $W$ depends on the global state of the FEM model. However, in the case of a deformable robot, the deformations are due to the state of the actuators and contacts on the robot. It is therefore possible to obtain $W$ only for states that can be reached by the robot, which depend on the state of the contacts and actuators. The computation of $W$ is expensive because it involves the multiplication and inversion of several large matrices and this needs to be computed at every simulation step. This is an incentive for learning it rather than computing it from mechanical simulation. Learning these mechanical quantities rather than non-supervised ones makes it possible to obtain interpretative robot representations useful for simulating, controlling and even design the robot using mechanical equations.

The learning problem is then resumed as the following. Let $\delta_a, \delta_c, W$ and $\delta^{free}$ be respectively the effective states reached by both actuation and contact constraints when applying a specific active constraint state as well as corresponding projected compliance matrix and free displacement of the robot. The goal is to reconstruct $(W, \delta^{free})$ from initial values $(W_0, \delta_0^{free})$ when $\lambda_a = 0$ and $\lambda_c = 0$, and current active constraint states $\delta_a$ and $\delta_c$. This leads to learn a function $F$ such that

$$ \tilde{W}, \tilde{\delta}^{free} = F(\delta_a, \delta_c, W_0, \delta_0^{free}) \qquad (5.2) $$
$W_0$ is a good compliance initialization since it already captures the mechanical coupling between constraints and the compliance matrix varies smoothly from that initial position in the workspace of the robot. This mathematical formulation excludes cases when the robot may have several configurations for a same actuation. This could happen when using highly non linear material models or when considering non-linear phenomenons such as buckling.

## Learning Model

The compliance matrix $W$ is a symmetric matrix that represents the relations between the different constraints. It is formed by the mechanical matrices introduced in Eq. 2.14:

$$W = \begin{bmatrix} W_{ee} & W_{ea} & W_{ec} \\ W_{ae} & W_{aa} & W_{ac} \\ W_{ce} & W_{ca} & W_{cc} \end{bmatrix} \qquad (5.3)$$

The magnitude of both the values of $W$ and $\delta^{free}$ depends of both the type of constraints and the measurement units chosen for the simulation. For avoiding vanishing gradients during the learning process, both the input and output data are element-wise standardized. This standardization is computed once using all the training set, using the formula:

$$X = \frac{X - X_{mean}}{X_{std} + \epsilon} \qquad (5.4)$$

where $X$ is the tensor $(\delta_a, \delta_c, W, \delta^{free})$, $X_{mean}$ and $X_{std}$ are respectively the mean and standard deviation of the training set, $\epsilon$ is a small quantities to avoid computational singularity and $X$ is the standardized data. In order not to overload the notations, standardized inputs for learning and not standardized inputs for control are systematically considered afterwards.

As the matrix is symmetric, the networks are trained in a supervised manner through minimizing the reconstruction loss:

$$L = \sum_{\delta_a, \delta_c} [||\tilde{W}^{tri} - W^{tri}||^2 + ||\tilde{\delta}^{free} - \delta^{free}||^2] \qquad (5.5)$$
where $W^{tri}$ is the upper triangular part of the matrix $W$, $\tilde{W}^{tri}$ and $\tilde{\delta}^{free}$ are the predicted values of $W^{tri}$ and $\delta^{free}$ as given by the network for actuation displacement $\delta_a$ and contact displacement $\delta_c$. As all data are standardized, the computed loss function obtained on different training data are somewhat comparable. MLP are sufficient for learning a mechanical representation for control and design tasks. Input and output of the network are vectorized data.

As the framework is a supervised learning framework, pre-computation steps where data are collected and the learning model is trained are first performed. The data ($\delta_a, \delta_c, W^{tri}, \delta^{free}$) are obtained through simulation and stored in a database. Each data point is associated to a different robot configuration after applying an active constraint displacement ($\delta_a, \delta_c$). The range of possible values of both $\delta_a$ and $\delta_c$ are bounded in continuous intervals. Sampling the active constraint space is done using Scrambled Halton sampling algorithm [Mas04] for building the training dataset and random search for the test dataset. Compared to homogeneous grid search strategies, it enables a homogeneous and better filling of a high dimensional sampling space and therefore a better scalability to robot having numerous actuators and contacts. The number of points of the test set is chosen to amount for 25% of the number of points of the training set. Once the data is acquired, learning consists in training the network to predict the quantities ($W^{tri}, \delta^{free}$) from the quantities ($\delta_a, \delta_c, W_0^{tri}, \delta_0^{free}$) using MLP and Eq. 5.5. Unless mentioned otherwise, the used network in the experiments introduced in this section is a fully connected MLP with 3 hidden layers of 450 nodes each. Regarding the learning rate, it is initialized to $10^{-4}$. In all our experiments, an adaptive scheduler is systematically applied for reducing the learning rate when the optimizer falls in a local optima.

## Control applications

Once learned, the condensed FEM model is used in a control loop as described in the following workflow:

1. $\delta_a, \delta_c, W_0^{tri}$ and $\delta_0^{free}$ are recovered from FEM simulation. During the first iteration of a soft robot control experiment, initial $\delta_a$ and $\delta_c$ are obtained for zero active constraint efforts applied on both the actuators and contacts.

2. Prediction of $(\tilde{W}^{tri}, \tilde{\delta}^{free}) = F(\delta_a, \delta_c, W_0^{tri}, \delta_0^{free})$ using the trained network. $\tilde{W}$ is reconstructed from $\tilde{W}^{tri}$. Although the condensed FEM model is learned
on a predefined set of fixed goals $p_{goal}^{train}$, one for each of the effectors, it can be easily adapted to other goals $p_{goal}$ during the control phase with the formula:

$$ \delta_e^{free} = \tilde{\delta}_e^{free} + p_{goal}^{train} - p_{goal} \qquad (5.6) $$

3. An inverse optimization problem such as the one from Eq. 2.15 is solved using a quadratic optimization solver. The aim is to compute the actuation effort vector $\lambda_a$ for controlling the robot. The C++ library Prox-QP [Bam+22] is used for implementing the solver in our work.

4. The actuation effort $\lambda_a$ is applied on the robot and the respective new actuation and contact states $\delta_a$ and $\delta_c$ are recovered. At this step, corrective motion can be applied for improving performance, stability and precision. In particular, it enables to palliate the two main sources of errors which are due both from the learning and the simulation-to-reality gap.

5. The control loop starts again from step 1 until convergence, that is until an accuracy or stability criterion is satisfied.

In sections 5.1.3 and 5.1.4, this pipeline is applied respectively without and with contacts on simulated robots. As the simulation-to-reality transfer of the robots considered in our experiments has already been studied in previous works, the simulation could be considered as a ground-truth for assessing our algorithms. Constraints states could then be directly retrieved as results from the simulation. In section 5.1.5, this workflow is applied on a physical prototype. Constraint states are retrieved from a set of sensors.

## Calibration and Design Optimization applications

In section 5.1.6, it is shown that both $W_0$ and $\delta_0^{free}$ are good characterizations of the design of a soft robot. It is therefore possible to build a differentiable link between design parameters $P$ and these quantities as follows:

$$ \tilde{W}_0, \tilde{\delta}_0^{free} = G(P) \qquad (5.7) $$

where $G$ is a MLP network. Data are collected in the same way as for the network $F$ and the training loss is similar to 5.5.
Then applications such as calibration and design optimization are explored. When design parameters are relevant, the condensed FEM model is extended following:

$$ \tilde{W}, \tilde{\delta}^{free} = F(\delta_a, \delta_c, G(P)) \qquad (5.8) $$

This enables to build a differentiable link between design parameters and mechanical quantities in given active constraint states. Fitness functions are then built from these mechanical quantities. The entire learning pipeline with design parameters is illustrated in Figure 5.3.

<figure><img src="image_51.png" /><figcaption>Fig. 5.3: Illustration of the pipeline for learning mechanical states of Soft Robot. G predicts the mechanical state of the robot in its rest position from design parameters, and F predicts the mechanical state of the robot from both G outputs and a given actuation displacement.</figcaption></figure>

### 5.1.3 Soft robot control using a condensed FEM modelling without contacts

Following the introduction of the condensed FEM model, a set of experiments are introduced below for highlighting control applications without taking contacts into account. The open-loop control is illustrated and validated using two FEM models of different soft robots driven by cables, the Diamond robot (see section 3.1.2) and a Soft Finger robot (see section 3.1.2). The simulation mesh of the Diamond robot is made of 4147 tetrahedrons. The Soft Finger robot is made up of three segments connected by accordion-shaped joints and actuated by 2 cables. Its simulation mesh is made of 1557 tetrahedrons.

For each robot, the data are retrieved from the simulation as described in subsection 5.1.2. Statistics about the data, the final loss and the best epoch<fnref n="1" /> are

<footnote n="1">¹An epoch corresponds to the training of the network with all the training data for one cycle.</footnote>
given in Table 5.1. During data acquisition, the space of displacements, that is the space of relative lengths $δ_a$ imposed on the cables, is discretized and the actuation state is observed as the reached length of the cables at equilibrium.

<table><caption>**Tab. 5.1:** Data acquisition parameters for each robot for $n_l = 6500$ sample points. Cables displacement are sampled in [$δ_{a,min}$, $δ_{a,max}$] (mm). Final test loss values for Diamond and Soft Finger robots for 9000 learning iterations. The best trained network encountered is kept at best epoch.</caption><thead><tr><th rowspan="2">Robot</th><th colspan="2">Cables</th><th rowspan="2">Final Loss</th><th rowspan="2">Best Epoch</th></tr><tr><th>δ<sub>a,min</sub></th><th>δ<sub>a,max</sub></th></tr></thead><tbody><tr><td>Diamond</td><td>0</td><td>30</td><td>6.52e-5</td><td>8960</td></tr><tr><td>Finger</td><td>0</td><td>20</td><td>3.79e-6</td><td>8850</td></tr></tbody></table>

## Open Loop control of a single robot from a learned condensed FEM modelling

<figure><img src="image_52.png" /><figcaption>**Fig. 5.4:** Control results for the Diamond and Soft Finger robots without contact. A-B-C) Trajectory of the end effector of the Diamond robot in the (x, y) plane (mm) for 30 different goals (orange dots) when using learned mechanical matrices (blue cross) or computed mechanical matrices (red cross). 3 different position of the Diamond robot along this trajectory are shown as examples. The effector is represented by a green dot, and the goal by an orange dot. D-E-F) Trajectory of the end effector of the Soft Finger robot in the (y, z) plane (mm) for 25 different goals (orange dots) when using learned mechanical matrices (blue cross) or computed mechanical matrices (red cross). 3 different position of the Finger robot along this trajectory are shown as examples. The effector is represented by a green dot, and the goal by an orange dot.</figcaption></figure>

For evaluating the learning process, the average loss is computed on the test set of random robots configurations. Learning results for the condensed FEM model are given in Table 5.1. Although there is more constraints in the Diamond problem, it takes the same amount of data samples for training the MLP for the Diamond and
the Soft Finger. This is because the projected mechanical matrices relative to the Soft Finger robot varies much more than that relative to the Diamond robot. In fact, most non-linearities in mechanics come from rotations, and in the Finger example there are more rotations than in the Diamond example. For the Soft Finger, the influence of the actuators on the end effector changes a lot depending on the configuration of the robot, that is depending on how bent the robot is. On the contrary, the effector always has similar behavior in terms of constraints for the Diamond.

In order to evaluate the use of the learned condensed FEM model for inverse modelling, a trajectory is considered in the form of several goal positions to be reached by the effector for each robot. As the soft robot dynamics is neglected in this study, the results are only valid for trajectories time-scales in the quasi-static regime. The results are shown in Figure 5.4.

A circular trajectory of the effector in a horizontal plane is considered for the Diamond robot. This trajectory is chosen for being convenient for two dimensions visualization although the robot end effector can be controlled in all three directions of space. Then the positions obtained for each goal using the condensed FEM model learning based control pipeline and the classical inverse model are compared. In most configurations, the two approaches lead to similar positions of the end effector. This shows that the condensed model is a good representation of the system. On some positions, small-scale errors between the learned model and the full model can occur: this comes from the learning errors specific to the method. This error can be further decreased by increasing the discretization of the actuation space during data collection and by judicious choices of networks hyper-parameters. For the Soft Finger robot, a trajectory is chosen to force the robot to bend. The trajectory is not in the robot's workspace. In such case, the optimization will minimize the distance between the end effector (in the workspace) and the goal (in the trajectory). However, both control pipeline using the learned condensed FEM model and classical FEM model achieve similar results. The same conclusions as for the Diamond robot are valid for the origin of these errors and how they can be minimized.

## Investigation on mesh influence

Increasing the fineness of the mesh generally makes it possible to obtain more precise simulations at the cost of significant computation time. A compromise between speed and precision has generally to be found for enabling to control a soft robot in real time. This can be critical for robots with complex behavior for which both precision and speed are determining factors. As the condensed FEM model is built from the projection of mechanical matrices in constraint space, its size is
independent of the mesh size, which makes it a good candidate to satisfy both of these criteria at the cost of more computationally expensive offline simulations. The following experiment highlights the property of the condensed FEM model which speed of use in a control loop is independent in time of the size of the mesh.

Three condensed FEM models of the Soft Finger are trained for different mesh sizes. The three Soft Fingers mesh discretization as well as simulation performances are introduced in Figure 5.5.

<figure><img src="image_53.png" /><figcaption>Fig. 5.5: Soft Fingers with different mesh resolutions. For each of them, the mesh resolution in terms of number of tetrahedrons and number of frames per second (FPS) of the simulation is indicated. A) Basic Soft Finger: 1557 tetrahedrons, 32.6 FPS. B) Fine Soft Finger: 8895 tetrahedrons, 4.8 FPS. C) Super Fine Soft Finger: 17852 tetrahedrons, 1.8 FPS.</figcaption></figure>

For being able to compare them, 6500 Soft Finger states are sampled for building the training set and a fixed number of 9000 training iterations is performed for each Soft Finger. Learning results are summarized in Table 5.2. Magnitude order of final training losses are similar.

<table><caption>Tab. 5.2: Final test loss values and FPS of the simulation when using the condensed FEM model for the 3 Soft Finger robots with different meshes discretizations. The best trained network encountered is kept at each epoch.</caption><thead><tr><td></td><td>Final Loss</td><td>Best Epoch</td><td>FPS with Learned Model</td></tr></thead><tbody><tr><td>Soft Finger</td><td>3.79 × 10<sup>-6</sup></td><td>8850</td><td>308.6</td></tr><tr><td>Fine Soft Finger</td><td>4.26 × 10<sup>-6</sup></td><td>8900</td><td>334.4</td></tr><tr><td>Super Fine Soft Finger</td><td>3.60 × 10<sup>-6</sup></td><td>8990</td><td>296.7</td></tr></tbody></table>

Whatever the size of the mesh, calling the learned neural network for getting a given mechanical state takes the same amount of time. All three models perform equally well on the considered trajectory. The reduced model does not depend on the mesh size, but on the number of constraints in the problem. In this experiment,
only the simulation is considered. In most cases, the chosen hardware also limits
the running frequency. When the fineness of the mesh starts to become the blocking
factor, the condensed FEM model enables constant time predictions compatible with
real-time control.

5.1.4 Soft robot control using a condensed FEM modelling
with contacts

The integration of contacts into the method is illustrated using the Soft Finger
robot. It is assumed that contacts are located at three fixed points on the surface
of the Soft Finger. For the moment, whatever the example, the method only works
if the contact points can be identified in advance. The robot is in contact with a
perfect rigid object. To avoid having to solve Coulomb's and Signorini's laws, the
simplified assumption that the Soft Finger contacts points share the same location
than the associated contact points on the object is taken. This assumption facilitates
the optimization problem, but it does not change the representation of contacts in
the condensed FEM model.

<figure><img src="image_54.png" /><figcaption>Fig. 5.6: Illustration of the condensed FEM model for a Soft Finger with contacts and the two considered control scenarios. The Soft Finger is actuated by 2 cables and has 3 fixed contact points located at the tip. The scenarios are A) a Soft Finger robot pushing a button, and B) a Soft Gripper robot manipulating a cube. The cube effector is represented by a green 6D frame.</figcaption></figure>

Two application cases are considered and illustrated in Figure 5.6:

* A Soft Finger pushing a button shaped like a cube. The resistance of the button is modeled as a constant horizontal force. This example is used for validating the learning of the various mechanical quantities of the condensed FEM model related to contacts i.e. *W*<sub>cc</sub>, *W*<sub>ca</sub> and *δ*<sub>c</sub><sup>free</sup>.
* A Soft Gripper handling a cube. The gripper consists of 3 identical Soft Finger robots. This example shows that a single condensed FEM model learned for a Soft Finger robot can be used in another inverse optimization scheme coupling several Soft Fingers.

The introduction of contact constraints increases the dimension of the constraints space. In the Finger example, there are two actuators (2 constraints) and three contact points (9 constraints). Exploring this space is exponentially complex, in terms of constraints. Moreover, using a box sampling on both actuation and contact constraint spaces, as for both the Diamond and Soft Finger without contact from section 5.1.3, would result in many robot state that are irrelevant in our applications. That is why in both cases, the dataset is generated using an inverse control scheme and sampling in the cube effector displacement space instead. Statistics about the datasets and the learning are given in Table 5.3.

<table><caption>Tab. 5.3: Data acquisition parameters and final loss for Soft Finger and Soft Gripper robots in the contact case. The number of training epochs for the Soft Finger and the Soft Gripper are respectively 50000 and 42540.</caption><thead><tr><th rowspan="2">Robot</th><th colspan="3">Cube (mm)</th><th rowspan="2">Final loss</th><th rowspan="2">Best Epoch</th></tr><tr><th>δ<sup>cube,x</sup><sub>e</sub></th><th>δ<sup>cube,y</sup><sub>e</sub></th><th>δ<sup>cube,z</sup><sub>e</sub></th></tr></thead><tbody><tr><td>Soft Finger</td><td>[-2, 2]</td><td>[33, 37]</td><td>[-95, -94]</td><td>1.67e-6</td><td>42540</td></tr><tr><td>Soft Gripper</td><td>[-2, 2]</td><td>[33, 37]</td><td>[-95, -94]</td><td>1.24e-3</td><td>38440</td></tr></tbody></table>

## Inverse problem for interaction between object and soft finger

A simplified pipeline for taking into account contacts in the optimization process, without needing to solve the Signorini's and Coulomb's law is now developed.

Let $X_{cube} \in \mathbb{R}^6$ be the location of the cubic object in both translation and rotation. Let $\text{prev}$ be the notation used to refer to quantities calculated at the previous time step in an iterative process. We want to control for both the translation and orientation of the cube with the help of a set of Fingers. It is then equivalent to minimize the following quantity:

$$ \min_{\lambda_a} ||X_{cube} - X_{cube}^{goal}||^2 \qquad (5.9) $$

where $X_{cube}^{goal}$ is the goal location to reach and $||.||$ is the euclidean norm. To solve this problem, $X_{cube}$ has to be written as a function depending on $\lambda_a$. This expression is obtained from the kinematics relations written for the cubic object and the condensed mechanics applied to the Soft Fingers. Limitations on the actuation forces
are added afterward, when solving the complete inverse problem.

**Kinematics relations between points fixed on a rigid object:** From the hypothesis that contact points location on both the Soft Finger and the object are shared, it results that $\lambda_c = \lambda_{c,finger} = -\lambda_{c,cube}$ where $x_{c,finger}$ and $x_{c,cube}$ are the euclidean positions of the matching contact points on the interface between the Soft Finger and the cube.

Using the Varignon formula on the cube, the location $X_{cube}$ and $x_{c,cube}$ are linked as:

$$x_{c,cube} - x_{c,cube}^{prev} = J_c(X_{cube} - X_{cube}^{prev}) \quad (5.10)$$

where $J_c \in \mathbb{R}^{N \times 6}$ is the Jacobian matrix linking the kinematics of the manipulated object to the contact points and $N$ the number of contact point. For one point, the Jacobian matrix is given by:

$$J_c = \begin{bmatrix} 1 & 0 & 0 & 0 & -C_z & C_y \\ 0 & 1 & 0 & C_z & 0 & -C_x \\ 0 & 0 & 1 & -C_y & C_x & 0 \end{bmatrix} \quad (5.11)$$

where $C_i = X_{cube,i} - x_{c,cube,i}$ : $i \in [x, y, z]$ are the coordinates of the vector $\vec{C}$ as illustrated in Figure 5.7. Generalization for several contact points is straightforward.

<figure><img src="image_55.png" /><figcaption>Fig. 5.7: Illustration of the quantities for computing the Jacobian $J_c$ for one contact point.</figcaption></figure>

From this Jacobian matrix the resulting contact force on the cube is then expressed as $T_{cube} = J_c^T \lambda_c$.
**Condensed mechanics applied on a Soft Manipulator:** By isolating the robot and writing the condensed mechanics according to Eq. 2.14, the values of $x_c$ are expressed from $\lambda_a$, $\lambda_c$ and the free displacement $\delta_c^{free}$ of the contact point as:

$$x_{c,cube} = W_{ca}\lambda_a + W_{cc}\lambda_c + \delta_c^{free} \qquad (5.12)$$

By integrating these results into Eq. 5.10 and multiplying by $D = J_c^T W_{cc}^{-1}$, we obtain:

$$ \begin{aligned} D x_{c,cube}^{prev} + D J_c (X_{cube} - X_{cube}^{prev}) &= D \delta_c^{free} + D W_{cc} \lambda_c + D W_{ca} \lambda_a \\ \Leftrightarrow D J_c X_{cube} &= D W_{ca} \lambda_a + [D \delta_c^{free} + D W_{cc} \lambda_c + D J_c X_{cube}^{prev} - D x_{c,cube}^{prev}] \\ \Leftrightarrow X_{cube} &= \underbrace{[D J_c]^{-1} D W_{ca}}_{\mathbb{A}} \lambda_a + \underbrace{[D J_c]^{-1} [D \delta_c^{free} + D W_{cc} \lambda_c + D J_c X_{cube}^{prev} - D x_{c,cube}^{prev}]}_{\mathbb{B}} \end{aligned} \qquad (5.13) $$

which is of the form $X_{cube} = \mathbb{A}\lambda_a + \mathbb{B}$ if we suppose that $D W_{cc} \lambda_c = T_{cube}$ the contact forces are known. The optimization objective from Eq. 5.9 is now well posed.

Both the Soft Finger robot and the Soft Gripper robot differ in the way the force $T_{cube}$ is computed. From the new values of $X_{cube}$ and $\lambda_a$, it is then possible to compute a new value of $\lambda_c$ using Eq. 5.13.

## Application: Soft Finger pushing a press-button

The learned condensed FEM model of the Soft Finger robot with contact is evaluated on the action of pushing a press-button. The last one is modeled as a cube on which a constant known horizontal force $T_{cube}$ opposed to the orientation of the bending movement is applied. The Soft Finger has to push this button following an horizontal trajectory. This trajectory is a set of positions of the center of the cube $X_{cube}$ sampled along the aforementioned trajectory. As the finger bending is only possible in one direction, only a single degree of freedom of the cube is controlled. The results are shown in Figure 5.8.

On this application, the Soft Robot open-loop control using the condensed FEM model shows similar degree of precision with the simulation-based control.
<figure><img src="image_56.png" /><figcaption>**Fig. 5.8:** Control results for the Soft Finger robot pushing a button. The reached horizontal position of the cube (mm) are compared on a trajectory of 10 successive goals (orange dots) when using learned mechanical matrices (blue cross) or mechanical matrices computed in simulation (red cross). 3 different positions of the Soft Finger robot met during this trajectory are shown as examples. On this representations, the effector is a green frame, and the goal a red frame.</figcaption></figure>

## Application: Soft Gripper Manipulating a Cube.

The mechanical matrices of the Soft Gripper robot are built from the mechanical matrices computed on three single Soft Finger. This shows that the condensed FEM model learned for a single robot can be used for predicting matrices of several similar robots in parallel. Compared to the scenario of the Soft Finger pushing a button, applied forces on the cube are not known this time.

After solving the same problem as for a single Soft Finger, Eq. 5.13 enables to express $\lambda_c$ with the new values of $\lambda_a$ and $X_{cube}$. The corresponding algorithm is written as follows:
**Algorithm 1** Control algorithm for object manipulation scenario.

Start from $\lambda_c = 0$.

for each time step :

| 1. Compute $\lambda_a$ and $X_{cube}$ according to $X_{cube} = A\lambda_a + B$ and the previous evaluation of $\lambda_c$. The value of $\lambda_c$ is used to calculate $B$.

| 2. Evaluate the new value of $\lambda_c$.

In this algorithm, $\lambda_a$ and $\lambda_c$ are updated once per time step. This is a simplifying assumption, which could lead to errors. However, it is not a major issue in the current scenario.

For the experiment, a circular trajectory of the object center is considered. The results are shown in Figure 5.9.

<figure><img src="image_57.png" /><figcaption>**Fig. 5.9:** Control results for the Soft Gripper robot. The position of the manipulated cube (mm) are compared on a trajectory for 20 different goals (orange dots) when using learned mechanical matrices (blue cross) or mechanical matrices computed from simulation (red cross).</figcaption></figure>

On the considered trajectory, slight differences between goals and reached effector positions are observed both with the simulation-based and learning-based controllers. In addition, a small effector-goal gap of about 0.04 mm is observed along the vertical z-axes in both cases. This is explained by the modelling of the manipulator which considers Soft Fingers contact points to always be at the same location on the manipulated object. These errors could be reduced using a closed-loop controller.

Moreover, small differences between reached positions of both the simulation-based and learning-based controllers are observed. Compared to other cases from
this paper, training results are less good, as can be seen in table 5.3. This is explained
by the complexity of the dataset and may be improved through data filtering and
hyper-parameters tuning.

5.1.5 Embedded control of soft robot using a condensed
FEM modelling

The considered soft robot is a pneumatic trunk robot (see section 3.1.2) presented in [Cia+14] and called the Stiff-Flop robot. This robot is a highly flexible and pneumatically driven soft robot composed of two modules including three fully fiber-reinforced chamber pairs each. This robot was developed for medical applications, for example as an in-vivo cancer diagnostic tools during minimally invasive interventions [Cha+23]. The physical robot and the associated simulation are shown in Figure 5.10.

<figure><img src="image_58.png" /><figcaption>Fig. 5.10: Illustration of the robot used to illustrate embedded control. A) Stiff-Flop robot simulated in SOFA. The effector of the robot is represented by a green dot, and the goal by an orange dot. B) Physical prototype of the Stiff-Flop robot.</figcaption></figure>

Concerning the simulation, the Stiff-Flop is built using a reduced finite element
method model. The pneumatic forces are distributed all along the FEM nodes of
the surface of the cavities. The distributed forces are projected on a reduced beam
mathematical space using SOFA mapping, where the equilibrium of forces is solved.
This reduced model compute faster than traditionnal FEM simulation, and enables a
real time control of the robot.

For learning the condensed FEM model, data are collected in the space of the
volume change of the cavities. Both the statistic about the sampling data and the
condensed FEM learning results are show in Table 5.4.
<table><caption>**Tab. 5.4:** Data acquisition parameters for the Stiff-Flop robot for $n_l = 50000$ sample points. Actuator displacements are sampled in [$\delta_{a,min}$, $\delta_{a,max}$] (mm³). Final test loss values are displayed for 50000 learning iterations. The best trained network encountered is kept at best epoch.</caption><thead><tr><th rowspan="2">Robot</th><th colspan="2">Sampling bounds</th><th rowspan="2">Final Loss</th><th rowspan="2">Best Epoch</th></tr><tr><th>δ<sub>a,min</sub></th><th>δ<sub>a,max</sub></th></tr></thead><tbody><tr><td>Stiff-Flop</td><td>0</td><td>30</td><td>9.22e-4</td><td>37070</td></tr></tbody></table>

**Embedded open and closed-loop control using a condensed FEM model.**

In this section, the framework for embedded open and closed-loop control of a soft robot based on the condensed FEM model is developed. The system is pressure-controlled and no additional actuator measurements are added. The condensed FEM model is used for computing the value of the volume increments based on the actuation $\lambda_a$ and the projected mechanical matrices from Eq.2.14. This strategy results in a time step shift on the predicted mechanical matrices used for computing the actuation displacement. This shift may be corrected using a suitable corrector in a closed loop setting.

The general embedded control loop framework is as follows:

**Algorithm 2 Embedded control loop.**

**Inputs:** $W_0$ and $\delta_0^{free}$ from the training dataset.
Set initial actuation $\lambda_a = 0$, and initial distance $\delta_a = \delta_a^{free}$.

**for** each time step:

1. Recover the true position of the effectors from sensors.

2. If considering to close the loop, compute the corrected goal position.

3. Predict the mechanical quantities $\tilde{W}$ and $\tilde{\delta}^{free}$ using the trained neural network.

4. Solve the optimization problem to recover the force $\lambda_a$.

5. Compute the new actuators state from mechanical matrices and forces as $\delta_a = W_{aa}\lambda_a + \delta_a^{free}$.

6. Apply the forces $\lambda_a$ on the robot.

Closing the loop can be written using the $\delta_a^{free}$ value used for optimization. In our experiments, a PID controller is set from Eq. 5.6 as:

$$p_{goal,t}^{corrected} = p_{goal,t} + K \times \epsilon_t + I \times \sum_{t' \le t} \epsilon_{t'} + D \times (\epsilon_t - \epsilon_{t-1}) \quad (5.14)$$

where $p_{goal,t}^{corrected}$ is the corrected goal at time $t$, $p_{goal,t}$ is the goal in the trajectory at time $t$, $\epsilon_t = p_{goal,t} - p_{effectors,t}$ is the position difference between the goal and the
actual position of the effectors, *K*, *I*, *D* are the weight for the proportional, integral and derivative error. A unitary step is considered both for integral and derivative errors.

The full algorithm can be directly embedded on a microprocessor such as a Raspberry.

<figure><img src="image_59.png" /><figcaption>Fig. 5.11: Control results for the Stiff-Flop robot. The 3D-position (mm) of the end effector is compared on a trajectory for 40 different goals (orange dots) when using learned mechanical matrices (red cross). 3 different position of the Stiff-Flop robot along this trajectory are shown as examples. The effector is represented by a green dot, and the goal by an orange dot.</figcaption></figure>

**Experimental results with simulation in digital twin conditions.**

The goal is to control the robot's physical prototype. To validate the learning, the robot simulation is used in digital twin conditions, to simulate the problem as if it was the physical prototype. The tests could not be carried out on the real prototype because of time constraints. The control is tested on the simulated robot and the
results are shown in Figure 5.1.5. Both open and closed loop scenarios follow a spiral trajectory. There is a small z-offset between the goals and the actual positions of the end-effectors, but this offset is consistent with the different results found with other models. This error can be further reduced by fine-tuning the PID corrector. This validates the use of this model for robot control, under the same conditions as the physical prototype.

### 5.1.6 Condensed FEM modelling extended to calibration and design

In this section, we demonstrate how the condensed FEM model previously used to control soft robots is extended to handle both calibration and design optimization applications. Both the *W* and $\delta^{free}$ mechanical matrices directly capture the link between the robot's actuators, contact points and effectors. This link depends directly on the robot's body and its mechanical properties. Moreover, the function *F* linking both the *W* and $\delta^{free}$ mechanical matrices with the actuation state $\delta_a$ of the robot and the initial compliance matrix projected in the constraint space $W_0$ is fully differentiable. Differentiable fitness functions for assessing a design performance can then be designed from *W* and $\delta^{free}$. As we will see, a model trained for several designs can be used to optimise that design without requiring FEM simulation. This is a critical point in design optimization loop when many designs have to be evaluated.

Many works in the community consider training a surrogate to directly learn the values of the fitness function relatively to the design variables. For instance, in [YHM23], a surrogate is learned for predicting angular deflections of pneumatic modules from both material properties and geometrical dimensions design parameters. Changing the fitness function then requires them to regenerate data and re-train a model. In the case of the condensed FEM model, since fitness functions are directly formulated from the learned matrices, a single condensed model can be used to optimize a robot across multiple tasks.

For this section, we use parametric design of a Soft Finger. This parametric design is lightly different than the one used in the previous sections and is an iteration of the one previously proposed in [Nav+23]. This Soft Finger is entirely made in silicone, contains two pneumatic cavities located at the joints and is actuated by one cable going from its base to the tip. This work showed how the Soft Finger geometry can be optimized both for its kinematics and the sensitivity of the pneumatic cavities which are used as a sensor, and also demonstrates the transfer to reality of the optimized results. Compared to this previous work, no pneumatic sensors and different design parameters are considered. The considered parametric Soft Finger
is shown in Figure 5.12. Pulling on the cable has the effect of bending the finger and rigidify it. An end-effector is located on the fingertip to control its position.

<figure><img src="image_60.png" /><figcaption>Fig. 5.12: Soft Finger parametric design. Design parameters are A) Length, B) Height and C) Joint Height.</figcaption></figure>

## Investigation on parametric design captured by a single condensed FEM model

In the following experiments, we show that the condensed FEM model is expressive enough to capture design variations. Two condensed FEM model are learned from two different dataset generated from the parametric Soft Finger. Both sampling strategies and learning results are introduced below:

* The **mechanical parameters dataset** considers variation of the Young's Modulus and Poisson Ratio mechanical parameters. Both displacement states and numerical simulation parameters are sampled using a Scrambled Halton sequence. A total of 10000 samples were simulated. Sampling parameters are described in Table 5.5.

<table><caption>Tab. 5.5: Bounds of sampling parameters for the condensed FEM model of the Soft Finger with varying mechanical parameters.</caption><thead><tr><td>Parameter</td><td>Minimum Value</td><td>Maximum Value</td></tr></thead><tbody><tr><td>Young Modulus (kPa)</td><td>1000</td><td>10000</td></tr><tr><td>Poisson Ratio</td><td>0.400</td><td>0.499</td></tr></tbody></table>
* The geometrical dimensions dataset considers geometrical dimensions variations around a baseline design of the Soft Finger. The three geometrical design parameters are described in Figure 5.12. A total of 50000 samples were simulated. Sampling parameters are described in Table 5.6.

<table><caption>**Tab. 5.6:** Bounds of sampling parameters for the condensed FEM model of the Soft Finger with varying geometrical dimensions.</caption><thead><tr><td>Parameter</td><td>Minimum Value</td><td>Maximum Value</td></tr></thead><tbody><tr><td>Length (mm)</td><td>38</td><td>42</td></tr><tr><td>Height (mm)</td><td>20</td><td>22</td></tr><tr><td>Joint Height (mm)</td><td>5.0</td><td>8.0</td></tr></tbody></table>

Training results are displayed in Table 5.7 for both datasets. For learning the condensed FEM model with geometrical variations of the Soft Finger, a more complex strategy is adopted. A neural network of 5 fully connected layers with 800 nodes each is chosen. As the design space is exponential in the number of parameters and actuations, and under-sampled in the training set, a dropout strategy is applied for regularizing the network. Practically, each hidden layer node has a probability of 0.4 of being disregarded at each epoch of the training. The orders of magnitude of the final loss for the geometrical dimensions dataset displayed in the Table is then much higher than for the usual losses, but the generalization is sufficient enough regarding targeted applications. Taking a closer look into element-wise prediction errors, mechanical matrices values related to the link between actuation and effectors in the direction where the geometrical dimensions (Length and Height) of the parametric Soft Finger evolve are harder to learn. However this standardized loss error values corresponds to a small difference between predicted and simulated effector positions of around 3mm in the worst case met on the entire test dataset. These results can be improved by increasing the size of the dataset or by learning conjointly the two networks *F* and *G*.

<table><caption>**Tab. 5.7:** Test loss values for each parametric Soft Finger dataset for both *G* and *F* neural networks.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="2">G</th><th colspan="2">F</th></tr><tr><th>Final Loss</th><th>Best Epoch</th><th>Final Loss</th><th>Best Epoch</th></tr></thead><tbody><tr><td>Mechanical Parameters</td><td>3.82e-7</td><td>18420</td><td>5.04e-5</td><td>12530</td></tr><tr><td>Geometrical Dimensions</td><td>0.19</td><td>4400</td><td>0.13</td><td>270</td></tr></tbody></table>

For showing that the condensed FEM model provides a good characterization of the design, several designs are evaluated on a trajectory using the condensed FEM model based open loop control. The considered trajectory is the one showed in Figure 5.4 (closing the finger). Results are displayed as errors between positions reached by the ground truth simulation and the corresponding location obtained with the condensed FEM model. They are displayed in Table 5.8. Whatever the
sampled design, it shows that a trained condensed FEM model on the proper dataset enables to control different designs from this dataset. On studied examples, maximal positioning error is about 1.2 mm which is approximately less than 1% of the total length of the parametric Soft Finger.

<table><caption>**Tab. 5.8:** Positioning error for different values of geometrical and mechanical parameters. Error is defined as $\frac{|d_{simu}-d_{learned}|}{d_{simu}}$, where $d_i$ corresponds to the mean euclidean distance between the effector and the goal evaluated over the trajectory, and obtained whether from simulation or condensed FEM model.</caption><caption>Trajectory errors with mechanical parameters dataset:</caption><thead><tr><th>Young Modulus (GPa)</th><th>Poisson Ratio</th><th>Error</th></tr></thead><tbody><tr><td>5000</td><td>0.47</td><td>0.91%</td></tr><tr><td>1000</td><td>0.45</td><td>0.29%</td></tr><tr><td>7000</td><td>0.4</td><td>0.90%</td></tr><tr><td>3000</td><td>0.47</td><td>1.03%</td></tr></tbody></table>

<table><caption>Trajectory errors with geometrical dimensions dataset:</caption><thead><tr><th>Length (mm)</th><th>Height (mm)</th><th>Joint Height (mm)</th><th>Error</th></tr></thead><tbody><tr><td>40</td><td>20</td><td>6</td><td>1.03%</td></tr><tr><td>38.5</td><td>21.5</td><td>6</td><td>3.11%</td></tr><tr><td>40</td><td>20.5</td><td>7.5</td><td>7.02%</td></tr><tr><td>41</td><td>21</td><td>5.5</td><td>1.62%</td></tr><tr><td>39.5</td><td>20</td><td>7</td><td>5.10%</td></tr></tbody></table>

## Application to calibration of the condensed FEM model of a parametric Soft Finger

Numerical parameters, such as the Young's Modulus and the Poisson Ratio, influence the behavior of the robot in simulation. Since the condensed FEM model is learned from simulations, one of the drawbacks is that it is learned for a fixed set of numerical parameters. Even if the simulation is calibrated beforehand on a physical prototype, defaults may appear during the manufacturing of other similar physical prototypes or due to silicone degradation overtime, and it would then be necessary to re-calibrate the simulation and learn a new condensed FEM model for controlling them. In the example of the Soft Finger, the appearance of air bubbles or an incorrect dosage of silicone during the molding process can lead to different corresponding mechanical parameters. For dealing with this issue, the extended condensed FEM model is learned with variations of the numerical parameters of the soft robot design. The aim is to address learning the condensed FEM model with zero-shot transfer to reality.

Once the condensed FEM model is learned, it is calibrated using data collected from the physical robot. The general workflow to calibrate a condensed FEM model
of any soft robot is as follows. First, positions of the effector(s) are measured for several actuation displacement states $\delta_a$. Increasing the number of actuation states enables better convergence. Then, a fitness function $O_{cal}(P)$ depending of the design parameters $P$ and computing the difference between the relative position of the effectors $\delta_e^*$ on the physical prototype and those estimated by the condensed FEM model $\tilde{\delta}_e$ is built as follows:

$$O_{cal}(P) = \sum_{\delta_a} |\tilde{\delta}_e(\delta_a, P) - \delta_e^*(\delta_a)| \quad (5.15)$$

In the previous equation, dependence of the different quantities to both $P$ and $\delta_a$ are explicitly shown for clarity.

In order to build a differentiable link with the design parameters $P$, $\delta_e$ is expressed from the predicted mechanical matrices. Using Eq. 2.14 without contacts, it gives:

$$\tilde{\delta}_e(\delta_a, P) = \tilde{W}_{ea}(\delta_a, P) \tilde{W}_{aa}^{-1}(\delta_a, P) (\delta_a - \tilde{\delta}_a^{free}) + \tilde{\delta}_e^{free} \quad (5.16)$$

The design parameters $P$ are optimized through gradient descent through minimizing $O_{cal}$.

Back to the Soft Finger example, the position of the effector on the tip is easily observable with a camera, which makes it a good candidate as a calibration data. Moreover, only the Poisson Ratio is optimized as the Soft Finger is controlled in displacement, the Young's Modulus therefore being not a relevant design parameter for the calibration. The experimental validation of the calibration of the learned condensed FEM model of the Soft Finger is explained below. As its simulation to reality transfer has already been analysed in previous work [Nav+23], the states $\delta_e^*$ of the robot are sampled in simulation for cable displacement values $\delta_a$ of 0mm, 5mm and 10mm. For the moment the method has not yet been tested with a physical robot. Young's Modulus and Poisson Ratio ground truth parameters are chosen equal respectively to 3000 GPa and 0.47. Then, the fitness function from Eq. 5.15 is minimized as previously introduced for retrieving the corresponding parameters. Obtained convergence history are displayed in Figure 5.13 for three different initializations of the mechanical parameters.

In each case, the algorithm succeeds in finding the good Poisson Ratio for reducing the distance between the positions of the effector given by the learned model and those of the physical robot. As similar final fitness functions are obtained, it shows that the algorithm converges regardless of the mechanical parameters
<figure><img src="image_61.png" /><figcaption>Fig. 5.13: Optimization history for the calibration of the Soft Finger robot. Results have been obtained with an initial learning rate of 0.01 used conjointly with an adaptive scheme as described in section 5.1.2. Both fitness function (A) and mechanical parameters histories (B,C,D) across optimization iterations are displayed for different initial guesses of the mechanical parameters. Initial mechanical parameters are a fixed Young Modulus of 3000 GPa as well as B) Poisson Ratio = 0.47, C) Poisson Ratio = 0.49 and D) Poisson Ratio = 0.45.</figcaption></figure>

initialization on this scenario. Here the Young's modulus has no impact on the fitness function from Eq. 5.15 because effector displacements are evaluated for different cable displacements. Changing for a servoing in force exerted on the cables instead of cable displacements would require to take into account the Young's Modulus during the optimization. With this experiment, it is demonstrated that it is possible to learn a single condensed FEM model that can be directly calibrated to a soft robot. By relying on the differentiability of the learned condensed FEM model, the calibration method only takes a few seconds on a pre-learned model, compared to other models relying on non-gradient solvers costing a lot of simulations.

## Application to design optimization of a parametric Soft Finger

In this subsection, we demonstrate how the condensed FEM model could be used for optimizing the design of a soft robot on the case of the parametric Soft Finger. Alongside the intended design optimization application, including design parameters such as geometrical dimensions makes possible to calibrate the condensed FEM
model to design variations which may appear during the manufacturing process,
using similar technique as the one shown in the previous section.

For this experiment, the parametric Soft Finger design introduced in Fig-
ure 5.12 is optimized successively for its dexterity and for precision grasping. Design
optimization objectives are formulated from the predicted mechanical matrices. The
two considered objectives are both illustrated in Figure 5.14.

<figure><img src="image_62.png" /><figcaption>Fig. 5.14: Illustration of design objectives for the parametric Soft Finger. A) Soft Finger in the initial state. B) Bending angle $\alpha(\delta_a, p)$ reached for a fixed cable displacement $\delta_a$. C) Contact force $\beta(\delta_a, p)$ generated for a fixed cable displacement $\delta_a$.</figcaption></figure>

The first fitness function $O_{dext}$ characterizes the kinematics of the parametric
Soft Finger, that is the reached bending angle for a fixed cable displacement $\delta_a$ of
10mm. Minimizing this function is equivalent to maximizing the workspace of the robot. $O_{dext}$ is expressed as follows:

$$ 
\begin{aligned}
O_{dext}(P) &= |\alpha^{max} - \alpha(\delta_a, P)| \\
\alpha(\delta_a, P) &= Arccos(\frac{\tilde{\delta}_e^z(\delta_a, P)}{|\delta_e(\mathbf{0}, P)|})
\end{aligned}
\qquad (5.17) $$

where $\alpha$ is the function giving the bending angle of the parametric Soft Finger in the z-plane and $\alpha^{max} = 1.57$ rad is the maximum reached angle.

The second fitness function $O_{str}$ measures the contact force generated at the tip of the Soft Finger for a fixed cable displacement $\delta_a$ of 10mm. Minimizing this function amounts to maximizing using the parametric Soft Finger for precision grasping. Again, for building a differentiable link with the design parameters $P$, the contact force $\lambda_c$ is expressed from the predicted mechanical matrices. If we assume that the object is not moving ($\delta_c = 0$) and that the tip of the end-effector is a contact point, we can use the expressions for $\delta_a$ and $\delta_c$ in Eq 2.14 to write:

$$ \lambda_c(P, \delta_a) = B(P, \delta_a)^{-1} [\tilde{W}_{ca} \tilde{W}_{aa}^{-1} (\delta_a - \tilde{\delta}_a^{free}) + \tilde{\delta}_c^{free}] \qquad (5.18) $$

with $B(P, \delta_a) = \tilde{W}_{ca} \tilde{W}_{aa}^{-1} \tilde{W}_{ac} - \tilde{W}_{cc}$. Finally, the fitness function $O_{str}$ is expressed as follows:

$$ 
\begin{aligned}
O_{str}(P) &= |\beta^{max} - \beta(\delta_a, P)| \\
\beta(\delta_a, P) &= |\lambda_c(P, \delta_a)^y|
\end{aligned}
\qquad (5.19) $$

where $\beta$ is the function giving the force generated at the tip of the parametric Soft Finger on the y-axis and $\beta^{max} = 10000$ N is an unreachable force.

### Single objective optimization:

As the considered fitness functions are highly non convex, the simple gradient descent optimizer used may fall in a local optima. For avoiding this behaviour, a grid search is first performed to determine a good starting point for the optimization. As the design space is of only three dimensions, the computation is a simple matter of seconds. A total of 600 designs are evaluated for both fitness functions. On high dimensional cases, an alternative for avoiding the dependency to the initialization would have been to use stochastic gradient descent instead of a simple gradient descent scheme as a solver.

For each fitness function, the objective function landscape is computed. For this purpose, a regular grid sampling strategy is applied around the chosen initialization
starter design. Landscapes computed for both fitness functions are displayed in Figure 5.15. They show that the length has a higher impact on both objectives than two other design variables.

<figure><img src="image_63.png" /><figcaption>Fig. 5.15: Fitness landscapes around the initialized design selected with grid search strategy, for A) dexterity fitness function (Length = 38.0mm, Height = 20.57mm, Joint Height = 6.28mm) and B) strength fitness function (Length = 42mm, Height = 22mm, Joint Height = 8mm). The best design after optimization is shown as a blue dot.</figcaption></figure>

<figure><img src="image_64.png" /><figcaption>Fig. 5.16: Optimization histories and results for single objectives case on the parametric Soft Finger. Fitness functions histories respectively for A) kinematics and C) strength. Design parameters histories respectively for B) kinematics and D) strength. Resulting designs respectively for E) kinematics and F) strength.</figcaption></figure>

Optimization history and final results are shown in Figure 5.16. Regarding the dexterity, the best design privileged a shorter length whereas longer length helps to generate more forces on the object thanks to the lever arm effect. The best design regarding contact forces generated at the tip is also a Soft Finger with more material, enabling to rigidify better around the contact location.
Multi-objective optimization:

In this experiment, it is highlighted how to determine the optimal design of a Soft Finger regarding several metrics computed using the condensed FEM model. Figure 5.17 shows the fitness functions values for 600 Soft Finger designs sampled with a grid search strategy. For having comparative values between the two losses, they are both normalized using $O_i^{norm}(P) = \frac{O_i(P)-min_{P}*(O_i(P*))}{max_{P}*(O_i(P*))-min_{P}*(O_i(P*))}$ where $P*$ are design parameters encountered during initial grid search.

A Pareto front is clearly visible, showing that the two considered metrics are antagonistic. A compromise has therefore to be found. This can be done using the gradients calculated via the condensed model, through formulating a single fitness function as an aggregation of the considered fitness functions:

$$O_{mult}(P) = \gamma_1 O_{dext}^{norm}(P) + \gamma_2 O_{str}^{norm}(P) \quad (5.20)$$

where both $\gamma_1$ and $\gamma_2$ are user-chosen weighting factors between 0 and 1 for selecting the priority to give to each fitness functions. This kind of approach enables to generate the optimal design of a Soft Robot given user-chosen requirements.

<figure><img src="image_65.png" /><figcaption>Fig. 5.17: Pareto Front for the design optimization of the Soft Finger regarding Dexterity and Strength metrics and generated using a Grid Search strategy. Design parameters for several design sampled: A) Length = 38.0mm, Height = 21.6mm, Joint Height = 5.0mm, B) Length = 40.4mm, Height = 20.0mm, Joint Height = 5.6mm, C) Length = 40.2mm, Height = 22.0mm, Joint Height = 7.4mm, D) Length = 42.0mm, Height = 22.0mm, Joint Height = 8.0mm</figcaption></figure>
5.1.7 Discussion

In this section, we have demonstrated the potential of learning a condensed
FEM model of the robot. First, we have shown that it can be use for the control of
the robot itself, in particular for direct or inverse kinematics. Then we have extended
the formulation to contact. Contacts are considered as constraints in the same way
as actuators and effectors. We show how to formulate inverse control problems
both for a single and coupled soft robots with contact modelling. In the current
framework, both the number and the location of the contact points on the robot are
fixed. This is an important limitation as it does not allow for unanticipated obstacles.
This limitation is due to the data input and output size dependency of the MLP used
for learning the condensed FEM model. As a workaround, size-independent neural
networks such as Transformers [Vas+17] may be integrated in the framework.

Secondly, tools for embedding control have been explored. We proposed an
open-loop and closed-loop control based on the learned condensed FEM model of
a digital twin conditions simulation. Future work has to be carried out to control
a physical prototype directly. Integration into a low-capacity controllers is possible
thanks to the high motor command generation frequency (1e3Hz for the considered
robot) and the low memory consumption of the condensed FEM model. The embed-
ded control could be applied on other robots. The more complex is the robot, the
more interesting the time gain is. So far, only kinematics has been considered, and
the condensed FEM model extension to dynamics may unlock applications in using
more robust controllers such as Model Predictive Control.

Finally, we show that a single condensed FEM model is able to simultaneously
capture design and actuation states variations. Thanks to the high prediction speed
and the inherent differentiability of the model, applications in direct calibration of
the condensed FEM model and design optimization of soft robots are demonstrated
on the example of a parametric Soft Finger. This could open a new path to design
and control co-optimization applications based on a design-dependant surrogate.

Although we consider entirely soft robots, the current condensed formalism
enables to consider hybrid robots with both rigid and soft parts.
## 5.2 Condensed dynamics for optimal control in soft robotics

Inspired by the one-step control in soft robotics and the trajectory optimization tools developed in rigid robotics (see Section 2.2.3), we propose a formulation of the trajectory optimization problem in soft robotics based on the elaboration of a condensed FEM dynamics. The general idea of the method is to use the complete FEM dynamics of the soft robot to simulate its behavior. Then a backward step uses the linearized condensed FEM dynamics to find the best actuation to solve the given task.

### 5.2.1 Condensed dynamics for trajectory optimisation

#### Condensed dynamics of soft robots

In the multi-step control problem, we have to write the dynamic of the system. In Chapter 2, we presented the choice of solving the general equation of dynamics using a semi-implicit Euler scheme. At each time step, the system presented in (2.4) and reproduced here has to be solved:

$$Ad\dot{q} = hf_{int}(q_{k-1}, \dot{q}_{k-1}) + h^2K\dot{q}_{k-1} + hf_{ext} \quad (5.21)$$

This scheme is semi-implicit because the internal forces are linearised around the current position, with only one linearisation per time step. Considering this equation and making the value of actuation explicit, we can write the following system:

$$d\dot{q} = A^{-1}b(q_{k-1}, \dot{q}_{k-1}) + hA^{-1}H_a^Tu_{k-1} \quad (5.22)$$

where $b(q_{i-1}, \dot{q}_{i-1}) = hf(q_{i-1}, \dot{q}_{i-1}) + h^2K\dot{q}_{i-1} + hf_{ext}$. Using the relation $p = H_e q$ between the position of the effectors $p$ and the position of all the point of the mesh $q$, we can project the dynamics in the effector space according to:

$$d\dot{p} = H_e A^{-1}b(q_{k-1}, \dot{q}_{k-1}) + hH_e A^{-1}H_a^Tu_{k-1} \quad (5.23)$$
where we recognize the condensed FEM matrices W in the right member of the equation. Using the implicit integration scheme, we can write:

$$ 
\begin{aligned}
\dot{p}_k &= \dot{p}_{k-1} + H_e A^{-1} b(q_{k-1}, \dot{q}_{k-1}) + h W_{ea} u_{k-1} \\
p_k &= p_{k-1} + \dot{p}_k h = p_{k-1} + h \dot{p}_{k-1} + h H_e A^{-1} b(q_{k-1}, \dot{q}_{k-1}) + h^2 W_{ea} u_{k-1}
\end{aligned}
\qquad (5.24) $$

The condensed dynamics of the soft robot can be written in matrix notation:

$$ 
\begin{aligned}
X_{k+1} &= \begin{bmatrix} h^2 W_{ea,k} \\ h W_{ea,k} \end{bmatrix} u_k + \begin{bmatrix} I & hI \\ 0 & I \end{bmatrix} X_k + \begin{bmatrix} hI \\ I \end{bmatrix} H_e A^{-1} b(q_k, \dot{q}_k) \\
\Leftrightarrow X_{k+1} &= f_k(X_k, u_k)
\end{aligned}
\qquad (5.25) $$

where $X_k = [p_k, \dot{p}_k]^T$

## Derivative of the condensed dynamic

As we explain in section 2.2.3, DDP uses the derivative of the dynamics to compute the Hamiltonian value. By derivating Eq. 5.25 according to $u_k$ and $X_k$ we have for the control part:

$$ \frac{\partial f_k}{\partial u} = \begin{bmatrix} h^2 W_{ea,k} \\ h W_{ea,k} \end{bmatrix} \qquad (5.26) $$

and for the state using the chain rules and the fact that $H_e$ is its own pseudo-inverse:

$$ \frac{\partial f_k}{\partial X} = \underbrace{\begin{bmatrix} I & hI \\ 0 & I \end{bmatrix}}_{\frac{\partial f_k}{\partial X \text{ linear}}} + \underbrace{\begin{bmatrix} H_e A^{-1} & 0 \\ 0 & H_e A^{-1} \end{bmatrix}}_{\frac{\partial f_k}{\partial X \text{ non-linear}}} \begin{bmatrix} h^2 K & h^3 K + h^2 D \\ hK & h^2 K + hD \end{bmatrix} \begin{bmatrix} H_e^T & 0 \\ 0 & H_e^T \end{bmatrix} \qquad (5.27) $$

This quantity is composed of two terms, one corresponding to the derivative of the linear part of the dynamics, and the other to the derivative of the non-linear part.
## Constraint on the actuation

We can define a mathematical constraint $h_k(X, u) \le 0$ that corresponds to the limits of the actuation values. This constraint can be expressed as:

$$h_k(X, u) = h_k(u) = \begin{bmatrix} -I \\ I \end{bmatrix} u + \begin{bmatrix} u_{min} \\ -u_{max} \end{bmatrix} \le 0 \quad (5.28)$$

<figure><img src="image_66.png" /><figcaption>Fig. 5.18: Illustration of the trajectory optimization pipeline. Starting from an initial position, the forward phase computes the different positions and velocities of the complete mesh using the full dynamics (in orange). From the simulation data, a nominal trajectory is extracted and the mechanical quantities necessary for the DDP are saved (in blue). The nominal trajectory (green) is used with the condensed trajectory (red) to compute the new action sequence that will be given to the forward pass. Several forward and backward steps may be necessary to converge to a solution.</figcaption></figure>

## General method to solve the trajectory optimisation problem

The derivatives used for the optimization problem depend on the mechanical quantities $W_{ea}$, $A^{-1}$, $K$ and $D$. Technically these quantities depend on the position
and velocity of all the points of the mesh of the robot. We assume that we are looking
for an actuation around a nominal trajectory and that the mathematical quantities
vary slightly around this trajectory. This means that these quantities are constant for
one time step of the backward phase.

It is therefore necessary to compute a faithful nominal trajectory to obtain
useful admittance, stiffness and damping values for the optimisation problem. To do
so, the forward phase uses the complete dynamics of the robot and the simulation
is used to compute the next values of W<sub>ea</sub>, A<sup>-1</sup>, K and D for the backward phase.
When only the derivative of the linear part of the dynamics is used in the backward
phase, only W<sub>ea</sub> is recovered from the simulation. The global method is summarized
in Figure 5.18 (see Section 2.2.3 for notations).

<figure><img src="image_67.png" /><figcaption>Fig. 5.19: Simulation of the Lianford robot.</figcaption></figure>

5.2.2 Trajectory optimisation

We propose to illustrate the method using three different robots: the Diamond
robot (see section 3.1.2), the Stiff-Flop robot (see section 5.1.5) and the Lianford
robot. The Lianford robot is a soft parallel robot with 6 deformable legs developed
for haptic applications. Each end of the legs is attached to a DC motor, whose
rotation causes the legs to deform. The effector is attached to a central handle,
whose movement is imposed by the deformation of the legs. This robot is illustrated
in Figure 5.19. The trajectory optimization is performed using the proxddp solver
[Jul+22]. This solver relies on a primal-dual augmented Lagangian approach to solve
the optimization problem. The user interface allows to define the underlying optimal control problem on a node-by-node basis, to write the dynamics and constraints, and to specify the differences between forward and backward stages. The forward step has been adapted in order to make the link with SOFA. The results are presented for a fixed maximum number of iterations and tolerance.

### 5.2.3 Scenario 1: Use both derivative of the linear and non-linear part of the dynamic in the backward step.

The first scenario compares the positioning performance obtained when the derivative of the non-linear part of the dynamics is used in the backward phase, and when this is not the case. This comparison is illustrated with the Diamond robot and shown in Figure 5.20. The goal is to position the effector of the Diamond at two successive positions.

From these results we can see that the use of condensed dynamics allows to control the position and velocity of the robot, both with and without the derivative of the non-linear part of the condensed dynamics. The results in terms of positioning error are similar with and without the use of $\frac{\partial f_i}{\partial X^{non-linear}}$. However, the number of iterations to reach the same performances is more important with the non linear part (reach the maximum of 1000 iterations set for this experiments) rather than without (208 iterations). In addition, using the derivative of the non-linear part leads to additional calculations, as the resulting matrix is not calculated directly in the forward stage. More precisely, the matrix $W$ is obtained with $O(m^2)$ operations, where $m$ is the number of constraints. The non-linear part of the derivative is obtained with $O(m \times n)$ operations, where $n$ is the number of elements in the FEM mesh. $n$ is a large number that depends on the accuracy of the FEM model. Therefore, considering only the linear part of the derivative of the dynamics according to $X$ in the backward stage improves the calculation time.

The use of the derivative of the linear part alone in the backward is sufficient to control the soft robot. This is because the full non-linear FEM model is used to find the nominal trajectory, and linearisation around this nominal trajectory in the backward is sufficient to converge to a solution. In the remainder of this work, only the derivative of the linear part of the dynamics is used.
<figure><img src="image_68.png" /><figcaption>Fig. 5.20: Results obtained for Diamond robot control. Top, from left to right: end effector position along *x*, *y* and *z* axes (mm). Middle, left to right: end effector speed along *x*, *y* and *z* axes (mm/s). Green: the objective. Blue: results obtained by considering the derivative of the non-linear part of the condensed dynamics. Orange: results obtained by considering only the linear part. Three robot positions along the trajectory (1, 2, 3) are shown.</figcaption></figure>

### 5.2.4 Scenario 2: Constraint actuation with the PneumaticTrunk.

The second scenario shows that we can constrain the actuation during the trajectory. This scenario is illustrated using the PneumaticTrunk. The results are shown in Figure 5.21. This robot illustrates the method with a different type of actuation. The objective is to position the end-effector attached to the centre or the end of the robot.

The results are shown for one axis and one actuators, but similar results can be plotted for the other axis and actuators. The framework enables one to take into account bounds on the actuation, and can integrate different kind of geometry and actuation.
<figure><img src="image_69.png" /><figcaption>**Fig. 5.21:** Results obtained for the PneumaticTrunk. First row: Effector at the middle of the robot. Second row: Effector at the end of the robot. From left to right: pressure value (bar) for one of the actuators; position of the effectors along one axis (mm); velocity of the effectors along the same axis (mm/s); three configuration of the robots (1, 2, 3). Green: the objective. Orange: position and velocity of the effector. Pink: actuation bounds.</figcaption></figure>

We can see that different deformations can be obtained, depending on the position of the end-effector and the target. In practice, all the positions in the robot's workspace can be reached if enough time steps are available to solve the task.

### 5.2.5 Scenario 3: Multiple goals with the Lianford robot.

The first two scenarios focus on positioning one effector with a maximum of two targets. In addition, the two previous robots are simulated with volumic FEM models, while the Liandford robot is simulated with a lattice of beam FEM elements. The third scenario involves optimizing a trajectory with multiple goals. We consider three different cases. The first is a spiral trajectory where each time step corresponds to a new goal. The second is a pick and place task where the end effector at the centre of the robot has to move back and forth between two positions. The last is an orientation task using two end-effectors at the same time. The results are shown in Figure 5.22.

For the first task, we can see that the robot can be controlled over several time steps, with a goal that varies at each time step. The obtained trajectory corresponds to the one that minimizes the cost function and follows the optimization constraints, at the tolerance level set. There are a few differences between the target trajectory and the computed trajectory: the target trajectory only takes into account
<figure><img src="image_70.png" /><figcaption>Fig. 5.22: Results obtained for the Lianford robot control. First row: Position on the x-z plane (mm) for the spiral trajectory. Green: the objective. Orange: position of the effector on the considering plane. Second row, from left to right: actuation (N), position (m), velocity (m/s), and three configuration of the robot for the pick and place task. Green: the objective. Orange: position and velocity of the effector. Pink: actuation bounds. Third row, from left to right: actuation (N), position (m), velocity (m/s), and three configuration of the robot for the pick and place task. Green and Blue: the objective. Orange and Red: position and velocity of the effector. Pink: actuation bounds.</figcaption></figure>

the positioning of the end-effector, whereas the computed trajectory minimises an objective that includes not only the position, but also the velocity of the effector and the actuation value. Changing the weight of these elements in the loss leads to different results. Here, the emphasis is on positioning. The second task shows that it is possible to control the robot's dynamics on more complex trajectories, such as tasks where it has to navigate between two different positions. Finally, the third task shows that it is possible to control several effectors at the same time.
### 5.2.6 Discussion

We have proposed a method for optimizing the trajectory of soft robots using dynamic FEM models. This method is based on the joint use of a full FEM model to obtain a nominal trajectory in the forward part, and the use of a condensed FEM model to find the actuation in the backward part. This method can handle several types of robot geometry (parallel or slender), several type of FEM elements (tetra, hexa, beam), several types of actuation (cable, pneumatic, joint) and several types of task (one goal, several goals, one trajectory). We have used three different examples to illustrate all these features.

FEM simulation is known to be slower than other methods such as PWCC, but it can faithfully simulate robots with more complex geometry or actuation. It would be possible to use faster models in forward, such as model reduction. The advantage of this method over previous ones is that the optimization does not depend on the choice of model used for the simulation, the condensation being valid for FEM models, reduced models or beam models.

The fact that only the derivative of the linear part of the dynamics is used in the backward step means that no additional computations are required compared with the forward step, and speeds up the convergence of the method. The Diamond example showed that the two approaches lead to similar results. We found that the non-linear part tended to degrade convergence, which is surprising. We have not yet been able to explain this problem. However, the fact that the problem converges with just the linear part in the 3 cases we studied is very positive as it is more efficient in terms of calculation time. In addition, the dynamics used in the forward stage and the linearised condensed dynamics used in the backward stage are not the same. This leads to different gradients between those calculated from the position of the end-effectors and those calculated from the condensed model. This can lead to the framework's dual error not converging, although in practice this does not affect the control of the robots.

It is also possible to extend the capabilities of the method, in particular by adding different constraints. For example, it would be possible to imagine constraints on the maximum displacement increment of actuators, or constraints on the position of the end effector by defining zones where it is not allowed to go. Another extension would be to consider perturbations in the trajectory. This would enable the development of tasks where a force is applied to the robot at a given point in the trajectory. Finally, we are using a line search approach to update the variables, but other methods such as trust region approaches could be considered.
5.3 Conclusion

In this Chapter we have demonstrated that the condensed FEM model can
be used to describe the state of the soft robot in the constraint space. This model
is sufficiently expressive to control (both with and without contact) the soft robot,
and when learned, its differentiability allows for calibration and design parameter
optimisation. Additionally, the prediction is made with a small-scale network, making
it well suited for embedded control algorithms on microprocessors, on which it would
not be possible to compute full FEM models. The prediction is fast and control can be
performed in real time. Finally, the condensed model can also be used to formulate
trajectory optimisation problems using optimal control formalism, allowing for the
control of the soft robot's position and velocity over several time steps.

However, this approach has some limitations. In the context of the learned
model, a large amount of data is required for learning. The amount of data increases
with the complexity of the task, such as control in a large workspace or multipara-
metric design optimisation. Moreover, the prediction performance of the learned
model depends on sampling. To address this issue, it would be beneficial to explore
the possibility of active sampling to update the learned model according to the
robot configurations encountered during exploitation. In the context of trajectory
optimisation, the forward phase involves computing the full robot dynamics, which
can be slow when the underlying FEM model is complex. However, by focusing on
the linear part of the problem in the backward phase, we could explore real-time
applications in which the forward phase is performed with a learned condensed
model.
# Conclusion

## Contents

<table><tr><td>6.1</td><td>Conclusion on the contributions</td><td>137</td></tr><tr><td>6.1.1</td><td>Modelling</td><td>137</td></tr><tr><td>6.1.2</td><td>Simulation.</td><td>137</td></tr><tr><td>6.1.3</td><td>Control</td><td>138</td></tr><tr><td>6.1.4</td><td>Relevance and limitations</td><td>139</td></tr><tr><td>6.2</td><td>Perspectives and futur works</td><td>140</td></tr><tr><td>6.2.1</td><td>Short-term perspective: implicit parameterization and MPC140</td><td></td></tr><tr><td>6.2.2</td><td>Medium- and long-term perspective: differentiable simulation, perception and self-representation</td><td>141</td></tr></table>
<figure><img src="image_71.png" /><figcaption>” 7. Rippling.</figcaption></figure>

— Nemo

And what remains at the end? After a 3-year PhD thesis concludes, after a romance terminates, or even after life itself? There is a captivating concept known as Rippling theory, which suggests that we persist through the effects we had while we were present. There is education we impart to our children, the lessons we teach as professors, the amiability we share with strangers on the street. There are collaborations with colleagues, the research we produce as scientists, and the art that we create. Just as a drop of water on a tranquil lake continues to have an impact after it has dispersed, what we share continues even after we have gone.
The research presented in this PhD thesis can be divided into three main categories: modelling, simulation, and control. In the first part of the conclusion (6.1), the contributions of the manuscript are summarized according to these categories. The relevance and limitations of the methods developed in this study are discussed. Finally, the prospects of this research in both the short- and long-term are examined in Section (6.2).

# 6.1 Conclusion on the contributions

## 6.1.1 Modelling

In this PhD thesis, we modeled soft robots for control and design tasks. In particular, we explored different methods for describing the state of soft robots. In Chapter 3, the state of the robot is described using the FEM model by selecting key points on the robot. In Chapter 4, two strategies for simplifying the description of a robot's state are developed. The first method involves using a proxy model to describe a soft robot in its configuration space. The second method involves partitioning the robot's observation space into different contact configuration spaces. Finally, in Chapter 5, the state of the robot is described using a condensed FEM model in the constraint space.

These models allow to explore the links between soft robotics and artificial intelligence. Full modelling of the soft robot is not useful at all stages of learning. With the proxy, we can see that modelling and simulating the deformations of the robot's skeleton are enough to generate trajectories for control purposes. The graph of configuration spaces demonstrates that it is not necessary to learn the behaviour of the entire system but rather how to move from one contact configuration to another. It is possible to model and learn how to solve the Maze before using other tools to solve it with a Tripod robot. Finally, only the condensed model is used in single- or multi-step control schemes. Through these various examples, we demonstrate that it is possible to be much more subtle than the opposition between physical modelling and learning. There are many ways of combining the different models in order to increase efficiency (in terms of simulation time and resources, as well as sometimes in terms of learning quality).

## 6.1.2 Simulation

This research on various soft robot models leads to the production of a significant amount of open-source code. Notable achievements include the development
of SofiaGym, the creation of a condensed FEM model learning pipeline, and the
implementation of a plugin that enables trajectory optimisation from a soft robot
FEM model.

First, all the results obtained with RL and planning algorithms are based
on SofiaGym. SofiaGym is the first software to integrate RL, planning and FEM
simulations of soft robots. It was developed when few software solutions were
available for these challenges, or when existing solutions had limited capabilities in
simulating complex robots. SofiaGym is built on Gym, a reference API that facilitates
the use of classical RL algorithms. This software is employed in Chapter 3 to
demonstrate the possibility of linking RL and FEM simulations. It is also used in
Chapter 4 to generate a trajectory with a proxy and to learn how to solve tasks using
a graph of configuration spaces.

Chapter 5 focuses on the use of a condensed FEM model for control and design
applications. Learning this condensed model introduces neural networks into the
direct and inverse simulations of soft robots. These neural networks can rapidly
predict the mechanical matrices used in control and design problems, enabling
applications such as embedded control or optimisation of design parameters. The
simulation pipeline developed for this method is distinct from a conventional FEM
simulation. Data are first collected and the networks are trained using these data.
Then, the FEM simulation can be entirely bypassed, and the networks can be used to
simulate the mechanical quantities of soft robots. The condensed model is also used
for trajectory optimisation. The software developed for trajectory simulation can
simulate soft robots from any position on the nominal trajectory and retrieve the
mechanical quantities required for optimisation. This is the first software package
that enables trajectory optimisation from a FEM model of a soft robot using the DDP
method.

6.1.3 Control

The three chapters, Chapter 3, Chapter 4, and Chapter 5, explore the relation-
ship between various methods for describing a robot's state and its control. The
general idea is to control soft robots to perform high level tasks. In Chapter 3, the
study of SofiaGym demonstrates that FEM models of soft robots can be controlled
using either RL algorithms or optimisation methods combined with inverse robot
modelling. However, this chapter also highlights the limitations of these methods.
RL algorithms require a significant amount of training data, which are expensive
to obtain with FEM models, and are not always interpretable. Optimisation-based
methods cannot solve high-level tasks alone and may sometimes result in local
minima.
Based on the examples of CartStemContact, Multigait and Maze, we can identify what is interesting to learn and what is interesting to calculate using optimisation methods. In Chapter 4, the proxy approach separates the control problem into two stages: generating an admissible trajectory using a simplified model of the robot, and tracking and correcting this trajectory using the FEM model of the robot. The graph approach uses optimisation methods when possible and learns action sequences to change the contact configuration when not. These two approaches effectively combine learning and optimisation approaches and address the challenges presented at the beginning of this work.

However, these approaches rely on simplifying the learning stage, either by using a simplified model to represent the robot or by incorporating prior knowledge into the learning process using the configuration spaces graph approach. To overcome this problem, we proposed in Chapter 5 the use of a condensed FEM model of the robot. This is a low-dimensional model that learns mechanical quantities that synthesize the link between constraints in the robot rather than directly learning the inverse model of the robot. It can be learned for control tasks without contact, with contact, and even as a substitute for the FEM simulation. It can also be used in a multi-step control approach, as proposed in the optimal control framework.

### 6.1.4 Relevance and limitations

This study demonstrates the strengths and weaknesses of various control approaches proposed for soft robotics. By focusing on modelling and describing the state of a soft robot, it is possible to effectively control soft robots to perform high level task. Some descriptions also have broader applications, such as the learned condensed FEM model, which can be used for calibration and parameter optimisation. This approach of "learning what is necessary and leaving the rest" has guided this PhD thesis and resulted in the development of new, systematic methods for control issues in soft robotics.

However, this study has some limitations. First, there are no significant testing on physical prototypes. Although FEM models have been used in the past to control physical prototypes, and all the work is based on these models, only the passage from simulation to reality can fully validate the tools presented in this work. However, the passage to reality presents its own challenges that need to be considered, such as the detection of contact configurations from sensor data. These issues offer a wide range of research possibilities outside the scope of this study. Finally, in this work we have defined high-level tasks as generating trajectories for positioning, manipulation or locomotion. However, we are far from the high-level tasks proposed in the rigid robotics literature, like cooking for example. We propose methods to take a further
step in this direction, and further studies are needed to reach this level of capability
for soft robots.

## 6.2 Perspectives and future works

### 6.2.1 Short-term perspective: implicit parameterization and MPC

The work proposed in this study offers numerous development prospects. First, two use cases of the condensed FEM model are presented in Chapter 5. In the framework of trajectory optimisation, we showed that it is possible to only use derivatives of the linear part of the dynamics in the backward step. These derivatives involve only $W$ the admittance matrix projected into constraint space. This is possible because the forward step captures the nonlinearities of the dynamics in order to build the nominal trajectory. The learned condensed FEM model can be integrated into both the forward and backward calculations. Nonlinearities in the dynamics are considered in the forward step using $\delta_e^{free}$. In addition, given the displacement of the actuators, it is possible to calculate the $W$ matrice required for backward computation of the framework. Integrating this learned condensed FEM model into the trajectory optimisation framework makes it possible to imagine MPC-type real-time control applications for physical prototypes.

The second element that can be explored is related to the design of the soft robots. In this study, we investigated two design applications using the condensed FEM model (calibration and parameter optimisation). This work uses a parametric model of the robot [Nav+23]. However, the choice of these parameters limits the available shape of the robot. Generative models [RNA22; Bon+22] can be used to generate realistic soft robot shapes from an existing robot shapes dataset. Using local rules such as Neural Cellular Automata [Mor+20; Sud+21], it is possible to imagine a framework in which realistic robot patches are generated using low-dimensional networks before being combined and assembled using evolutionary algorithms and topological optimisation schemes. The robot's state is described implicitly, as a collection of realistic patches. This would generate robots that look like the robots we know but are optimized for a given geometric or mechanical task.
6.2.2 Medium- and long-term perspective: differentiable
simulation, perception and self-representation

From a slightly longer-term perspective, other elements can be explored. For
instance, work on the simulation differentiability could improve the design and
control applications. The design applications discussed in Chapter 5 are enabled by
a differentiable network that connects design parameters to mechanical quantities.
Research on simulators could result in the ability to obtain gradient-based parameter
optimisation methods without first having to learn a condensed model. This topic
has been explored using different types of simulators, such as the MPM simulator
presented in Chapter 2 [Mat+23]. From a control perspective, differentiability
enables the integration of gradient information into the control methods. Specifically,
contact differentiability, as proposed for rigid robotics [Mon+22], would allow for
the integration of 1st-order contact information, which would change the way we
understand the contact reconfiguration.

Although we have not explored the topic of perception in this PhD study, it is an
important aspect that is fully integrated in describing the state of robots. Specifically,
it answers the question of how the robot describes itself using the data given by
the sensors. An interesting idea is to draw inspiration from both morphological
computation [Hau+11] and the holographic principle defined for biological systems
[Fie+23]. The idea is to represent the robot's mind as a deformable structure that
changes according to the information coming from the sensors, leading to describe
the state of the sensors in a condensed manner. This idea can be applied to any type
of robotic system, whether rigid, soft, or biological [EL21; Bla+22].
Bibliography

[AD19] Sanjeevan Ahilan and Peter Dayan. „Feudal Multi-Agent Hierarchies for Co-operative Reinforcement Learning“. In: ArXiv abs/1901.08492 (2019) (cit. on p. 25).

[Agr+16] Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. „Learning to Poke by Poking: Experiential Learning of Intuitive Physics“. In: CoRR abs/1606.07419 (2016). arXiv: 1606.07419 (cit. on p. 20).

[Akr+18] Riad Akrour, Filipe Veiga, Jan Peters, and Gerhard Neumann. „Regularizing Reinforcement Learning with State Abstraction“. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2018), pp. 534–539 (cit. on p. 25).

[Alm+21] João Damião Almeida, Paul Schydlo, Atabak Dehban, and José Santos-Victor. „Sensorimotor Graph: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics“. In: CoRR abs/2107.08492 (2021). arXiv: 2107.08492 (cit. on p. 36).

[Alo+22] John Irvin Alora, Mattia Cenedese, Edward Schmerling, George Haller, and Marco Pavone. „Data-Driven Spectral Submanifold Reduction for Nonlinear Optimal Control of High-Dimensional Robots“. In: 2022, pp. 2627–2633 (cit. on p. 31).

[And+17] Marcin Andrychowicz, Dwight Crow, Alex Ray, et al. „Hindsight Experience Replay“. In: vol. abs/1707.01495. 2017 (cit. on p. 84).

[Ant73] Stuart S. Antman. „The Theory of Rods“. In: 1973 (cit. on p. 15).

[AZK21] Walid Amehri, Gang Zheng, and Alexandre Kruszewski. „Workspace Boundary Estimation for Soft Manipulators Using a Continuation Approach“. In: vol. 6. 2021, pp. 7169–7176 (cit. on p. 66).

[AZK22] Walid Amehri, Gang Zheng, and Alexandre Kruszewski. „FEM-Based Exterior Workspace Boundary Estimation for Soft Robots via Optimization“. In: vol. 7. 2022, pp. 3672–3678 (cit. on p. 66).

[Bam+22] Antoine Bambade, Sarah El-Kazdadi, Adrien Taylor, and Justin Carpentier. „PROX-QP: Yet another Quadratic Programming Solver for Robotics and beyond“. In: 2022 (cit. on p. 100).

[Bel57] Richard Bellman. „A Markovian Decision Process“. In: vol. 6. 5. Indiana University Mathematics Department, 1957, pp. 679–684 (cit. on p. 24).
[Ber+15] Christos Bergeles, Andrew H. C. Gosline, Nikolay V. Vasilyev, et al. „Concentric Tube Robot Design and Optimization Based on Task and Anatomical Constraints“. In: vol. 31. 2015, pp. 67–84 (cit. on p. 34).

[Ber+20] James M. Bern, Yannick Schnider, P. Banzet, Nitish Kumar, and Stelian Coros. „Soft Robot Control With a Learned Differentiable Model“. In: 2020, pp. 417–423 (cit. on p. 36).

[Bha+19a] Sarthak Bhagat, Hritwick Banerjee, Zion Ho Tse, and Hongliang Ren. „Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges“. en. In: *Robotics* 8.1 (Jan. 2019), p. 4 (cit. on p. 20).

[Bha+19b] Sarthak Bhagat, Hritwick Banerjee, Zion Tsz Ho Tse, and Hongliang Ren. „Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges“. In: *Robotics* 8 (2019), p. 4 (cit. on p. 25).

[Bla+22] D. Blackiston, S. Kriegman, J. Bongard, and M. Levin. „Biological Robots: Perspectives on an Emerging Interdisciplinary Field“. In: 2022. arXiv: 2207.00880 [cs.RO] (cit. on p. 141).

[BM10] Sébastien Bubeck and Rémi Munos. „Open Loop Optimistic Planning“. In: *Annual Conference Computational Learning Theory*. 2010 (cit. on p. 27).

[Bon+22] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G. Willcocks. „Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models“. In: vol. 44. 11. Institute of Electrical and Electronics Engineers (IEEE), Nov. 2022, pp. 7327–7347 (cit. on p. 140).

[Bou+18] Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, et al. „Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping“. In: *2018 IEEE International Conference on Robotics and Automation (ICRA)* (2018), pp. 4243–4250 (cit. on p. 25).

[Bra+07] David Braganza, Darren M. Dawson, Ian D. Walker, and Nitendra Nath. „A Neural Network Controller for Continuum Robots“. In: *IEEE Transactions on Robotics* 23 (2007), pp. 1270–1277 (cit. on p. 20).

[Bro+16] Greg Brockman, Vicki Cheung, Ludwig Pettersson, et al. „Openai gym“. In: 2016 (cit. on p. 16).

[Bud+18] Rohan Budhiraja, Justin Carpentier, Carlos Mastalli, and Nicolas Mansard. „Dif-ferential Dynamic Programming for Multi-Phase Rigid Contact Dynamics“. In: *2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids)* (2018), pp. 1–9 (cit. on pp. 28, 36, 37).

[CB21] Erwin Coumans and Yunfei Bai. *PyBullet*, a Python module for physics simulation for games, robotics and machine learning. http://pybullet.org. 2016–2021 (cit. on p. 16).

[CED17] Eulalie Coevoet, Adrien Escande, and Christian Duriez. „Optimization-Based Inverse Model of Soft Robots With Contact Handling“. In: *IEEE Robotics and Automation Letters* PP (Feb. 2017), pp. 1–1 (cit. on pp. 4, 8, 31, 46).
[CED19] Eulalie Coevoet, Adrien Escande, and Christian Duriez. „Soft robots locomotion and manipulation control using FEM simulation and quadratic programming“. In: *2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)*. IEEE. 2019, pp. 739–745 (cit. on pp. 15, 20, 21).

[Cha+23] Paul Chaillou, Jialei Shi, Alexandre Kruszewski, et al. „Reduced finite element modelling and closed-loop control of pneumatic-driven soft continuum robots“. In: *2023 IEEE International Conference on Soft Robotics (RoboSoft)*. 2023, pp. 1–8 (cit. on p. 111).

[Che+18] Feifei Chen, Wenjun Xu, Hongying Zhang, and al. „Topology Optimized Design, Fabrication, and Characterization of a Soft Cable-Driven Gripper“. In: *IEEE Robotics and Automation Letters* 3 (2018), pp. 2463–2470 (cit. on p. 34).

[Che+21] ShiTong Chen, YuSheng Wang, DeChen Li, FeiFei Chen, and XiangYang Zhu. „Enhancing interaction performance of soft pneumatic-networks grippers by skeleton topology optimization“. In: *Science China Technological Sciences* 64.12 (2021), pp. 2709–2717 (cit. on p. 34).

[Cho+20] Heesun Choi, Cindy Crump, Christian Duriez, et al. „On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward“. In: *Proceedings of the National Academy of Sciences of the United States of America* 118 (2020) (cit. on p. 15).

[Cho+21] HeeSun Choi, Cindy Crump, Christian Duriez, et al. „On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward“. In: *Proceedings of the National Academy of Sciences* 118.1 (2021). eprint: https://www.pnas.org/content/118/1/e1907856118.full.pdf (cit. on p. 21).

[Cia+14] Matteo Cianchetti, Tommaso Ranzani, Giada Gerboni, et al. „Soft Robotics Technologies to Address Shortcomings in Today’s Minimally Invasive Surgery: The STIFF-FLOP Approach“. In: 2014 (cit. on p. 111).

[Cia+18] Matteo Cianchetti, Cecilia Laschi, Arianna Menciassi, and Paolo Dario. „Biomedical applications of soft robotics“. In: vol. 3. 2018, pp. 143–153 (cit. on p. 3).

[Coe+17] Eulalie Coevoet, Thor Morales-Bieze, Frederick Largilliere, et al. „Software toolkit for modeling, simulation, and control of soft robots“. In: *Advanced Robotics* 31.22 (2017), pp. 1208–1224 (cit. on pp. 14, 17).

[Coe19] Eulalie Coevoet. „Optimization-based inverse model of soft robots, with contact handling“. In: 2019 (cit. on pp. 3, 20, 31).

[CW20] Feifei Chen and Michael Yu Wang. „Design Optimization of Soft Robots: A Review of the State of the Art“. In: *IEEE Robotics Automation Magazine* 27.4 (2020), pp. 27–43 (cit. on p. 33).

[DH92] Peter Dayan and Geoffrey E. Hinton. „Feudal Reinforcement Learning“. In: *NIPS*. 1992 (cit. on p. 25).

[DI19] Zhuoran Dang and Mamoru Ishii. „A physics-informed reinforcement learning approach for the interfacial area transport in two-phase flow“. In: *arXiv: Computational Physics* (2019) (cit. on p. 25).

[Dij59] Edsger W. Dijkstra. „A note on two problems in connexion with graphs“. In: vol. 1. 1959, pp. 269–271 (cit. on p. 48).
[DJA22] Fei Deng, Ingook Jang, and Sungjin Ahn. DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations. 2022 (cit. on p. 25).

[Dur+18] Christian Duriez, Hadrien Courtecuisse, Juan-Pablo de La Plata Alcalde, and Pierre-Jean Bensoussan. „Contact Skinning“. In: 2018 (cit. on p. 68).

[Dur13] Christian Duriez. „Control of elastic soft robots based on real-time finite element method“. In: 2013 IEEE International Conference on Robotics and Automation (2013), pp. 3982–3987 (cit. on pp. 15, 46).

[DVT14] Hongkai Dai, Andres K. Valenzuela, and Russ Tedrake. „Whole-body motion planning with centroidal dynamics and full kinematics“. In: 2014, pp. 295–302 (cit. on p. 36).

[Ebe+21] Imme Ebert-Uphoff, Ryan Lagerquist, Kyle Hilburn, et al. „CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences - Version 1“. In: CoRR abs/2106.09757 (2021). arXiv: 2106.09757 (cit. on p. 24).

[EL21] Mohammed Ebrahimkhani and Michael Levin. „Synthetic living machines: A new window on life“. In: vol. 24. 2021 (cit. on p. 141).

[ELS21] Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. „Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification“. In: NeurIPS. 2021 (cit. on p. 25).

[Fa12] Hervé Delingette François Faure Christian Duriez and al. „SOFA: A Multi-Model Framework for Interactive Physical Simulation“. In: Soft Tissue Biomechanical Modeling for Computer Assisted Surgery 11 (2012), pp. 283–321 (cit. on pp. 17, 20).

[Fal+15] Valentin Falkenhahn, Alexander Hildebrandt, Rudiger Neumann, and Oliver Sawodny. „Model-based feedforward position control of constant curvature continuum robots using feedback linearization“. en. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). Seattle, WA, USA: IEEE, May 2015, pp. 762–767 (cit. on p. 16).

[FDA17] Carlos Florensa, Yan Duan, and P. Abbeel. „Stochastic Neural Networks for Hierarchical Reinforcement Learning“. In: ArXiv abs/1704.03012 (2017) (cit. on p. 25).

[Fie+23] Chris Fields, Filippo Fabrocini, Karl Friston, et al. „Control flow in active inference systems“. In: 2023. arXiv: 2303.01514 [q-bio.NC] (cit. on p. 141).

[FL16] Chelsea Finn and Sergey Levine. „Deep visual foresight for planning robot motion“. In: 2017 IEEE International Conference on Robotics and Automation (ICRA) (2016), pp. 2786–2793 (cit. on p. 24).

[Fox+17] Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. „Multi-Level Discovery of Deep Options“. In: ArXiv abs/1703.08294 (2017) (cit. on p. 25).

[GC14] Tipakorn Greigarn and M. Cenk Cavusoglu. „Task-space motion planning of MRI-actuated catheters for catheter ablation of atrial fibrillation“. en. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Chicago, IL, USA: IEEE, Sept. 2014, pp. 3476–3482 (cit. on p. 16).
[GD18] Olivier Goury and Christian Duriez. „Fast, generic and reliable control and simulation of soft robots using model order reduction“. In: *IEEE Transactions on Robotics* (2018) (cit. on pp. 8, 36, 43, 49).

[GD19] S. Gros and M. Diehl. „Numerical optimal control“. In: 2019 (cit. on p. 28).

[Geo+16] Thomas George Thuruthel, Egidio Falotico, Matteo Cianchetti, and Cecilia Laschi. „Learning Global Inverse Kinematics Solutions for a Continuum Robot“. In: vol. 569. Jan. 2016, pp. 47–54 (cit. on p. 15).

[GJC09] Yusof Ganji, Farrokh Janabi-Sharifi, and Asim N. Cheema. „Robot-assisted catheter manipulation for intracardiac navigation“. en. In: *International Journal of Computer Assisted Radiology and Surgery* 4.4 (June 2009), pp. 307–315 (cit. on p. 16).

[Gra+21] Moritz A. Graule, Clark B. Teeple, Thomas P. McCarthy, et al. „SoMo: Fast and Accurate Simulations of Continuum Robots in Complex Environments“. In: *IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 - Oct. 1, 2021*. IEEE, 2021, pp. 3934–3941 (cit. on pp. 16, 17).

[Gu+19] Yueqin Gu, Xuecheng Zhang, Qiuxuan Wu, et al. „Research on Motion Evolution of Soft Robot Based on VoxCAD“. In: *International Conference on Intelligent Robotics and Applications*. 2019 (cit. on p. 34).

[Gul90] Vijaykumar Gullapalli. „A stochastic reinforcement learning algorithm for learning real-valued functions“. In: *Neural networks* 3.6 (1990), pp. 671–692 (cit. on p. 24).

[Gup+19] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. „Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning“. In: *CoRL*. 2019 (cit. on p. 25).

[Haa+18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. „Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor“. In: *CoRR abs/1801.01290* (2018). arXiv: 1801.01290 (cit. on p. 24).

[Haf+18] Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, et al. „Learning Latent Dynamics for Planning from Pixels“. In: *CoRR abs/1811.04551* (2018). arXiv: 1811.04551 (cit. on p. 20).

[Hau+11] Helmut Hauser, Auke Jan Ijspeert, Rudolf Marcel Füchslin, Rolf Pfeifer, and Wolfgang Maass. „Towards a theoretical foundation for morphological computation with compliant bodies“. In: vol. 105. 2011, pp. 355–370 (cit. on p. 141).

[Haw+19] Zachary Hawks, Chase G. Frazelle, Keith Evan Green, and Ian D. Walker. „Motion Planning for a Continuum Robotic Mobile Lamp: Defining and Navigating the Configuration Space“. In: *2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)* (2019), pp. 2559–2566 (cit. on p. 20).

[Hen17] Bernhard Hengst. „Hierarchical Reinforcement Learning“. In: Jan. 2017 (cit. on p. 25).
[Hes+13] Hesheng Wang, Weidong Chen, Xiaojin Yu, et al. „Visual servo control of cable-driven soft robotic manipulator“. en. In: *2013 IEEE/RSJ International Conference on Intelligent Robots and Systems*. Tokyo: IEEE, Nov. 2013, pp. 57–62 (cit. on p. 16).

[HGS15] H. V. Hasselt, Arthur Guez, and David Silver. „Deep Reinforcement Learning with Double Q-Learning“. In: *ArXiv abs/1509.06461* (2015) (cit. on p. 24).

[HM08] Jean-François Hren and Rémi Munos. „Optimistic Planning of Deterministic Systems“. In: *European Workshop on Reinforcement Learning*. 2008 (cit. on p. 27).

[Hom+18] Bianca Homberg, Robert K. Katzschmann, Mehmet Remzi Dogar, and Daniela Rus. „Robust proprioceptive grasping with a soft robot hand“. In: *Autonomous Robots* 43 (2018), pp. 681–696 (cit. on p. 20).

[How60] R. A. Howard. „Dynamic Programming and Markov Processes“. In: Cambridge, MA: MIT Press, 1960 (cit. on p. 24).

[Hu+19a] Yeping Hu, Alireza Nakhaei, Masayoshi Tomizuka, and Kikuo Fujimura. „Interaction-aware Decision Making with Adaptive Strategies under Merging Scenarios“. In: *2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)* (2019), pp. 151–158 (cit. on p. 25).

[Hu+19b] Yuanming Hu, Jiancheng Liu, Andrew Spielberg, et al. „ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics“. In: *2019 International Conference on Robotics and Automation (ICRA)* (2019), pp. 6265–6271 (cit. on p. 17).

[Hu+20] Yuanming Hu, Luke Anderson, Tzu-Mao Li, et al. „DiffTaichi: Differentiable Programming for Physical Simulation“. In: *ArXiv abs/1910.00935* (2020) (cit. on p. 17).

[Hua+21a] Jiang Hua, Liangcai Zeng, Gongfa Li, and Zhaojie Ju. „Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning“. In: *Sensors (Basel, Switzerland)* 21 (2021) (cit. on p. 25).

[Hua+21b] Zhiao Huang, Yuanming Hu, Tao Du, et al. *PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics*. Apr. 2021 (cit. on p. 16).

[Jal+22] Wilson Jallet, Antoine Bambade, Nicolas Mansard, and Justin Carpentier. „Constrained Differential Dynamic Programming: A primal-dual augmented Lagrangian approach“. In: *2022 IEEE/RSJ International Conference on Intelligent Robots and Systems*. Kyoto, Japan, Oct. 2022 (cit. on p. 128).

[JH21] Anthony Vanderkop Jack Collins Shelvin Chand and David Howard. „A Review of Physics Simulators for Robotic Applications“. In: *IEEE Access* 9 (2021), pp. 51416–51431 (cit. on p. 16).

[Jia+21] Xiaowei Jia, Jared D. Willard, Anuj Karpatne, et al. „Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles“. In: *ACM/IMS Transactions on Data Science* 2 (2021), pp. 1–26 (cit. on p. 25).
[JMC22] Wilson Jallet, Nicolas Mansard, and Justin Carpentier. „Implicit Differential Dynamic Programming“. In: *International Conference on Robotics and Automation (ICRA 2022)*. IEEE Robotics and Automation Society. Philadelphia, United States, May 2022 (cit. on pp. 28–30).

[JP13] J. Andrew Bagnell, Jens Kober and Jan Peters. „Reinforcement learning in robotics: A survey“. In: *International Journal of Robotics Research* 32 (2013), pp. 1238–1274 (cit. on p. 20).

[Kah+20] Adar Kahana, Eli Turkel, Shai Dekel, and Dan Givoli. „Obstacle segmentation based on the wave equation and deep learning“. In: *J. Comput. Phys.* 413 (2020), p. 109458 (cit. on p. 24).

[Kar+17] Anuj Karpatne, Gowtham Atluri, James H. Faghmous, et al. „Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data“. In: *IEEE Transactions on Knowledge and Data Engineering* 29 (2017), pp. 2318–2331 (cit. on p. 24).

[KG17] Wah Loon Keng and Laura Graesser. „SLM Lab“. In: GitHub, 2017 (cit. on p. 84).

[Kim+21a] Daekyum Kim, Sang-Hun Kim, Taekyoung Kim, et al. „Review of machine learning methods in soft robotics“. In: *PLoS ONE* 16 (2021) (cit. on pp. 14, 15).

[Kim+21b] J. Taery Kim, Jeongeun Park, Sungjoon Choi, and Sehoon Ha. „Learning Robot Structure and Motion Embeddings using Graph Neural Networks“. In: *ArXiv abs/2109.07543* (2021) (cit. on p. 36).

[KP13a] Mahta Khoshnam and Rajni V. Patel. „A pseudo-rigid-body 3R model for a steerable ablation catheter“. en. In: *2013 IEEE International Conference on Robotics and Automation*. Karlsruhe, Germany: IEEE, May 2013, pp. 4427–4432 (cit. on p. 16).

[KP13b] Mahta Khoshnam and Rajnikant V. Patel. „A pseudo-rigid-body 3R model for a steerable ablation catheter“. In: *2013 IEEE International Conference on Robotics and Automation* (2013), pp. 4427–4432 (cit. on p. 15).

[KS06] Levente Kocsis and Csaba Szepesvári. „Bandit Based Monte-Carlo Planning“. In: *Proc. European Conference on Machine Learning (ECML)*. Vol. 2006. Sept. 2006, pp. 282–293 (cit. on p. 27).

[Kul+16] Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Joshua B. Tenenbaum. „Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation“. In: *NIPS*. 2016 (cit. on p. 25).

[LaV06] Steven LaVallée. *Planning Algorithms*. available freely online at http://lavalle.pl/planning/. Cambridge University Press, 2006 (cit. on p. 27).

[LCY22] Chih-Hsing Liu, Yang Chen, and Sy-Yeu Yang. „Topology Optimization and Prototype of a Multimaterial-Like Compliant Finger by Varying the Infill Density in 3D Printing“. In: *Soft Robotics* 9.5 (Oct. 2022), pp. 837–849 (cit. on p. 34).

[Les+17] Timothée Lesort, Mathieu Seurin, Xinrui Li, Natalia Díaz Rodríguez, and David Filliat. „Unsupervised state representation learning with robotic priors: a robustness benchmark“. In: *ArXiv abs/1709.05185* (2017) (cit. on p. 36).
[Les+18] Timothée Lesort, Natalia Díaz Rodríguez, Jean-François Goudou, and David Filliat. „State Representation Learning for Control: An Overview“. In: *Neural networks : the official journal of the International Neural Network Society* 108 (2018), pp. 379–392 (cit. on pp. 35, 36).

[Leu20] Edouard Leurent. „Safe and Efficient Reinforcement Learning for Behavioural Planning in Autonomous Driving“. In: 2020 (cit. on p. 27).

[Lev+16] Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. „Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection“. en. In: *arXiv:1603.02199 [cs]* (Aug. 2016). arXiv: 1603.02199 (cit. on p. 20).

[Lev+19] Andrew Levy, George Dimitri Konidaris, Robert W. Platt, and Kate Saenko. „Learning Multi-Level Hierarchies with Hindsight“. In: *ICLR*. 2019 (cit. on p. 25).

[Lid+22] Quentin Le Lidec, Louis-Rene Montaut, Cordelia Schmid, Ivan Laptev, and Justin Carpentier. „Leveraging Randomized Smoothing for Optimal Control of Nonsmooth Dynamical Systems“. In: vol. abs/2203.03986. 2022 (cit. on p. 80).

[Lil+15] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, et al. „Continuous control with deep reinforcement learning“. In: *CoRR* abs/1509.02971 (2015) (cit. on p. 24).

[Lin+20] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. „SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation“. In: *CoRL*. 2020 (cit. on p. 16).

[Lin92] Long-Ji Lin. „Reinforcement learning for robots using neural networks“. PhD thesis. Carnegie Mellon University, 1992 (cit. on p. 24).

[Lip14] Hod Lipson. „Challenges and Opportunities for Design, Simulation, and Fabrication of Soft Robots“. In: 2014 (cit. on p. 14).

[Lof+15] Robert Tyler Loftin, Bei Peng, James MacGlashan, et al. „Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning“. In: *Autonomous Agents and Multi-Agent Systems* 30 (2015), pp. 30–59 (cit. on p. 25).

[Low+17] Ryan Lowe, Yi Wu, Aviv Tamar, et al. „Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments“. In: *ArXiv* abs/1706.02275 (2017) (cit. on p. 25).

[LS11] Kfir Yehuda Levy and Nahum Shimkin. „Unified Inter and Intra Options Learning Using Policy Gradient Methods“. In: *EWRL*. 2011 (cit. on p. 25).

[LSH07] Bryn A. Lloyd, Gábor Székely, and Matthias Harders. „Identification of Spring Parameters for Deformable Object Simulation“. In: *IEEE Transactions on Visualization and Computer Graphics* 13 (2007) (cit. on p. 16).

[LSH19] Brett T. Lopez, Jean-Jacques E. Slotine, and Jonathan P. How. „Dynamic Tube MPC for Nonlinear Systems“. In: 2019. arXiv: 1907.06553 [eess.SY] (cit. on p. 80).

[Maj13] Carmel Majidi. „Soft Robotics : A Perspective — Current Trends and Prospects for the Future“. In: 2013 (cit. on p. 14).
[Man+04] Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. „Dynamic abstraction in reinforcement learning via clustering“. In: *Proceedings of the twenty-first international conference on Machine learning* (2004) (cit. on p. 25).

[Mar+14] Andrew D. Marchese, Konrad Komorowski, Cagdas D. Onal, and Daniela Rus. „Design and control of a soft and continuously deformable 2D robotic manipulation system“. en. In: *2014 IEEE International Conference on Robotics and Automation (ICRA)*. Hong Kong, China: IEEE, May 2014, pp. 2189–2196 (cit. on p. 16).

[Mar+22] Tobia Marcucci, Mark Petersen, David von Wrangel, and Russ Tedrake. „Motion Planning around Obstacles with Convex Optimization“. In: 2022. arXiv: 2205.04422 [cs.RO] (cit. on p. 80).

[Mar+23] Tobia Marcucci, Jack Umenberger, Pablo A. Parrilo, and Russ Tedrake. „Shortest Paths in Graphs of Convex Sets“. In: 2023. arXiv: 2101.11565 [cs.DM] (cit. on p. 80).

[Mas+17] Carlos Mastalli, Michele Focchi, Ioannis Havouris, et al. „Trajectory and foothold optimization using low-dimensional models for rough terrain locomotion“. In: 2017, pp. 1096–1103 (cit. on p. 36).

[Mas04] Michael Mascagni. „On the Scrambled Halton Sequence“. In: vol. 10. Dec. 2004, pp. 435–442 (cit. on p. 99).

[Mat+23] David Matthews, Andrew Spielberg, Daniela Rus, Sam Kriegman, and Josh Bongard. „Efficient automatic design of robots“. In: 2023. arXiv: 2306.03263 [cs.RO] (cit. on p. 141).

[MAY73] DAVID Q. MAYNE. „Differential Dynamic Programming–A Unified Approach to the Optimization of Dynamic Systems* *This work was done during the author’s visit to the Division of Engineering and Applied Physics, Harvard University, and was supported by the U.S. Army Research Office, the U.S. Air Force Office of Scientific Research and the U.S. Office of Naval Research under the Joint Services Electronics Program by Contracts N00014-67-A-0298-0006, 0005. and 0008.“ In: ed. by C.T. LEONDES. Vol. 10. Control and Dynamic Systems. Academic Press, 1973, pp. 179–254 (cit. on p. 29).

[MC16] Barbara Mazzolai and Matteo Cianchetti. „Soft robotics: Technologies and systems pushing the boundaries of robot abilities“. In: *Science Robotics* 1 (2016) (cit. on pp. 3, 67).

[MGK22] Brandon H. Meng, Isuru S. Godage, and Iyad Kang. „RRT*-Based Path Planning for Continuum Arms“. In: *IEEE Robot Autom Lett.* 7 (2022), pp. 6830–6937 (cit. on pp. 20, 68).

[MJD18] Jan Matas, Stephen James, and Andrew J. Davison. „Sim-to-Real Reinforcement Learning for Deformable Object Manipulation“. In: *CoRR abs/1806.07851* (2018). arXiv: 1806.07851 (cit. on p. 16).

[MLH10] Igor Mordatch, Martin de Lasa, and Aaron Hertzmann. „Robust physics-based locomotion using low-dimensional planning“. In: 2010 (cit. on p. 36).

[MMD19] Thomas Morzadec, Damien Marchal, and Christian Duriez. „Toward Shape Optimization of Soft Robots“. In: *2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)* (2019), pp. 521–526 (cit. on p. 34).
[Mni+13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. „Playing Atari with Deep Reinforcement Learning“. In: *ArXiv abs/1312.5602* (2013) (cit. on p. 24).

[Mni+15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. „Human-level control through deep reinforcement learning“. In: *nature* 518.7540 (2015), pp. 529–533 (cit. on p. 20).

[Mni+16] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, et al. „Asynchronous Methods for Deep Reinforcement Learning“. In: *ArXiv abs/1602.01783* (2016) (cit. on p. 24).

[Mon+22] Louis Montaut, Quentin Le Lidec, Antoine Bambade, et al. „Differentiable Collision Detection: a Randomized Smoothing Approach“. In: 2022. arXiv: 2209.09012 [cs.RO] (cit. on p. 141).

[Mor+20] Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. „Growing Neural Cellular Automata“. In: 2020 (cit. on p. 140).

[Mun14] Rémi Munos. „From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning“. In: vol. 7. 2014, pp. 1–129 (cit. on p. 27).

[Nau+20] Noel M. Naughton, Jiarui Sun, Arman Tekinalp, et al. „Elastica: A Compliant Mechanics Environment for Soft Robotic Control“. In: *IEEE Robotics and Automation Letters* 6 (2020), pp. 3389–3396 (cit. on p. 17).

[Nav+20] Stefan Escaida Navarro, Steven Nagels, Hosam Alagi, et al. „A Model-Based Sensor Fusion Approach for Force and Shape Estimation in Soft Robotics“. In: vol. 5. 2020, pp. 5621–5628 (cit. on pp. 13, 45).

[Nav+21] Stefan Navarro, Sepaldeep Dhaliwal, Mario Sanz-Lopez, et al. „A Bio-Inspired Active Prostate Phantom for Adaptive Interventions“. In: Oct. 2021 (cit. on p. 45).

[Nav+23] Stefan Escaida Navarro, Tanguy Navez, Olivier Goury, Luis Molina, and Christian Duriez. „An Open Source Design Optimization Toolbox Evaluated on a Soft Finger“. In: vol. 8. 9. Institute of Electrical and Electronics Engineers (IEEE), Sept. 2023, pp. 6044–6051 (cit. on pp. 114, 118, 140).

[Nic+18] Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. „Gotta Learn Fast: A New Benchmark for Generalization in RL“. In: *arXiv preprint arXiv:1804.03720* (2018) (cit. on p. 20).

[NW06] J. Nocedal and S. J. Wright. „Numerical optimization“. In: Springer, 2nd ed., 2006 (cit. on p. 30).

[OGL13] David E. Orin, Ambarish Goswami, and Sung-Hee Lee. „Centroidal dynamics of a humanoid robot“. In: vol. 35. 2013, pp. 161–176 (cit. on p. 36).

[Ope+19a] OpenAI, Ilge Akkaya, Marcin Andrychowicz, et al. „Solving Rubik’s Cube with a Robot Hand“. en. In: *arXiv:1910.07113 [cs, stat]* (Oct. 2019). arXiv: 1910.07113 (cit. on p. 20).

[Ope+19b] OpenAI, Marcin Andrychowicz, Bowen Baker, et al. „Learning Dexterous In-Hand Manipulation“. en. In: *arXiv:1808.00177 [cs, stat]* (Jan. 2019). arXiv: 1808.00177 (cit. on pp. 20, 50).
[Pan+23] Tao Pang, H. J. Terry Suh, Lujie Yang, and Russ Tedrake. „Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Models“. In: 2023. arXiv: 2206.10787 [cs.RO] (cit. on p. 80).

[Par20] J Parker-Holder. „Minimal pytorch soft actor critic (sac) implementation“. In: GitHub, 2020 (cit. on p. 84).

[Pat+21] Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. „Hierarchical Reinforcement Learning: A Comprehensive Survey“. In: *ACM Computing Surveys* 54 (June 2021), pp. 1–35 (cit. on p. 25).

[PG15] Lerrel Pinto and Abhinav Gupta. „Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours“. en. In: *arXiv:1509.06825 [cs]* (Sept. 2015). arXiv: 1509.06825 (cit. on p. 20).

[PH22] Joshua Pinskier and David Howard. „From Bioinspiration to Computer Generation: Developments in Autonomous Soft Robot Design“. In: *Advanced Intelligent Systems* (2022) (cit. on p. 34).

[PR97] Ronald E. Parr and Stuart J. Russell. „Reinforcement Learning with Hierarchies of Machines“. In: *NIPS*. 1997 (cit. on p. 25).

[Put94] Martin L. Puterman. „Markov Decision Processes: Discrete Stochastic Dynamic Programming“. In: *Wiley Series in Probability and Statistics*. 1994 (cit. on p. 22).

[RDM19] Mark Runciman, Ara Darzi, and George P Mylonas. „Soft robotics in minimally invasive surgery“. In: *Soft robotics* 6.4 (2019), pp. 423–443 (cit. on p. 3).

[Ren+17] Federico Renda, Frédéric Boyer, Jorge Manuel Miranda Dias, and Lakmal D. Seneviratne. „Discrete Cosserat Approach for Multisection Soft Manipulator Dynamics“. In: *IEEE Transactions on Robotics* 34 (2017), pp. 1518–1533 (cit. on p. 15).

[RN94] Gavin Rummery and Mahesan Niranjan. *On-line Q-learning using connectionist systems*. Tech. rep. University of Cambridge, Department of Engineering Cambridge, UK, 1994 (cit. on p. 24).

[RNA22] Lyle Regenwetter, Amin Heyrani Nobari, and Faez Ahmed. „Deep Generative Models in Engineering Design: A Review“. In: 2022. arXiv: 2110.10863 [cs.LG] (cit. on p. 140).

[Ros+19] Laura Ros-Freixedes, Anzhu Gao, Ning Liu, Mali Shen, and Guang-Zhong Yang. „Design optimization of a contact-aided continuum robot for endobronchial interventions based on anatomical constraints“. In: vol. 14. 2019, pp. 1137–1146 (cit. on p. 34).

[RT15] Daniela Rus and Michael T. Tolley. „Design, fabrication and control of soft robots“. In: *Nature* 521.7553 (2015), pp. 467–475 (cit. on p. 33).

[Rus+16] Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, et al. „Policy Distillation“. In: *CoRR* abs/1511.06295 (2016) (cit. on p. 25).

[Sch+17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. „Proximal Policy Optimization Algorithms“. In: *ArXiv abs/1707.06347 (2017)* (cit. on p. 24).
[Sei+20] Daniel Seita, Pete Florence, Jonathan Tompson, et al. „Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks“. In: arXiv preprint arXiv:2012.03385 (2020) (cit. on p. 17).

[SG05] Linda Smith and Michael Gasser. „The Development of Embodied Cognition: Six Lessons from Babies“. In: Artificial Life 11 (2005), pp. 13–29 (cit. on p. 5).

[SG19] Andrew N. Sloss and Steven Gustafson. 2019 Evolutionary Algorithms Review. 2019 (cit. on p. 34).

[SGW15] Yoel Shapiro, Kosa Gabor, and Alon Wolf. „Modeling a hyperflexible planar bending actuator as an inextensible euler-bernoulli beam for use in flexible robots“. In: Soft Robotics 2.2 (2015), pp. 71–79 (cit. on pp. 15, 68).

[Sha+16] Bobak Shahriari, Kevin Swersky, Ziyun Wang, Ryan P. Adams, and Nando de Freitas. „Taking the Human Out of the Loop: A Review of Bayesian Optimization“. In: vol. 104. 2016, pp. 148–175 (cit. on p. 68).

[She+11] Robert Shepherd, Filip Ilievski, Wonjae Choi, et al. „Multigait Soft Robot“. In: Proceedings of the National Academy of Sciences of the United States of America 108 (Nov. 2011), pp. 20400–3 (cit. on pp. 4, 48).

[Sig97] Ole Sigmund. „On the design of compliant mechanisms using topology optimization“. In: J. Struct. Mech. 25.4 (1997), pp. 493–524 (cit. on p. 34).

[Sin+19] Nina R. Sinatra, Clark B. Teeple, Daniel M. Vogt, et al. „Ultragentle manipulation of delicate structures using a soft robotic gripper“. In: vol. 4. 33. 2019, eaax5425. eprint: https://www.science.org/doi/pdf/10.1126/scirobotics.aax5425 (cit. on p. 3).

[SIT91] Koichi Suzumori, Shoichi Iikura, and Hirohisa Tanaka. „Development of flexible microactuator and its applications to robotic mechanisms“. In: Proceedings. 1991 IEEE International Conference on Robotics and Automation (1991), 1622–1627 vol.2 (cit. on pp. 3, 4).

[SK22] Filippo A. Spinelli and Robert K. Katzschmann. „A Unified and Modular Model Predictive Control Framework for Soft Continuum Manipulators under Internal and External Constraints“. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Oct. 2022 (cit. on pp. 30, 80).

[SP08] A. J. Shaiju and Ian R. Petersen. „Formulas for Discrete Time LQR, LQG, LEQG and Minimax LQG Optimal Control Problems“. In: vol. 41. 2008, pp. 8773–8778 (cit. on p. 28).

[Spi+19] Andrew Spielberg, Allan Zhao, Tao Du, and al. „Learn- ing-in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations“. In: roc. Adv. Neural Inf. Process. Syst. (2019), pp. 8284–8294 (cit. on p. 34).

[SPS99] Richard S. Sutton, Doina Precup, and Satinder Singh. „Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning“. In: Artificial Intelligence 112.1 (1999), pp. 181–211 (cit. on pp. 23, 25).

[SSW22] Charles Schaff, Audrey Sedal, and Matthew R. Walter. „Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer“. In: (2022). arXiv: 2202.04575 [cs.RO] (cit. on pp. 3, 4, 35).
[Sud+21] Shyam Sudhakaran, Djordje Grbic, Siyan Li, et al. „Growing 3D Artefacts and Functional Machines with Neural Cellular Automata“. In: 2021. arXiv: 2103.08737 [cs.LG] (cit. on p. 140).

[Sui+19] Xin Sui, Hegao Cai, Dongyang Bie, et al. „Automatic Generation of Locomotion Patterns for Soft Modular Reconfigurable Robots“. In: *Applied Sciences* 10.1 (Dec. 2019), p. 294 (cit. on p. 34).

[Tag+20] Eleonora Tagliabue, Ameya Pore, Diego Dall'Alba, et al. „Soft tissue simulation environment to learn manipulation tasks in autonomous robotic surgery“. In: *2020 IEEE International Conference on Intelligent Robots and Systems (IROS)*. *IEEE* (2020) (cit. on p. 17).

[Tal+15] Anthony Talvas, Maud Marchal, Christian Duriez, and Miguel A. Otaduy. „Aggregate Constraints for Virtual Manipulation with Soft Fingers“. In: vol. 21. 2015, pp. 452–461 (cit. on p. 19).

[Tea19] Team DEFROST (INRIA/CRISTAL) Lille. *SoftRobots plugin for SOFA*. https://github.com/SofaDefrost/SoftRobots. 2019 (cit. on p. 17).

[Tes95] Gerald Tesauro. „Temporal difference learning and TD-Gammon“. In: *Commun. ACM* 38 (1995), pp. 58–68 (cit. on p. 24).

[TET12] Emanuel Todorov, Tom Erez, and Yuval Tassa. „Mujoco: A physics engine for model-based control“. In: *Intelligent Robots and Systems (IROS)*, 2012 IEEE/RSJ International Conference on. IEEE. 2012, pp. 5026–5033 (cit. on p. 16).

[Tha94] Chen Khong Tham. „Modular on-line function approximation for scaling up reinforcement learning“. PhD thesis. University of Cambridge, 1994 (cit. on p. 24).

[Thu+17] Thomas George Thuruthel, Egidio Falotico, Mariangela Manti, et al. „Learning Closed Loop Kinematic Controllers for Continuum Manipulators in Unstructured Environments.“ In: *Soft robotics* 4 3 (2017), pp. 285–296 (cit. on p. 15).

[Thu+19] Thomas George Thuruthel, Egidio Falotico, Federico Renda, and Cecilia Laschi. „Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators“. In: *IEEE Transactions on Robotics* 35 (2019), pp. 124–134 (cit. on p. 15).

[TLP20] Sander Tonkens, Joseph Lorenzetti, and Marco Pavone. „Soft Robot Optimal Control Via Reduced Order Finite Element Models“. In: 2020, pp. 12010–12016 (cit. on p. 31).

[Tob+17] Joshua Tobin, Rachel Fong, Alex Ray, et al. „Domain randomization for transferring deep neural networks from simulation to the real world“. In: *2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)* (2017), pp. 23–30 (cit. on p. 25).

[Tri+08] Deepak Trivedi, Christopher D. Rahn, William M. Kier, and Ian D. Walker. „Soft robotics: Biological inspiration, state of the art, and future research“. In: *Applied Bionics and Biomechanics* 5 (2008), pp. 99–117 (cit. on p. 14).

[UA22] Wyatt Ubellacker and Aaron Ames. „Robust Locomotion on Legged Robots through Planning on Motion Primitive Graphs“. In: 2022. arXiv: 2209.07503 [cs.RO] (cit. on p. 80).
[Van+20] Félix Vanneste, Olivier Goury, Jonàs Martínez, et al. „Anisotropic Soft Robots Based on 3D Printed Meso-Structured Materials: Design, Modeling by Homogenization and Simulation“. In: vol. 5. 2020, pp. 2380–2386 (cit. on pp. 47, 66).

[VAP20] Filipe Veiga, Riad Akrour, and Jan Peters. „Hierarchical Tactile-Based Control Decomposition of Dexterous In-Hand Manipulation Tasks“. In: *Frontiers in Robotics and AI* 7 (2020) (cit. on p. 25).

[Vas+17] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, et al. „Attention is All you Need“. In: *NIPS*. 2017 (cit. on pp. 83, 124).

[Wan+16] Hongqiang Wang, Jie Chen, Henry Ying Kei Lau, and Hongliang Ren. „Motion Planning Based on Learning From Demonstration for Multiple-Segment Flexible Soft Robots Actuated by Electroactive Polymers“. In: *IEEE Robotics and Automation Letters* 1 (2016), pp. 391–398 (cit. on p. 20).

[Wan+20] Rixin Wang, Xianmin Zhang, Benliang Zhu, et al. „Topology optimization of a cable-driven soft robotic gripper“. In: *Structural and Multidisciplinary Optimization* 62.5 (2020), pp. 2749–2763 (cit. on p. 34).

[Wil+22] Jared D. Willard, Xiaowei Jia, Shaoming Xu, Michael S. Steinbach, and Vipin Kumar. „Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems“. In: *ACM Computing Surveys (CSUR)* (2022) (cit. on pp. 20, 24).

[WJ10] Robert J. Webster and Bryan A. Jones. „Design and Kinematic Modeling of Constant Curvature Continuum Robots: A Review“. In: *The International Journal of Robotics Research* 29 (2010), pp. 1661–1683 (cit. on pp. 14, 15, 67).

[WLK21] Xiaomei Wang, Yingqi Li, and Ka-Wai Kwok. „A Survey for Machine Learning-Based Control of Continuum Robots“. In: *Frontiers in Robotics and AI* 8 (2021) (cit. on p. 20).

[Wu+19] Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. „Learning to manipulate deformable objects without demonstrations“. In: *arXiv preprint arXiv:1910.13439* (2019) (cit. on p. 17).

[Yar+21] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. „Reinforcement Learning with Prototypical Representations“. In: *ICML*. 2021 (cit. on p. 25).

[YHM23] Yao Yao, Liang He, and Perla Maiolino. „SPADA: A Toolbox of Designing Soft Pneumatic Actuators for Shape Matching based on Surrogate Modeling“. In: 2023. arXiv: 2305.19509 [cs.RO] (cit. on p. 114).

[YZK20] Liu Yang, Dongkun Zhang, and George Em Karniadakis. „Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations“. In: *ArXiv abs/1811.02033* (2020) (cit. on p. 24).

[Zha+15] Xiong Zhang, Zhen Chen, Jianhui Liao, and Yan Liu. „The material point method“. In: 2015 (cit. on p. 16).

[Zha+17] Hongying Zhang, Michael Yu Wang, Feifei Chen, et al. „Design and development of a soft gripper with topology optimization“. In: *2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*. 2017, pp. 6239–6244 (cit. on p. 34).
# List of Figures

1.1 Four examples of soft robots, including: 1) The first soft gripper introduced in [SIT91]. 2) A robot inspired by an elephant trunk [CED17], simulated (bottom), and manufactured (top). 3) A pneumatic crawling robot presented in [She+11]. 4) The automatic design of a soft robot presented in [SSW22]. 4

1.2 Illustration of different spaces that can be used to describe the state of a robot. A trunk-like robot (represented by a dark gray segment) fixed at one end is driven by a cable (represented by a thin red segment) pulled by a motor (represented by a red square with a white circle inside). These elements, as well as particular points of the robot (in dark gray) or the end-effector (in green) can be used to describe the state of the robot. 1) Robot at rest. 2) Robot in the actuated state. 6

1.3 Contributions of this PhD thesis. Chapter 3: Using FEM simulation for learning. Exploring control methods using the SofaGym API and applying a RL algorithm with a trunk-like robot in space. Chapter 4: Coupling learning-based and convex-optimisation-based control methods. Representation of the robot in the configuration space with the robot proxy and splitting the observation space with respect to the contact configuration of the robot. Chapter 5: Using condensed FEM model. Learning a reduced mechanical model for control and design applications. 8

2.1 Examples of soft-robotic modelling using: 1) a geometric method, image from [WJ10]. 2) a mechanical model, image from [Coe+17]. 3) a learning-based approach, image from [Kim+21a]. 14

2.2 Overview of popular robot simulation frameworks, table from [Gra+21]. 16

2.3 Illustration of a simple MDP for a trunk-like robot. The action space is composed of the possible values of torque applied to the motor (red square). The state space includes the positions of the different points of the robot (gray dot), the effector (green dot), and the goal to reach (red dot). The reward is the Euclidean distance between the end effector and the target in the 2D plane $(x, y)$. The transition function is given by the dynamic of the soft robot. 22
<table>
  <tbody>
    <tr>
      <td>2.4</td>
      <td>Illustration of RL algorithm function. An agent interacts with an environment via an action, and, given a current state, it receives information about the next state (transition function) and the immediate reward (reward function).</td>
      <td>23</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>A visual representation of a tree search algorithm for a trunk-like robot (dark gray in the image) with five actions. These five actions represent five different motor configurations (red and blue squares in the image). Starting from an initial configuration c<sub>0</sub> (1), five actions are tested (2), which yields five new configurations of the robot. A leaf c<sub>i</sub> is selected for expansion (3). The best path in the tree leads to the lowest reward.</td>
      <td>26</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Illustration of optimal control problem resolution. The goal is to minimize the distance between the nominal trajectory of points x<sub>k</sub> (red points) and an objective sequence (green points). The backward step computes an action sequence u* that minimizes the sum of the point-to-point distances δ<sub>k</sub> between the trajectories. The forward step then computes a new nominal trajectory defined by states x*.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>Illustration of the use of simplified models in motion planning in humanoid robotics, from [Bud+18]. From left to right: defining an admissible trajectory with the robot's base, planning the contact sequence from the trajectory, and calculating the actual actuation required to follow the contact sequence using optimal control methods.</td>
      <td>37</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>Illustration of SofaGym components. SofaGym links SOFA (soft robot simulation) and Gym (RL environment) via a toolbox. This toolbox allows to apply in the SOFA simulation the elements given by Gym like the action and to recover the elements of interest like the observation of the robot's state or the immediate reward.</td>
      <td>42</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Representation of the CartStemContact robot. Left: CartStemContact in SofaGym. A vertical deformable beam is attached to a mobile base. The base can translate along a defined horizontal axis. Two obstacles obstruct the movement of the beam. The objective is to position the tip of the beam (green point in the image) at a given abscissa (red vertical bar in the image), which may be located behind one of the obstacles. Middle: Elements used to describe the robot's state, actuation and reward. Right: RTF obtained for different values of dt (s).</td>
      <td>44</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Representation of the Gripper robot. Left: Gripper in SofaGym. A cable-actuated gripper is used to bring a deformable cube to a certain height, as indicated by the red point in the figure. Middle: Elements used to describe the robot's state, actuation and reward. Right: RTF obtained for different values of dt (s).</td>
      <td>45</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>3.11</td>
      <td>Example of solving a positioning task with CartStemContact and QPCC algorithm. 1) When the robot starts without contact, the QP algorithm falls into a local minimum. If the cart moves to the right, it creates contact and moves away from the goal; if it moves to the left, it also moves away from the goal (top right). 2) To solve this task, the beam must first be in contact with the opposite obstacle.</td>
      <td>56</td>
    </tr>
    <tr>
      <td>3.12</td>
      <td>Solving the CartStemContact problem using the SAC algorithm. Top: cumulative reward. Bottom: solving steps. 1) Random goal and base positions. 2) Using the opposite contact to deform the beam. 3) Solving the positioning task.</td>
      <td>57</td>
    </tr>
    <tr>
      <td>3.13</td>
      <td>Illustration of the Trunk robot handling a cup.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.14</td>
      <td>Hand-defined and learned strategy steps for solving the locomotion task with the Multigait robot.</td>
      <td>59</td>
    </tr>
    <tr>
      <td>3.15</td>
      <td>Solving the maze in simulation with inverse model and in real world with open-loop control.</td>
      <td>60</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>Pipeline for using the proxy model to solve a given task in the case of the trunk like robot. Starting from a complete model, an abstract model called proxy is created (1). The proxy is then used to perform the learning and find a realistic trajectory of the robot (2). The actual actuation of the robot is calculated from the learned trajectory using an inverse model (3). The obtained motor control is applied to the complete model of the robot.</td>
      <td>65</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>Control scheme used during the transfer. (1) General case. A trained RL algorithm gives a curvature instruction α<sub>b,t</sub> to the proxy model to solve a task. Following this instruction, a position reference P&#773;<sub>t</sub> is extracted and compared to the current position of the robot P<sub>t</sub> to form a corrected position command c&#773;<sub>t</sub>. The realignment allows taking into account the position actually reached by the robot P<sub>t</sub> or the manipulated objects O<sub>t</sub>. An inverse model gives an actuation command that can be corrected by taking into account information from both the proxy model and the robot. (2) The inverse model is used to find a pneumatic actuation V<sub>t</sub>. A corrector C is used to correct the actuation to take into account the angles observed in the proxy θ&#773;<sub>t</sub>. (3) Only the inverse model is used to find the cable lengths l<sub>t</sub>.</td>
      <td>70</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>Representation of the FEM (1) and proxy models (2) of the Multigait and Trunk soft robots. The deformable beams composing the skeleton of the proxy model are depicted in red. The external meshes for both models are identical.</td>
      <td>71</td>
    </tr>
  </tbody>
</table>
4.4 Example of strategy obtained on the proxy model and transferred to the complete model in the case of the Multigait and Trunk robots. (1) Strategy learned with the proxy model of the Multigait robot. (2) Transfer of the strategy to the FEM model of the Multigait robot. (3) Strategy learned with the proxy model of the Trunk robot. (4) Transfer of the strategy to the FEM model of the Trunk robot. Despite differences in deformation, the strategy can be transferred between these two models. 74

4.5 Results for the Multigait robot. A) Reward obtained with the FEM models (mm) by following different proxies over the time (s). B) Results obtained with the corresponding proxies over the time (s). For A) and B), the blue line represent the results obtained for the optimized proxy, while other colors correspond to proxy with noisy parameters. C) Evolution of the position of one leg's end in contact with the ground, along the x-axis (mm). D) Evolution of the position of one leg's middle point in the plan (x, z) (mm). For C) and D), the blue dots correspond to the position obtained with the FEM model following the position of the proxy model given by the red dots. 75

4.6 Results for the Trunk robot. A) Cube to goal distance obtained with the FEM models (mm) by following different proxies over the time (s). B) Cube to goal distance obtained with the corresponding proxies over the time (s). For A) and B), the blue line represent the results obtained for the optimized proxy, while other colors correspond to proxy with noisy parameters. The algorithm stops when the distance between the cube and the goal is less than 5mm, represented by a horizontal green line. The task is then considered solved. C) Evolution of the position of the point in contact with the cube in the plan (x, z) (mm). D) Evolution of the position of the tip of the Trunk in the plan (x, z) (mm). For C) and D), the blue dots correspond to the position obtained with the FEM model following the position of the proxy model given by the red dots. 76

4.7 Schematic of the different elements constituting the knowledge graph. Each node in the graph corresponds to a configuration space, and each observation belongs to one node. Each node has an identifier, a neighborhood (other configuration spaces), an internal agent (in red), and an external agent (in blue). The Evaluator (in pink) can either belong to a node or be shared between the nodes. The Selector (in orange) is global. Three steps are performed. (1) The Selector determines in which configuration space the robot is. (2) The Evaluator decides in which configuration space to go. (3) The internal agent or external agent solves the task in the current space or moves to a different space. 82
<table>
  <tbody>
    <tr>
      <td>4.8</td>
      <td>Splitting the state space into different configuration spaces (right) for two soft systems (left) based on the contact configuration. (A) CartStemContact. (B) RodManipulator. In this example, some contact configurations are gathered in one configuration space, not useful for the manipulation task.</td>
      <td>86</td>
    </tr>
    <tr>
      <td>4.9</td>
      <td>Learning results, reward as a function of the iterations. (A) Results obtained with the SAC algorithm (orange), with our method (blue) and with our method with internal agents performed with convex optimisation (purple) in the case of the CartStemContact. The learning conditions are the same in all three examples. The initial difference comes from the fact that the first results are obtained after 500 iterations, and that the method with internal agent performed with optimisation learns to solve the task faster than the other methods. (B) Results obtained with our method in the case of RodManipulator. Sliding average is used to facilitate the reading of the results. The size of the windows for the sliding average is approximately 2.5% of the number of iterations.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>4.10</td>
      <td>Example of RodManipulator task resolution for a target angle of 280°. Starting without contact, the algorithm first uses an expert trajectory to position the robot in contact with the rod. The rod is then manipulated until it reaches a configuration where it is not possible to move further without changing the contact configuration. The Evaluator then uses another expert trajectory to reconfigure the contacts and continues to rotate the rod until it reaches 280°.</td>
      <td>88</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Illustration of the proposed framework and its applications. A physical soft robot is modeled using the FEM. The condensed modeling is obtained by projecting the FEM characteristic matrices in the constraint space. A neural network is trained offline for predicting the condensed FEM modeling from both the initial robot condensed state and applied actuation. For a given robot, a single learned condensed FEM modeling may be used in several applications i.e. for real-time embedded control, for inverse control involving a set of predefined contact points, for controlling several coupled similar robots altogether as well as for calibration and design optimization.</td>
      <td>95</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Illustration of the condensed FEM model for both the Diamond and Soft Finger robots. They are respectively actuated by 4 and 2 cables. In both cases, a FEM model of the robot (left) is used to learn a condensed model (right). The reduced model is based on the projection of the compliance matrix into the constraint space, resulting in drastically reducing the number of characteristic dimensions describing a robot state.</td>
      <td>96</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>5.10</td>
      <td>Illustration of the robot used to illustrate embedded control. A) Stiff-Flop robot simulated in SOFA. The effector of the robot is represented by a green dot, and the goal by an orange dot. B) Physical prototype of the Stiff-Flop robot.</td>
      <td>111</td>
    </tr>
    <tr>
      <td>5.11</td>
      <td>Control results for the Stiff-Flop robot. The 3D-position (mm) of the end effector is compared on a trajectory for 40 different goals (orange dots) when using learned mechanical matrices (red cross). 3 different position of the Stiff-Flopp robot along this trajectory are shown as examples. The effector is represented by a green dot, and the goal by an orange dot.</td>
      <td>113</td>
    </tr>
    <tr>
      <td>5.12</td>
      <td>Soft Finger parametric design. Design parameters are A) Length, B) Height and C) Joint Height.</td>
      <td>115</td>
    </tr>
    <tr>
      <td>5.13</td>
      <td>Optimization history for the calibration of the Soft Finger robot. Results have been obtained with an initial learning rate of 0.01 used conjointly with an adaptive scheme as described in section 5.1.2. Both fitness function (A) and mechanical parameters histories (B,C,D) across optimization iterations are displayed for different initial guesses of the mechanical parameters. Initial mechanical parameters are a fixed Young Modulus of 3000 GPa as well as B) Poisson Ratio = 0.47, C) Poisson Ratio = 0.49 and D) Poisson Ratio = 0.45.</td>
      <td>119</td>
    </tr>
    <tr>
      <td>5.14</td>
      <td>Illustration of design objectives for the parametric Soft Finger. A) Soft Finger in the initial state. B) Bending angle α(δ<sub>a</sub>, p) reached for a fixed cable displacement δ<sub>a</sub>. C) Contact force β(δ<sub>a</sub>, p) generated for a fixed cable displacement δ<sub>a</sub>.</td>
      <td>120</td>
    </tr>
    <tr>
      <td>5.15</td>
      <td>Fitness landscapes around the initialized design selected with grid search strategy, for A) dexterity fitness function (Length = 38.0mm, Height = 20.57mm, Joint Height = 6.28mm) and B) strength fitness function (Length = 42mm, Height = 22mm, Joint Height = 8mm). The best design after optimization is shown as a blue dot.</td>
      <td>122</td>
    </tr>
    <tr>
      <td>5.16</td>
      <td>Optimization histories and results for single objectives case on the parametric Soft Finger. Fitness functions histories respectively for A) kinematics and C) strength. Design parameters histories respectively for B) kinematics and D) strength. Resulting designs respectively for E) kinematics and F) strength.</td>
      <td>122</td>
    </tr>
    <tr>
      <td>5.17</td>
      <td>Pareto Front for the design optimization of the Soft Finger regarding Dexterity and Strength metrics and generated using a Grid Search strategy. Design parameters for several design sampled: A) Length = 38.0mm, Height = 21.6mm, Joint Height = 5.0mm, B) Length = 40.4mm, Height = 20.0mm, Joint Height = 5.6mm, C) Length = 40.2mm, Height = 22.0mm, Joint Height = 7.4mm, D) Length = 42.0mm, Height = 22.0mm, Joint Height = 8.0mm</td>
      <td>123</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>5.18</td>
      <td>Illustration of the trajectory optimization pipeline. Starting from an initial position, the forward phase computes the different positions and velocities of the complete mesh using the full dynamics (in orange). From the simulation data, a nominal trajectory is extracted and the mechanical quantities necessary for the DDP are saved (in blue). The nominal trajectory (green) is used with the condensed trajectory (red) to compute the new action sequence that will be given to the forward pass. Several forward and backward steps may be necessary to converge to a solution.</td>
      <td>127</td>
    </tr>
    <tr>
      <td>5.19</td>
      <td>Simulation of the Lianford robot.</td>
      <td>128</td>
    </tr>
    <tr>
      <td>5.20</td>
      <td>Results obtained for Diamond robot control. Top, from left to right: end effector position along x, y and z axes (mm). Middle, left to right: end effector speed along x, y and z axes (mm/s). Green: the objective. Blue: results obtained by considering the derivative of the non-linear part of the condensed dynamics. Orange: results obtained by considering only the linear part. Three robot positions along the trajectory (1, 2, 3) are shown.</td>
      <td>130</td>
    </tr>
    <tr>
      <td>5.21</td>
      <td>Results obtained for the PneumaticTrunk. First row: Effector at the middle of the robot. Second row: Effector at the end of the robot. From left to right: pressure value (bar) for one of the actuators; position of the effectors along one axis (mm); velocity of the effectors along the same axis (mm/s); three configuration of the robots (1, 2, 3). Green: the objective. Orange: position and velocity of the effector. Pink: actuation bounds.</td>
      <td>131</td>
    </tr>
    <tr>
      <td>5.22</td>
      <td>Results obtained for the Lianford robot control. First row: Position on the x-z plane (mm) for the spiral trajectory. Green: the objective. Orange: position of the effector on the considering plane. Second row, from left to right: actuation (N), position (m), velocity (m/s), and three configuration of the robot for the pick and place task. Green: the objective. Orange: position and velocity of the effector. Pink: actuation bounds. Third row, from left to right: actuation (N), position (m), velocity (m/s), and three configuration of the robot for the pick and place task. Green and Blue: the objective. Orange and Red: position and velocity of the effector. Pink: actuation bounds.</td>
      <td>132</td>
    </tr>
  </tbody>
</table>
List of Tables

<table>
  <tbody>
    <tr>
      <td>3.1</td>
      <td>Comparison of the computation time using QP and OPD to solve a positioning task with the Diamond robot.</td>
      <td>55</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Comparison of the cumulative reward and final x position, for complete and reduced model, obtained with hand defined and learned strategy, for an episode of 6 time steps.</td>
      <td>59</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>RTF obtained for the two robots, Mu. = Multigait, Tr. = Trunk, F. = FEM, P. = Proxy, MOR = Model Order Reduction</td>
      <td>73</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Data acquisition parameters for each robot for n<sub>l</sub> = 6500 sample points. Cables displacement are sampled in [δ<sub>a,min</sub>, δ<sub>a,max</sub>] (mm). Final test loss values for Diamond and Soft Finger robots for 9000 learning iterations. The best trained network encountered is kept at best epoch.</td>
      <td>102</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Final test loss values and FPS of the simulation when using the condensed FEM model for the 3 Soft Finger robots with different meshes discretizations. The best trained network encountered is kept at each epoch.</td>
      <td>104</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>Data acquisition parameters and final loss for Soft Finger and Soft Gripper robots in the contact case. The number of training epochs for the Soft Finger and the Soft Gripper are respectively 50000 and 42540.</td>
      <td>106</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Data acquisition parameters for the Stiff-Flop robot for n<sub>l</sub> = 50000 sample points. Actuator displacements are sampled in [δ<sub>a,min</sub>, δ<sub>a,max</sub>] (mm<sup>3</sup>). Final test loss values are displayed for 50000 learning iterations. The best trained network encountered is kept at best epoch.</td>
      <td>112</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>Bounds of sampling parameters for the condensed FEM model of the Soft Finger with varying mechanical parameters.</td>
      <td>115</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>Bounds of sampling parameters for the condensed FEM model of the Soft Finger with varying geometrical dimensions.</td>
      <td>116</td>
    </tr>
    <tr>
      <td>5.7</td>
      <td>Test loss values for each parametric Soft Finger dataset for both G and F neural networks.</td>
      <td>116</td>
    </tr>
    <tr>
      <td>5.8</td>
      <td>Positioning error for different values of geometrical and mechanical parameters. Error is defined as |d<sub>simu</sub> - d<sub>learned</sub>| / d<sub>simu</sub>, where d<sub>i</sub> corresponds to the mean euclidean distance between the effector and the goal evaluated over the trajectory, and obtained whether from simulation or condensed FEM model.</td>
      <td>117</td>
    </tr>
  </tbody>
</table>
Declaration

I declare here that I have completed my work solely and only with the help of the
references I mentioned and the advice of my supervisor and colleagues.

Lille, December 21st, 2023

Etienne Ménager