# Comparison of Various Reinforcement Learning Environments in the Context of Continuum Robot Control
## 2 Materials and Methods
### The Continuum Robot Environment  
The environment is designed to simulate the kinematics of a continuum robot for directing purposes. The environment is composed of the state space, action space, and reward function. The robot's current location and target location are included in the state space. The action space contains information on how the robot moves, such as changes in location or velocity. Feedback from the reward function is used to evaluate the effectiveness of a control policy. The DDPG algorithm is depicted in Figure 2 as a basic schematic diagram. It is evident from Figure 2 that a simulation-based environment should be created to control the robot. In this section of the paper, the environment created for continuous robot simulation will be described in depth. To provide a clearer understanding of the planar continuum robot used in our study, we present the kinematics parameters in Table 1. These parameters include the length and curvature of each section of the robot. The kinematics of a continuum robot have been extensively researched in the literature [16; 17]. As previously mentioned, the continuum robot's environment is described by its kinematics and input signals. To achieve this, a framework based on forward and velocity kinematics has been developed. The model is described as follows:  
\[\dot{\mathbf{x}}=J\dot{\mathbf{q}} \tag{1}\]  
where \(x\in\mathbb{R}^{m\times 1}\) is the vector that represents the robot's position in task space, and the dot indicates differentiation with respect to time. It is important to mention that this applies specifically to our planar robot with three sections as follows:  
\[\dot{\mathbf{q}}=[\kappa_{1},\kappa_{2},\kappa_{3}]^{T},\dot{\mathbf{x}}=[ \dot{x},\dot{y}]^{T} \tag{2}\]  
The matrix \(J\) is known as the Jacobian matrix, and is a function of the 'curvature' variable \(\mathbf{q}\). We can obtain the final equation, Equation (3), from Equations (1) and (2). \begin{table}
\begin{tabular}{c c c} \hline \hline
**Section** & **Length (m)** & **Curvature (m\({}^{-1}\))** \\ \hline
1 & 0.1 & \([-4,16]\) \\ \hline
2 & 0.1 & \([-4,16]\) \\ \hline
3 & 0.1 & \([-4,16]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Kinematics parameters of the planar continuum robot. Figure 2: Schematic diagram of Deep Deterministic Policy Gradient. \[\left[ \begin{matrix}\frac{dx}{dt}\\ \frac{dt}{dt}\end{matrix} \right]=\left[ \begin{matrix}\frac{\partial x}{\partial x _{1}}&\frac{\partial x}{\partial x_{2}}&\frac{\partial x}{\partial x_{3}}\\ \frac{\partial x}{\partial x_{1}}&\frac{\partial y}{\partial x_{2}}&\frac{ \partial y}{\partial x_{3}}\end{matrix} \right]\left[ \begin{matrix}\frac{dx_ {1}}{dt}\\ \frac{dx_{2}}{dt}\\ \frac{dx_{3}}{dt}\end{matrix} \right] \tag{3}\]  
In Equation (3), the Jacobian expresses the connection between two separate system representations in a dynamic manner. This connection is a result of changes in both position and curvature in our scenario. The aim of velocity kinematics is to define the movement of endpoints. Since curvatures limit this movement quite a bit, explaining the state of motion at certain curvatures is comparable to explaining the movement of the endpoint. According to Equation (3), the DDPG algorithm calculates the actions that affect the environment as derivatives of the curvatures for each segment over time. The Jacobian matrix is computed with planar forward kinematics [16] and numerical differentiation. The resulting motion is then transformed to the final position using Equation (4) after each action. \[\mathbf{x_{t+1}}=\mathbf{x_{t}}+\dot{\mathbf{x_{t}}}\cdot dt \tag{4}\]  
where \(dt\) is the time step which is decided to be 0.05 s. The objective of this work is to navigate the robot from its initial state \(\mathbf{x_{init}}\) to the goal state \(x_{goal}\) of [\(-\)0.2, 0.15]. The decision to keep the goal state fixed was made in order to reduce the learning time required. This allows for more efficient testing of various environment designs.