# Comparison of Various Reinforcement Learning Environments in the Context of Continuum Robot Control
## 2 Materials and Methods
### The Continuum Robot Environment
The initial position \(\mathbf{x_{init}}\) of the continuum robot is selected randomly within the target space, as illustrated in Figure 3, which also displays the goal state and potential initial positions. In our study, we primarily focus on the kinematic aspects of the continuum robot, as represented by Equations (3) and (4). We acknowledge that these equations do not capture the mechanical and physical features of the robot. However, our aim was to demonstrate that effective control can be achieved using RL, even with a simplified model. The complexity and computational demand of a more detailed model could slow down the learning process. The most important aspect of the RL approach is defining the environment. In our work, we incorporate the environment directly from the continuum robot model, as described in Equations (3) and (4). As a result, the states and the actions of the environment are shown in Table 2. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**State Space** & \(x_{t}\) & \(y_{t}\) & \(x_{goal}\) & \(y_{goal}\) \\
**Action Space** & \(x_{1}\) & \(x_{2}\) & \(x_{3}\) & \\ \hline \hline \end{tabular} Here, each \(\kappa\) value is limited to the range of \([-1,+1]\), and the state variables are confined to the task space shown in Figure 3.  
\end{table} 
Table 2: State and action space of the environment. Figure 3: Task space of three-section continuum robot. In order to finalize the environment, it is necessary to design the reward. To simplify the description of the reward function, the state space is designed to not exceed the robot's task space. Additionally, we define the Euclidean distance to \(x_{goal}\) as shown in Equation (5). \[d_{u}=\sqrt{(x-x_{goal})^{2}+(y-y_{goal})^{2}} \tag{5}\]  
The definition of a reward can vary depending on its intended impact on the learning process. In this proposal, we suggest a set of rewards that can help address the issue of continuum robot control, with the goal of facilitating the robot's access from its initial state to its goal state. The rewards take into account the robot's position, with the first reward being given below:  
\[r_{1}=-1\cdot(d_{u})^{2} \tag{6}\]  
The first reward function proposed in the paper for solving the problem of continuum robot control is defined in Equation (6). The reward function takes into account the robot's current position in relation to the goal state. It is calculated as the negative of the squared Euclidean distance between the current state and the goal state, as shown in Equation (5). The purpose of the reward function is to motivate the robot to move from its starting position to the goal state. The reward is set as negative to discourage the robot from deviating from the goal state, with the amount of the penalty increasing as the Euclidean distance between the robot and the goal state increases. This reward function serves as the foundation of the RL algorithm used for controlling the continuum robot and plays a pivotal role in shaping the robot's behavior during the learning phase. The second reward function is defined as  
\[r_{2}=\left\{ \begin{array}{ll}1&d_{u}<d_{u-1}\\ -0.5&d_{u}=d_{u-1}\\ -1&d_{u}>d_{u-1}\end{array} \right. \tag{7}\]  
The second reward function considers the change in Euclidean distance between the current state and the goal state, represented by \(d_{u}\). The reward is assigned based on whether the distance between the current state and the goal state is decreasing, constant, or increasing. If the Euclidean distance between the current state and the goal state decreases, a reward of \(1\) is assigned. If the distance remains constant, a reward of \(-0.5\) is assigned. If the distance increases, a reward of \(-1\) is assigned. This reward function aims to incentivize the robot to move towards the goal state, and penalize it for moving away or staying at the same distance from the goal state.