# Comparison of Various Reinforcement Learning Environments in the Context of Continuum Robot Control
## 1 Introduction  
Reinforcement learning (RL) is an area of machine learning that is expanding rapidly and has various applications, such as robotics, control systems, and decision support systems [1, 2, 3]. An active area of research involves the application of RL in the control of continuum robots. These robots typically have continuous state and action spaces [4, 5, 6]. In this study, our objective was to design and compare various RL environments to control continuum robots by varying the reward functions. The importance of this work lies in the growing utilization of continuum robots in various industrial and medical applications, and the potential of RL to enhance their control performance. However, the configuration of the RL environment, including the selection of the reward function, can significantly impact the performance of the RL algorithm. Through the comparison of various environments, this study aims to offer valuable insights into the design of RL environments for continuum robot control. Current research in the field frequently uses actor-critic methods, specifically the Deep Deterministic Policy Gradient (DDPG) [2], for continuum robots. However, there is a lack of comparative research on RL environments for controlling these robots. This study aims to address this gap by examining the performance of different environments in controlling the continuum robot. In this study, our goal is to control a three-section planar continuum robot, which is a type of robot that has a structure that is continuous and flexible. There are several approaches to controlling continuum robots, including classical control algorithms based on explicitly implemented kinematics or dynamics, such as PID, and predictive control [7, 8, 9, 10]. However, an alternative and increasingly popular approach is to use RL to solve this problem [11, 12, 13, 14]. For example, in [13], a control strategy was used based on deep Q learning to achieve 2D position reaching. Additionally, in [14], a model-free control method based on Q learning was introduced and implemented on a multi-segment soft manipulator to achieve 2D position reaching. Using RL algorithms enables a control policyto be learned by interacting with the continuum robot, which is treated as the environment. This approach offers the advantage of not having to explicitly consider the dynamics or kinematics of the robot in the design process. Moreover, it allows the control policy to be learned through experimentation with the robot. Continuum robots present unique control challenges due to their flexibility and high degree of freedom. Traditional methods struggle to handle their non-linear dynamics and uncertainties [15]. RL can learn effective control policies even with model inaccuracies, making it a promising approach to address existing limitations and advance the control of continuum robots. While the three-section planar continuum robot may seem simple, controlling it effectively can still be a challenging problem, especially when we consider the potential for non-linearities, uncertainties, and changes in the robot or the environment. Therefore, we believe that this system is a suitable testbed for exploring the potential of RL. The primary aim of this study is to examine the application of the DDPG algorithm, as well as the effect of various environment designs, to control three-section continuum robots. To sum up, the primary objective of this study is to create different RL environments for the control of continuum robots by varying the reward functions and assess their effectiveness. The primary outcome of this study will offer valuable insights into the development of RL environments for controlling continuum robots, which will be useful for researchers and practitioners working in the fields of robotics and control systems. The continuum robot is described by its kinematics and input signals. To investigate the influence of the design choices of the RL environment on the learning process, we consider different reward functions.