# Position Control of a Mobile Robot through Deep Reinforcement Learning
## 4 Simulated Environment and Mobile Robot
### Environments  
To generate an interface that converts the CoppeliaSim simulation to an environment compatible with RL agents and control algorithms, the OpenAI Gym library is used. Gym provides a simple definition of RL environments, formalized as partially observable Markov decision processes (POMDPs) [40]. Figure 6 is a diagram of the communication that happens in the environments and shows the flow of information between the algorithm or agent, the gym environment, the API and, finally, the CoppeliaSim program. The environments correspond to a simulation of a platform in the CoppeliaSim program, where there is a simulated mobile robot Khepera IV and a red hemisphere (see Figure 7) located on the surface of the platform (in some cases, obstacles are present). Figure 5: Mobile Robot Khepera IV: (**a**) real robot [37], (**b**) simulated robot. Figure 6: Communication diagram. The objective is to drive the robot from its current position to a predefined target point (the red sphere). Each episode ends when the TP is reached, when the maximum number of steps allowed per episode is exceeded, when the robot leaves the maximum allowed area (4 square meters) or when the robot collides with an obstacle, if any. At the beginning of a new episode, the position of the TP and that of the robot are randomly changed, as is the orientation of the robot. The reward function is observed in Equation (9), As the robot draws close to the TP, the reward increases until it becomes zero. In addition, when using the environment to train an agent, collisions are allowed, but they are penalized, adding a negative factor (\(r_{collision}=-10\)) to the reward; in this way, the robot learns to not collide with obstacles. Additionally, when the robot reaches the TP, a positive reward is delivered (\(r_{arrival}=100\)), rewarding the robot for reaching its goal. The values of \(r_{collision}\) and \(r_{arrival}\) were determined empirically to minimize the training time of the RL agents and maximize their performance in the experiments. \[\text{reward}=\left\{ \begin{matrix}r_{arrival}&if\text{ the robot reaches the TP}\\ r_{collision}-\left(d^{2}\right)&if\text{ the robot collides}\\ -\left(d^{2}\right)&in\text{another case}\end{matrix} \right. \tag{9}\]  
Two environments were created with different action spaces, depending on the agent to be used. In the case of DQN, there is a discrete action space of size 3: turn left, go straight or turn right, which correspond to the minimum set of instructions that allow for free movement in the environment. For DDPG, the set of actions is continuous so that the linear and angular velocities of the robot can be manipulated, these two being the action space. Continuous actions are normalized to obtain better agent behavior, but in the environment, they are mapped to the allowed values of linear and angular velocity. The observation space for these environments is presented in Table 1. Table 1 shows that the values used correspond to the distance to the TP (\(d\)), which varies between 0 and 2.82 m (minimum and maximum distance between the robot and the TP); the angular error (\(O_{c}\)) between the orientation of the robot and the TP, with values between \(-\pi\) and \(\pi\); the linear and angular velocity for the previous step; and an arrangement with the measurements of the eight sensors that the Kephera IV robot has. The sensors take measurements from 0 to 20 cm, but the data are normalized between 0 and 1. All these values can be obtained from the simulation or calculated by requesting information from the API, such as the position of the robot and the TP, which helps to calculate the distance to the TP and the angular error. In a real case, the absolute position and orientation of the robot can be obtained by using an Indoor Positioning System (IPS) [50]. Note that the action space depends only on the agent or algorithm to be used; instead, the observation space depends on the environment and its physical constraints.