# Sim-Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning
## 1 Introduction
To reduce the difficulty of the task, a 2D simulator was created for the robot to reach the target. In this case, the authors used an input simulator image of 160 \(\times\) 320 pixels that was reduced and gray-scaled to 84 \(\times\) 84 pixels. In this case, the joints of the robot arm were position-controlled and moved \(\pm\)0.02 rad per step. For each step, the robot increased, decreased, or did not change the joint angle. The reward setting was determined according to the change in distance between the end-effector and the target position: +1 reward if the distance was closer from the state before the step, \(-1\) reward if it was further away, and 0 if it did not change. Similar research was done by [PERSON] et al., who they used image-based RL to optimize the grasping motion of a 7-DOF (3-translational and rotational motions of the tip and an extra rotation of the elbow) robot arm in 3D simulations [15]. In the study, they used a 3D simulation to optimize the grasping motion of a cube as the grasping object. The target was for the robot arm to grasp the cube and lift it to a height of 30 cm. It was also confirmed that the task success rate varied depending on the initial angles. However, when the optimal controller obtained in virtual space was implemented on the actual machine, the robot arm moved to the cube but did not close the gripper. These results show that the gripper has limited timing for opening and closing compared to the control of the six joints, making it difficult to effect an accurate closing of the gripper. Among the sim-real approaches, [PERSON] et al. proposed a real-sim-real transfer learning vision-based robot grasping task [16]. In the study, a task-relevant simulated environment was created utilizing semantic information from a real-world scenario and transformed into a virtual scenario for real-to-sim training using the DRL technique. Other studies that utilize synthetically generated environments are reported in [10; 11; 17]. The challenge with these approaches is they need sophisticated processing to re-generate synthetic data that best approximate the real world. Research in [18] reported vision-based robot operations with an auto-tuned sim-real transfer system utilizing search parameter modeling. The proposal attempts to match simulator system parameters to the real world without updating RL states. In this strategy, a search algorithm predicts whether a sequence of actions and the corresponding system parameters are higher or lower than the real-world values for corresponding observations. The approach reported successful transfers to simulation and reality in environments where other strategies such as domain randomization would fail. From the review of the literature, the DRL robot control method that feeds back external information obtained through camera images is effective to certain degrees. As such, the present study utilized the visual observation of the environment to perform pick and place robot operations. A sim-real transfer learning was adopted to enhance the training process in the virtual environment. To address the problem of RG, we propose employ image processing techniques as well as a search algorithm that maps reality to simulation. In this way, the current proposed DRL system consists of a search algorithm motivated by [18] but is different in the search modality. In the proposed approach, an extra layer of a convolutional neural network (CNN) model is added as an attempt to match the observed real-world to a learnt-simulator scene. By mapping the real-world observation to an already learnt scene in the simulator, the policy is able to approximate a true representation of reality, leading to a policy that is likely to succeed in the real world without updating RL states. This study therefore presents a computationally efficient algorithm that combines the DRL system with a CNN mapping model to realize sim-real transfer. In summary, the main contributions of this paper are listed as follows:  
1.