# Sim-Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning
## 2 Materials and Methods
### Deep Reinforcement Learning System  
Reinforcement learning is a kind of machine learning in which an agent observes the current state in a certain environment and acquires the optimum behavior through trial and error. If the agent takes a certain action stochastically from the current state, the agent will be rewarded by the environment. In 2013, [PERSON] et al. proposed a DRL model using a CNN in the behavioral value function \(Q(s,a)\) of RL [20]. Deep Q-Network is a method that combines deep learning with a RL method called Q-learning. The update formula of the action value function \(Q(s,a)\) can be expressed as shown in (1), assuming the state \(s_{t}\), action \(a_{t}\), and reward \(r_{t}\) at time \(t\). The update process involves selecting the action with the highest value in the state \(s_{t+1}\) and updating it with the obtained immediate reward \(r_{t+1}\)  
Figure 4: Variation in the initialization position of the grasping target. (**a**) Left. (**b**) Center. (**c**) Right. Figure 3: Grasping object used in the actual machine and simulated environment. (**a**) 3D printed target object. (**b**) Virtual target object. When renewing, the value is discounted at the discount rate \(\gamma\) because the future reward is uncertain. \[Q(s_{t},\ a_{t})=Q(s_{t},\ a_{t})+\]  
\[\alpha\Bigg{(}r_{t+1}+\gamma\max_{a_{t+1}\in A(s_{t+1})}Q(s_{t+1},a_{t+1})-Q(s_ {t},\ a_{t})\Bigg{)} \tag{1}\]  
From the above, when the number of states becomes enormous, it is difficult to calculate the action value function \(Q(s,\ a)\). [PERSON] et al. proposed a method to reduce the overestimation of the behavioral value by using two types of networks, a network to be learned and a network for calculating value: Double Deep Q-Network [21]. The update formulae and states are shown in Equations (2) and (3). The Double Deep Q-Network is used as the algorithm in the proposed system. \[a_{main}=\max_{a_{t+1}\in A(s_{t+1})}Q(s_{t+1},a_{t+1}) \tag{2}\]  
\[Q(s_{t},\ a_{t})=Q(s_{t},\ a_{t})+\]  
\[\alpha\Bigg{(}r_{t+1}+\gamma\max_{a_{t+1}\in A(s_{t+1})}Q(s_{t+1},a_{t+1})-Q(s_ {t},\ a_{t})\Bigg{)} \tag{3}\]  
Figure 5 shows the proposed system that takes in the camera images of the real or virtual space and feeds it to deep learning CNN for inference. The image of 480 \(\times\) 640 pixels acquired from three cameras is preprocessed to reduce the dimensions to 64 \(\times\) 64 pixels. In visual-based ML, the computational load as well as the network size increase with image dimension. Dimensionality reduction is one of the approaches adopted in RL literature to reduce model complexity while maintaining performance [22; 23]. Next, an image from either virtual space or a real camera is passed to the training/inference deep reinforcement learning model. The model outputs discrete actions affecting (increasing/decreasing) the motor values of individual joints. The robot arm is connected to the PC via serial communication. Deep Q-Network uses a CNN composed of three convolutional layers/pooling layers and two fully connected layers. In this case, the CNN was set up with a 5 \(\times\) 5 kernel size with single stride. The kernel size of the pooling layer was 2 \(\times\) 2, and Max pooling was used. The activation function of all layers was adapted to the ReLU function, and Adam was used as the optimization function. The architecture is shown in Figure 6. The input  
Figure 5: Proposed deep learning control scheme. image from each of the three cameras was 64 \(\times\) 64 pixels, and the output was the number of actions (10 actions). The Huber loss function was applied to the error function. The output drives the corresponding motor as per the action set described below. In this experiment, the search space for the RL model was set as a discrete space with 10 action sets as shown in Table 3. From this, three distinct actions can be set for each joint: clockwise: R, anti-clockwise: L, and stop: S. The fourth action is reserved for the gripper as shown in action 10, \"close: C\".