# Sim-Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning
## 4 Real-Sim Mapping
### Real-Sim Model  
The proposed real-to-sim mapping model is shown in Figure 9 where the real camera inputs are passed through a CNN model before feeding them to the DRL model. From this, the robot utilizes the already trained model without the need for further adaptations.  
To achieve the mapping, the key frames of a successfully executed trial run of DRL robot control in a simulated environment are recorded and saved as a reference database. In this case, 60-70 images (frames) of a successful operation are saved in varying stages of the robot motion as shown in Table 8. After that, random images in subsequent successful trial runs are recorded to create a CNN training dataset as described in Table 9. The CNN model performs a classification of the current input to the nearest approximation of the simulation database.  
Figure 9: Proposed real–sim mapping model showing the selection of the best corresponding virtual image set from the real-world camera input.  
The dataset in use is shown in Table 9. For training, the database was split into 80:20 for training and testing (validation). To supplement the training, data augmentation was performed on the dataset by adding variations (noise) in images that are expected in real-world robot operations, e.g., image rotations, pixel shifting, and blurs, amongst others.  
The architecture of the CNN model is shown in Figure 10 consisting of four convolutional and pooling layers, followed by one fully connected layer. The convolutional layer has a kernel size of 3 \(\times\) 3 and a stride of 1. The pooling layer has a kernel size of 2 \(\times\) 2 and uses Max pooling. Dropout is applied to the all-coupling layer. The output layer adapts the SoftMax function, the other layers adapt the ReLU function, and Adam is used as the optimization function. Each input image was concatenated to 64 \(\times\) 192 pixels from image processing. The loss function is the cross-entropy error.  
A trained model was created for each of the image processing strategies supplied in the previous section. The accuracy of the mapping was successful for all of the dimension reduction strategies adopted in the study. Figure 11 shows the learning curve of the grayscale and binarization strategies, as a representative sample of the training process. The accuracy converges to approximately 100% around 40 epochs for all image processing, indicating that a model capable of mapping has been successfully obtained.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Image Processing** & **Key Frames (Labels)** \\ \hline \
\begin{tabular}{c} Grayscale \\ Binarization \\ 10-gradation gray scale \\ 4-gradation gray scale \\ 10-gradation gray scale w/median filter \\ 4-gradation gray scale w/median filter \\ \end{tabular} & 65 \\
\begin{tabular}{c} 64 \\ 62 \\ 59 \\ 63 \\ 67 \\ \end{tabular} & 62 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Key frames (labels) per processing strategy.  
Figure 10: Architecture of the CNN model in use.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Image Processing** & **Training Data** & **Validation Data** & **Total** \\ \hline \begin{tabular}{c} Grayscale \\ Binarization \\ 10-gradation gray scale \\ 4-gradation gray scale \\ 10-gradation gray scale \\ 4-gradation gray scale \\ 10-gradation gray scale w/median filter \\ \end{tabular} & 6912 & 1728 & 8640 \\ \begin{tabular}{c} 10-gradation gray scale \\ 6404 \\ 6804 \\ 6804 \\ 7128 \\ \end{tabular} & 6674 & 8370 \\ \begin{tabular}{c} 4-gradation gray scale \\ 1701 \\ 1782 \\ \end{tabular} & 6805 \\ \
\begin{tabular}{c} 10-gradation gray scale w/median filter \\ 1782 \\ \end{tabular} & 1701 & 8505 \\
\begin{tabular}{c} 4-gradation gray scale w/median filter \\ \end{tabular} & 7128 & 1782 & 8910 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Deep learning CNN dataset.