# Occlusion Avoidance in a Simulated Environment Using Reinforcement Learning
## 5 Architecture and Training
### Pre-Training  
We created a labeled dataset using the above-defined environment for performing pre-training. For the database, all the ShapeNet-provided object categories were used. A total of 20 images for 6500 random scenes were generated, which gave us a total of 130,000 256 \(\times\) 256 RGB-D images with the corresponding gradient directions and occlusion masks. Moreover, the camera positions for each image were saved. The algorithm used for dataset generation is shown in Algorithm 1.  
```
1:Input:
2# of scenes: \(N_{s}\)
3:# of images per scene: \(N_{i}\)
4:Output:
5:Dataset: \(D\)
6:for\(j\) in range(\(N_{s}\)) do
7: Initialize Camera
8:for Object in range(\(3\)) do
9: Initialize Object \(\rightarrow\) Objects
10:for\(k\) in range(\(N_{i}\)) do
11: Render Camera Image \(\to\)\(I_{j,k}\)
12: Calculate Occlusion Mask \(\rightarrow\)\(M_{j,k}\)
13: Calculate Camera Gradient via SGD \(\rightarrow\)\(G_{j,k}\)
14: Zip Data as [\(I_{j,k}\), \(M_{j,k}\), \(G_{j,k}\)] \(\rightarrow\)\(D_{j,k}\)
15: Move Camera Position Randomly
16: Zip Data \(D_{j,k}\) over \(k\)\(\rightarrow\)\(D_{j}\)
17: Zip Data \(D_{j}\) over \(j\)\(\rightarrow\)\(D\)
18:return\(D\)
```  
**Algorithm 1** Dataset generation  
Figure 2: The neural network architecture. All variants used \(3\times 3\) convolution kernels, in the case of the atrous variant, dilation was set to 2. In the S variants, the \(3\times 3\) convolution was separated to a sequence of \(3\times 1\), \(1\times 3\) and \(1\times 1\) convolutions.  
Backface culling and HardFlatShader were used, respectively, for rasterizing and shading the 3D models. We then split the dataset randomly using an 80-20% ratio. We did so while keeping all 20 images resulting from the individual scenes in the same split--that is either training or validation. The resulting task was, therefore, more complex than if the dataset split jad been carried out along the individual images.  
The resulting dataset provided images with occluded objects as well as occlusion maps for three objects. Moreover, it provided the environment gradient for the optimal step at each environment state, i.e., relative camera and object positions. In contrast with previously published occlusion datasets [18; 19], this array of data at each given datapoint allowed the training of a PAL agent.  
We evaluated the model following every epoch during the training time, using the validation split, and assured that the best performing models were saved. We augmented the dataset using Color Jitter and randomized horizontal flipping, and trained the model using an AdamW optimizer with Cosine Annealing Learning Rate Schedule. For the two heads of the network we used different loss functions. The occlusion mask prediction was evaluated using the Soft Dice Coefficient (SDC), whereas the gradient direction was optimized with MSE loss between the ground truth and predicted values. In practice, the sine and cosine values of the gradient direction were used for calculating the error. To summarize, the loss function used for pre-training is the following:  
\[L_{SD}=1-\frac{2~{}f_{\theta}(Img)~{}\bigodot~{}M_{O}}{f_{\theta}(Img)^{2}+M _{O}^{2}}, \tag{10}\]  
\[L_{MSE}=\|g_{\Psi}(Img)-[sin(\alpha),~{}cos(\alpha)]\|^{2}, \tag{11}\]  
\[L=\beta_{1}L_{SD}+\beta_{2}L_{MSE}, \tag{12}\]  
where \(f\) and \(g\) are the segmenter and gradient predictor networks parameterized by \(\theta\) and \(\Psi\), respectively. \(Img\) is the input image, while \(\alpha\) and \(M_{O}\) are the ground truth gradient direction and occlusion mask. \(\bigodot\) is an element-wise multiplication, and \(\beta_{1}\) and \(\beta_{2}\) are the relative weights of the two loss terms. Table 1 shows the hyperparameters we used during training.