# Occlusion Avoidance in a Simulated Environment Using Reinforcement Learning
## 7 Discussion  
In this paper, a novel method was presented that successfully implemented occlusion avoidance as a perception-action-loop. The method was trained using a combination of self-supervised and reinforcement learning techniques, and was able to efficiently optimize the camera position in an OpenAI Gym-compatible virtual environment. The environment was created using PyTorch3D's differentiable renderer and provided three random occluded objects from the ShapeNet dataset. In total, the following four agents were presented: the first used the gradients calculated analytically using differentiable rendering; the second \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **Random** & **Env. Grad** & **Pred. Grad** & **Agent-UN** & **Agent-IN** \\ \hline Mean steps & 31.5 & 20.8 & 18.5 & 15.9 & **15.8** \\ Std & 13.9 & 10.2 & 8.2 & 4.1 & **3.8** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Effectiveness of occlusion avoidance: The table shows the average number of steps the strategies took to reach the unoccluded state, and the variance. Best results are emphasized in bold. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Baseline** & **Random** & **Random** & **Random** & **Env. Grad** & **Env. Grad** & **Pred. Grad** \\
**Method** & **Env. Grad** & **Pred. Grad** & **Agent-IN** & **Grad** & **Agent-IN** & **Agent-IN** \\ \hline HDI low & 6.42 & 9.15 & 11.69 & \(-\)0.21 & 2.11 & 0.05 \\ HDI high & 14.97 & 17 & 19.36 & 2.43 & 6.74 & 1.63 \\ \hline Diff of & & & & & & \\ Means & 10.73 & 12.98 & 15.58 & 1.03 & 4.34 & 0.84 \\ \hline Confidence & 100 & 100 & 100 & 94.6 & 100 & 98.47 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Bayesian \(t\)-test comparing the four methods of occlusion avoidance. We display the bounds of the 95% HDI between the two methods, as well as the difference of means and the confidence that the second of the compared methods reached the unoccluded state _significantly_ faster that the first. employed a random policy; the third used self-supervised learning to predict the optimal direction of movement; the fourth employed reinforcement learning to avoid occlusion in a minimal number of steps. The first and second agents were used as baselines for comparison. The third and fourth methods used the RGB-D images as input. Our experiments demonstrated that all agents performed well in the environment; however, the agents that employed learning managed to significantly outperform all others. The agent using reinforcement learning proved to be the most reliable, as demonstrated by its credibly superior performance according to the Bayesian \(t\)-test performed during our investigations. The RL-based agent also had the smallest variance in the number of steps taken to avoid occlusion, further cementing it as the highest performing solution. Naturally, the methods come with some limitations, namely that they have been trained and tested in synthetic environments, where both the color and depth images used as inputs were devoid of any errors. This likely meant that, in order to use this agent in a real environment, sim-to-real transfer would have to be done. Furthermore, the background used in the environment would also need to be updated to include a more realistic background, with walls, floor and ceiling, as the existence of these in a real setting may confuse an agent trained on fully synthetic data. Our datasets and the code used have been made available at [[https://github.com/MILAB-IIT-CV/OcclusionEnv](https://github.com/MILAB-IIT-CV/OcclusionEnv)]([https://github.com/MILAB-IIT-CV/OcclusionEnv](https://github.com/MILAB-IIT-CV/OcclusionEnv)) (accessed on 27 January 2023). Conceptualization, M.S. ([PERSON]) and M.S. (Mat\(\hat{\text{y}}\)s Szanto); methodology, M.S. ([PERSON]) and M.S. ([PERSON]\(\hat{\text{y}}\)s Szanto); software, M.S. ([PERSON]) and M.S. ([PERSON]\(\hat{\text{y}}\)s Szanto); validation, M.S. ([PERSON]) and M.S. ([PERSON]\(\hat{\text{y}}\)s Szanto); formal analysis, M.S. ([PERSON]) and M.S.