# Balanced Domain Randomization for Safe Reinforcement Learning
## 4 Balanced Domain Randomization
For safety-specific evaluation, BDR reduces the minimum episodic cost by 25.35% (from 109.97 to 82.09), the mean episodic cost (worst 10%) by 34.46% (from 484.53 to 317.56), and the mean episodic cost by 9.62% (from 61.62 to 55.69), over the baseline. These results demonstrate the efficacy of BDR in reducing safety risks while maintaining task performance. Figure 6: Performance trends of the baseline and BDR over training time. In both the Pendulum and Walker tasks, BDR consistently outperforms the baseline in worst-case performance (minimum episodic return and mean episodic return in worst 10%) while maintaining average-case performance (mean episodic return) during training. ## 6 Conclusions

This paper presents Balanced Domain Randomization (BDR) to address the imbalance problem in contextual Reinforcement Learning (RL) by prioritizing the agent to learn its policy in rare domains during training. Our key idea is to automatically identify insufficiently learned domains by evaluating context rarity based on the statistics of context vectors. BDR leverages Mahalanobis distance in the context embedding space to assess rarity and adaptively adjust the training weights, ensuring sufficient exposure to rare scenarios. Our experimental results on the Pendulum and Walker tasks demonstrate that BDR improves worst-case performance, with a 5.11% gain over the baseline in Pendulum and 24.81% in Walker, while maintaining average performance. BDR has the potential to enhance the robustness of RL agents for real-world applications, such as robotics and autonomous driving, by improving sim-to-real transfer and reducing the risk of failure in unpredictable scenarios. While some hyperparameter tuning is required, this could be further explored in future work. Moreover, we expect that BDR's generalization capabilities can be extended to entirely unseen domains. Conceptualization, C.K., W.C. and J.C.; methodology, C.K. and W.C.; software, C.K. ; validation, C.K. and W.C.; formal analysis, C.K. and W.C.; investigation, C.K. and W.C.; resources, C.K., W.C. and J.C.; data curation, C.K. ; writing--original draft preparation, C.K. and W.C.; writing--review and editing, C.K., W.C. and J.C.; visualization, C.K. ; supervision, J.C.; project administration, J.C.; funding acquisition, J.C. All authors have read and agreed to the published version of the manuscript. This work was supported by the Unmanned Swarm CPS Research Laboratory Program of Defense Acquisition Program Administration and Agency for Defense Development (UD220005 VD). Not applicable. The data and code are available at [[https://github.com/CheongWoong/CARL_analysis](https://github.com/CheongWoong/CARL_analysis)]([https://github.com/CheongWoong/CARL_analysis](https://github.com/CheongWoong/CARL_analysis)) (accessed on 15 August 2024). Author [PERSON] was employed by the company INEEJI Corporation. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. ## Abbreviations

The following abbreviations are used in this manuscript: \begin{tabular}{l l} RL & Reinforcement Learning \\ UDR & Uniform Domain Randomization \\ ADR & Automatic Domain Randomization \\ BDR & Balanced Domain Randomization \\ MDP & Markov Decision Process \\ POMDPs & Partially Observable MDPs \\ \end{tabular} \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Task** & **Method** & **Minimum Episodic Return** & **Mean Episodic Return (Worst 10\%)** & **Mean Episodic Return** \\ \hline \multirow{4}{*}{Navigation} & Baseline & 0.12 & \(-\)1.82 & 2.81 \(\pm\) 1.37 \\  & BDR & **0.72** & \(-\)**0.35** & **9.88 \(\pm\) 4.84** \\ \cline{2-5}  & \multicolumn{2}{c}{**Maximum Episodic Cost**} & **Mean Episodic Cost (Worst 10\%)** & **Mean Episodic Cost** \\ \cline{2-5}  & Baseline & 109.97 & 484.53 & 61.62 \(\pm\) 16.96 \\  & BDR & **82.09** & **317.56** & **55.69 \(\pm\) 11.09** \\ \hline \hline \end{tabular}
\end{table}