# Balanced Domain Randomization for Safe Reinforcement Learning
## 4 Balanced Domain Randomization
In the Pendulum task, BDR improves the minimum episodic return by \(5.11\%\) (from \(-1263.97\) to \(-1199.37\)), the mean episodic return (worst \(10\%\)) by \(7.31\%\) (from \(-1221.57\) to \(-1132.30\)), and the mean episodic return by \(3.45\%\) (from \(-392.70\) to \(-379.14\)), over the baseline. In the Walker task, BDR enhances the minimum episodic return by \(24.81\%\) (from \(539.62\) to \(673.49\)), the mean episodic return (worst \(10\%\)) by \(23.57\%\) (from \(516.80\) to \(638.59\)), and the mean episodic return by \(0.79\%\) (from \(878.21\) to \(885.18\)). These consistent improvements across both tasks suggest that BDR reduces overfitting to common domains and enhances adaptability in rare scenarios. Figures 3 and 4 show the average episodic return for each domain in the Pendulum and Walker tasks, respectively. We observe that the improvements are especially evident in domains with extreme conditions, such as the low length in the Pendulum task and the low gravity in the Walker task. Specifically, when the length is \(0.1\) in the Pendulum task, the BDR achieves an average episodic return of \(-670.6\), outperforming the baseline, which exhibits an average return of \(-772\). This represents a \(13.13\%\) improvement. When the gravity is \(1\) in the Walker task, the BDR improves an average episodic return by \(11.38\%\)

achieving an average episodic return of 763.4 while the baseline only achieves an average return of 685.4. We also find that ADR is not effective in learning a balanced behavior over diverse environmental contexts, as it does not improve the performance over the baseline both in the Pendulum and Walker tasks. Although there is still room for further improvement in worst-case scenarios, BDR effectively improves performance in challenging domains while maintaining its average-case performance. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Task**} & \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Minimum Episodic**} & \multicolumn{2}{c}{**Mean Episodic**} & \multicolumn{1}{c}{**Mean Episodic**} \\  & & **Return** & **Return (Worst 10\%)** & **Return** \\ \hline \multirow{3}{*}{Pendulum} & Baseline & \(-1263.97\) & \(-1221.57\) & \(-392.70\pm 319.62\) \\  & ADR & \(-1312.10\) & \(-1332.14\) & \(-569.46\pm 428.23\) \\  & BDR & \(\mathbf{-1199.37}\) & \(\mathbf{-1132.30}\) & \(\mathbf{-379.14}\pm 295.16\) \\ \hline \multirow{3}{*}{Walker} & Baseline & 539.62 & 516.80 & 878.21 \(\pm\) 106.04 \\  & ADR & 224.65 & 220.40 & \(560.88\pm 198.98\) \\ \cline{1-1}  & BDR & \(\mathbf{673.49}\) & \(\mathbf{638.59}\) & \(\mathbf{885.18}\pm 88.89\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Mean and minimum episodic returns in Pendulum and Walker tasks. The best performance is marked in bold. Figure 4: Episodic returns of the baseline and BDR in the Walker task. The BDR (**b**) improves performance, particularly when the gravity is 1, where the baseline policy (**a**) struggles. Figure 3: Episodic returns of the baseline and BDR in the Pendulum task. The BDR (**b**) improves performance in domains with possibly rare dynamics, particularly when the pendulum length is 0.1, where the baseline policy (**a**) struggles. ### Analysis

Weight assignment. We also analyze how BDR adjusts the weights for training samples from different domains. Figure 5 visualizes the average sample weights assigned to each domain. The analysis indicates that BDR gives higher weights to domains with smaller lengths in the Pendulum task (Figure 4(a)) and lower gravity in the Walker task (Figure 4(b)). It demonstrates that BDR effectively targets and emphasizes rare domains where the baseline performs poorly by measuring the rarity in the embedding space. Finally, we assess the efficiency of BDR by observing performance changes throughout the training process. Figure 6 illustrates the differences in learning performance between the two methods at each training checkpoint.