# Balanced Domain Randomization for Safe Reinforcement Learning
## 4 Balanced Domain Randomization
To test the adaptability and robustness of the Reinforcement Learning (RL) agents across varied dynamic conditions, we apply domain randomization facilitated by the CARL framework [19]. In the Pendulum task, the goal is to swing a pendulum from a hanging position to an upright position. The agent controls the torque applied to the pendulum, based on its position and angular velocity. The agent is penalized for deviations from the upright position as well as control costs. We introduce variability in the environment by altering the mass and length of the pendulum. For both training and evaluation, we use combinations of mass and length values from the set {0.1, 0.5, 1.0, 1.5, 2.0}, creating a broader range of scenarios to assess the agent's adaptability. In the Walker task, the goal of a bipedal walker is to move forward efficiently while keeping balance. The agent controls the torques at its joints, based on its height, orientation, and velocity. The agent is rewarded for reaching the target velocity and maintaining balance. We alter the friction coefficients and the gravitational forces, using combinations of gravity values [1; 5; 10; 15; 20] and friction coefficients {0.1, 0.5, 1.0, 1.5, 2.0}. In this framework, the context encoder maps the history of the agent's state-action pairs into a context vector using a Multi-Layer Perceptron (MLP) network. Here, we use five recent state-action pairs as we empirically find it sufficient to infer the environmental dynamics. This MLP consists of four layers, with 256, 128, 64, and 10 neurons, respectively. The context encoder outputs a context vector \(z_{t}\), which represents a compressed embedding of the environmental dynamics. This vector is then fed into the policy network to condition the policy on the current environmental context. The context encoder is trained jointly with the contextual policy in an end-to-end manner to maximize the reward. For RL optimization, we employ the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm [48]. ### Evaluation Metric

We evaluate the performance of the agents based on the episodic return, defined as the total sum of rewards accumulated over an episode. For a comprehensive evaluation, we run \(N\) episodes and aggregate the episodic returns by calculating the average for each domain. In this work, we run 100 episodes for each domain. Our evaluation focuses on both worst-case and average-case performance, using the following metrics:

\[\text{Minimum Episodic Return}=\min_{d\in D}(R_{d})\]

The minimum episodic return metric captures the agent's performance in the most challenging domain, reflecting its robustness in difficult scenarios. \[\text{Mean Episodic Return (Worst-}k)=\frac{1}{k}\sum_{i=1}^{k}(R_{\text{sorted},i})\]

The mean episodic return of the worst-\(k\) samples is calculated by first sorting all episodic returns in ascending order, from worst to best. The worst-performing \(k\) episodes are then selected, and their mean is computed. This metric provides insight into its resilience under unfavorable conditions, as it focuses on the episodes where the agent performs the worst. \[\text{Mean Episodic Return}=\frac{1}{|D|}\sum_{d\in D}(R_{d})\]

The mean episodic return indicates the average-case performance, highlighting how well the agent adapts to the variety of environments on average. ### Results

We evaluate the performance of the Uniform Domain Randomization (UDR) baseline and Balanced Domain Randomization (BDR) by keeping their configurations identical, including model architecture and hyperparameters, with the only distinction being the weighted RL loss component. Additionally, we compare the performance with Automatic Domain Randomization (ADR) [49], which gradually expands the randomization ranges during training. ADR realizes a training curriculum that gradually expands a distribution over training environments, from the initial range concentrated on a single environment. Table 1 summarizes the quantitative results.