SlowGuess/NeurIPS_2023_OCR huggingface dataset:
RGB Images in Locomotion Tasks We run SIPO-WD in the visual Humanoid task based on Isaac Gym [42]. The training protocol is similar to the state-only version (i.e., the input of policy and intrinsic rewards are both locomotion states of the Humanoid) except that we stack recent 4 RGB camera observations  $(84 \times 84)$  as the input of intrinsic rewards in Eq. 9. We adopt the training code developed in Isaac Gym and the default PPO configuration. The backbone of the discriminator is composed of 4 convolutional layers with kernel size 3, stride 2, padding 1, and [16, 32, 32, 32] channels. Then the feature is passed to an MLP with 1 hidden layer and 256 hidden units. The activation function is leaky ReLU with slope 0.2. We also compute the pairwise distance of joint torques as in the state-only version and show the result in Table 12. Visualizations are shown in Fig. 10. SIPO-WD can also learn meaningful diverse behaviors with RGB images as the state input thanks to the learnable Wasserstein discriminator. This implies that our algorithm can be naturally extended to high-dimensional states and incorporated with advances in representation learning, which may be a potential future direction.
B.4.2 Combining State- and Action-based Diversity Measures
Based on SIPO-RBF, we introduce additional action information by directly concatenating the global state, used for diversity calculation, with the one-hot encoded actions of all agents within the GRF domain. Table 13 presents the outcomes, indicating the number of policies obtained. For scenarios with a limited number of agents, the action-augmented variant demonstrates comparable performance. However, when the agent count increases (as evident in the 11-agent cases of CA and corner), the incorporation of actions can introduce misleading diversity, detracting from the authenticity of the outcomes.
B.5 How to Adjust Constraint-Related Hyperparameters
Three hyperparameters are essential in the implementation of the intrinsic reward  $r_{\mathrm{int}}$ : the threshold  $\delta$ , the intrinsic reward scale factor  $\alpha$ , and the variance factor  $\sigma$  in  $r_{\mathrm{int}}^{\mathrm{RBF}}$ . These parameters differ under different domains and must be adjusted individually. We find proper parameters by running two
(a) normal running
(b) left-leg jumping
(c) right-leg jumping
(d) hand-balanced mincing
Figure 10: Results of SIPO-WD in the visual Humanoid task.
Table 12: Pairwise distance of joint torques (i.e., diversity score) in Humanoid with visual input. Results in visual experiments are averaged over 3 seeds.
SIPO-WD (visual)	SIPO-WD	RSPO (best baseline)
0.62 (0.26)	0.71 (0.23)	0.53 (0.05)
3v1	CA	Corner
SIPO-RBF	3.0 (0.8)	3.3 (0.5)	2.7 (0.5)
SIPO-RBF w. Action	3.0 (0.0)	2.3 (0.5)	1.0 (0.0)
Table 13: Ablation study of combining state- and -action-based diversity measures. The number of different strategies across a population of 4 is shown with standard deviation in the brackets.
iterations without constraints and get two similar policies  $\pi_0$  and  $\pi_{1}$ . We record  $r_{\mathrm{int}}$  during training  $\pi_{1}$  and the trend is shown in Fig. 11. Not surprisingly,  $r_{\mathrm{int}}$  gradually decreases as training proceeds.
Threshold We set  $\delta = c_{1}D_{S}(\pi_{0},\pi_{1})$ . We try several different  $c_{1}\in \{1,1.2,1.4,1.6,1.8,2.0\}$  and find that  $c_{1} = 1.2$  or 1.4 are universal proper solutions for all the experimental environments.
Intrinsic Scale Factor We need to balance the intrinsic reward  $r_{int}$  and the original reward  $J$  so that neither of the two rewards can dominate the training process. Empirically, the maximums of the two rewards should be in the same order of magnitude. i.e.,  $\max_{\pi} J(\pi) = \alpha \times c_2 \lambda_{max} \delta$ , where  $c_2 = O(1)$ . When  $c_2$  is too large, the new-trained policy  $\pi_j$  will oscillate near the boundary of  $D(\pi_i, \pi_j) = \delta$  for some pre-trained policy  $p_i$ . Conversely, when  $c_2$  is too small, the intrinsic reward  $r_{int}$  cannot yield diverse strategies. In experiments, we set  $c_2 = 1.0$ .
Variance Factor We sweep the variance factor across  $\{1e - 3,5e - 3,1e - 2,2e - 2,1e - 3\}$  by training  $\pi_{1}$  and observe the trend of intrinsic rewards. We find the steepest trend and select the corresponding  $\sigma$ . Empirically, we find that our algorithm performs robustly well when  $\sigma^2 = 0.02$ .
The  $\delta$  and  $\alpha$  of GRF and SMAC are listed in Table 14.
B.6 Computation of Action-Based Measures in the Grid-World Example
We consider the policies illustrated in Fig. 12. These policies are all optimal since these actions only include "right" and "down" and actions on non-visited states can be arbitrary. We only mark actions on states visited by any of these 3 policies and actions on other states can be considered the same.
B.6.1 Action-Distribution-Based Measures
Action-distribution-based diversity measures can be defined as
$$
D _ {\mathcal {A}} \left(\pi_ {i}, \pi_ {j}\right) = \mathbb {E} _ {s \sim q (s)} \left[ \tilde {D} \left(\pi_ {i} (\cdot | s) \| \pi_ {j} (\cdot | s)\right) \right], \tag {10}
$$
where  $\tilde{D}(\cdot, \cdot): \triangle \times \triangle \to \mathbb{R}$  is a measure over action distributions and  $q: \triangle(\mathcal{S})$  is a state distribution. Here, we consider  $q$  to be the joint state distribution visited by  $\pi_i$  and  $\pi_j$ .
Figure 11: Average intrinsic reward during training  $\pi_1$ .
Table 14: The values of  $\delta$  and  $\alpha$  in different environments.
football	smac
3v1	corner	CA	2m_vs_1z	2c_vs_64zg
δWD	0.004	0.01	0.012	0.02	0.2
αWD	1	1	0.5	0.5	0.05
δRBF	0.03	0.01	0.015	0.002	0.001
αRBF	0.001	0.001	0.001	0.001	0.001
σ²	0.02	0.02	0.02	0.02	0.02
KL Divergence KL divergence is defined by
$$
D _ {\mathrm {K L}} \left(\pi_ {i} (\cdot \mid s), \pi_ {j} (\cdot \mid s)\right) = \int_ {\mathcal {A}} \pi_ {i} (a \mid s) \log \frac {\pi_ {i} (a \mid s)}{\pi_ {j} (a \mid s)} \mathrm {d} a.
$$
When  $\pi_j(a\mid s) = 0$  at any state  $s$ , KL divergence is  $+\infty$ . Since the trajectories of these policies have disjoint states,  $D_{\mathcal{A}}^{\mathrm{KL}}(\pi_1,\pi_2) = D_{\mathcal{A}}^{\mathrm{KL}}(\pi_1,\pi_3) = +\infty$ . Similar results can be obtained for cross-entropy.
$\mathbf{JSD}_{\gamma}$ $\mathrm{JSD}_{\gamma}$  was defined in [38] and we consider two special cases when  $\gamma = 0$  and  $\gamma = 1$
As illustrated by [38],  $\mathrm{JSD}_0$  measures the expected number of times two policies will "disagree" by selecting different actions. On trajectories induced by  $\pi_1$  and  $\pi_2$ , there are  $4 + 4$  states that  $\pi_1$  disagrees with  $\pi_2$  ( $\pi_1$  and  $\pi_2$  are symmetric) and  $D_{\mathcal{A}}^{\mathrm{JSD}_0}(\pi_1, \pi_2) = 8/16 = 1/2$ . Similarly,  $\pi_1$  and  $\pi_3$  only disagree at the initial state, therefore we have  $D_{\mathcal{A}}^{\mathrm{JSD}_0}(\pi_1, \pi_3) = 2/16 = 1/8$ .
$\mathrm{JSD}_1$  is defined by
$$
\begin{array}{l} \mathrm {J S D} _ {1} (\pi_ {i}, \pi_ {j}) = - \frac {1}{2} \sum_ {\tau_ {i}} P (\tau_ {i} \mid \pi_ {i}) \sum_ {t = 1} ^ {T} \frac {1}{T} \log \frac {\pi_ {i} (\tau_ {i}) + \pi_ {j} (\tau_ {i})}{2 \pi_ {i} (\tau_ {i})} \\ - \frac {1}{2} \sum_ {\tau_ {j}} P (\tau_ {j} \mid \pi_ {j}) \sum_ {t = 1} ^ {T} \frac {1}{T} \log \frac {\pi_ {i} (\tau_ {j}) + \pi_ {j} (\tau_ {j})}{2 \pi_ {j} (\tau_ {j})}. \\ \end{array}
$$
Since each of the policies considered only induces a single trajectory and  $\pi_i(\tau_j) = 0 (i \neq j)$ , we can easily compute
$$
D _ {\mathcal {A}} ^ {\mathrm {J S D} _ {1}} \left(\pi_ {1}, \pi_ {2}\right) = D _ {\mathcal {A}} ^ {\mathrm {J S D} _ {1}} \left(\pi_ {1}, \pi_ {3}\right) = \log 2
$$
Wasserstein Distance Wasserstein distance or Earth Moving Distance (EMD) is 1 if two policies disagree on a state and 0 otherwise. Therefore, it equals to  $D_{\mathcal{A}}^{\mathrm{JSD}_0}$ .
B.6.2 Action Norm
We embed the action "right" as vector  $[1,0]$  since it increases the x-coordinate by 1 and the action "down" as vector  $[0,-1]$  since it decreases the y-coordinate by 1. This embedding can be naturally extended to a continuous action space with velocity actions. Following [52], we compute the action norm over a uniform distribution on states. We can see that there are 7 states where  $\pi_1$  and  $\pi_2$  perform differently and 1 state (the initial state) where  $\pi_1$  and  $\pi_3$  perform differently. Therefore, we can get  $D(\pi_1,\pi_2) = \sqrt{7}$  and  $D(\pi_1,\pi_3) = 1$ .
B.6.3 State-Distance-Based Measures
State  $L_{2}$  Norm Similar to action  $L_{2}$  norm, we concatenate the coordinates instead of actions as the embedding and compute the  $L_{2}$  norm between embedding.
π1
π2
π3
Figure 12: Policies in the grid-world example when  $N_G = 5$ .
Wasserstein Distance Wasserstein distance is tractable in the grid-world example. We consider 7 states (except the initial and final states) in each trajectory and compute the pair-wise distance as matrix  $C$ . Then we solve the following linear programming
$$
\min  _ {\gamma} \quad \sum_ {i, j} \gamma \odot C
$$
$$
\begin{array}{l l} \text {s . t .} & \gamma \mathbf {1} = a, \gamma^ {T} \mathbf {1} = b \end{array}
$$
$$
\gamma_ {i, j} \geq 0
$$
where  $\odot$  means element-wise multiplication,  $\mathbf{1}$  is a all-one vector,  $a = [\mathbf{1}^T,\mathbf{0}^T]^T$  and  $b = [\mathbf{0}^T,\mathbf{1}^T]^T$  is the marginal state distribution of each policy.
C.1 Details of the 2D Navigation Environment
The navigation environment has an agent circle with size  $a$  and 4 landmark circles with size  $b$ . We pre-specified a threshold  $c$  and constrain that the distance of final states reaching different landmarks must be larger than  $c$ . Correspondingly, landmark circles are randomly initialized by constraining the pairwise distance between centers to be larger than a threshold  $c + 2(a + b)$  such that the final-state constraint is valid. An episode ends if the agent touches any landmarks, i.e., the distance between the center of the agent and the center of the landmark  $d < a + b$ , or 1000 timesteps have elapsed. The observation space includes the positions of the agent and all landmarks, which is a 10-dimensional vector. The action space is a 2-dimensional vector, which is the agent velocity. The time interval is set to be  $\Delta t = 0.1$ , i.e., the next position is computed by  $x_{t+1} = x_t + \Delta t \cdot v$ . The reward is 1 if the agent touches the landmark and 0 otherwise.
C.2 Details of Environments
We provide training configurations and environment introductions below and refer readers to our project website in App. A for visualizations of these environments.
Humanoid We use the Humanoid environment in IsaacGym [42] with default observation and action spaces. The input of intrinsic rewards or diversity measure is the observation without all torque states.
SMAC We adopt the SMAC environment in the MAPPO codebase<sup>2</sup> with the same configuration as Yu et al. [69]. The input of intrinsic rewards or diversity measure is the state of all allies, including positions, health, etc.
On the "easy" map  $2m\_vs\_1z$ , two marines must be controlled to defeat a Zealot. The marines can attack from a distance, while the Zealot's attacks are limited to close range. A successful strategy involves alternating the marines' attacks to distract the Zealot. On the "hard" map  $2c\_vs\_64zg$ , two colossi must be controlled by the agents to fight against 64 zergs. The colossi have a wider attack range and can move over cliffs. Strategies on this map may include hit-and-run tactics, waiting in corners, or dividing and conquering enemies. The level of difficulty is determined by the learning performance of existing MARL algorithms. Harder maps require more exploration and training steps.
GRF We adopt the "simple115v2" representation as observation with both "scoring" and "checkpoint" reward. The reward is shared across all agents. The input of intrinsic rewards or diversity measure is the position and velocity of all attackers and the ball. All policies are trained to control the left team to score against built-in bots.
academy_3_vs_1_withKeeper: In this scenario, a team of three players (left) tries to score a goal against a single defender and a goalkeeper. The left team starts with the ball and has to dribble past the defender and the goalkeeper to score a goal.
$^{2}$ https://github.com/marlbenchmark/on-policy
Table 15: Hyperparameters in the 2D navigation environment.
discount	GAE λ	PPO epochs	clip parameter	entropy bonus	λmax	actor lr	critic lr	Lagrange lr	batch size
0.997	0.95	10	0.2	0	10	3e-4	1e-3	0.5	4000
Table 16: Common hyperparameters for SIPO, baselines, and ablations.
discount	GAE λ	actor lr	critic lr	clip parameter	entropy bonus	GRF batch size	SMAC batch size
0.99	0.95	5e-4	1e-3	0.2	0.01	9600	3200
academy counterattackeasy: In this scenario, the left team starts with the ball in the front yard and tries to score a goal against several defenders. All eleven players in the left players can be controlled.
academy-corner: In this scenario, the left team tries to score a goal from a corner kick. The right team defends the goal and tries to prevent the left team from scoring. All eleven players in the left players can be controlled.
D.1 2D Navigation
We apply PPO with Lagrange multipliers to optimize the policy and hyperparameters are summarized in Table 15.  $D(\pi_i, \pi_j)$  is simply taken as the  $L_2$  distance of the final state reached by  $\pi_i$  and  $\pi_j$ , i.e.,  $D(\pi_i, \pi_j) = \| s_H^{\pi_i} - s_H^{\pi_j} \|^2$ . The applied algorithm is the same as SIPO (see Appendix G) except that the intrinsic reward is only computed at the last timestep.
D.2 SIPO
In the  $i$ -th iteration  $(1 \leq i \leq M)$ , we learn an actor and a critic with  $i$  separate value heads to accurately predict different return terms, including  $i - 1$  intrinsic returns for the diversity constraints and the environment reward. We include all practical tricks mentioned in [69] because we find them all critical to algorithm performance. We use separate actor and critic networks, both with hidden size 64 and a GRU layer with hidden size 64. The common hyperparameters for SIPO, baselines, and ablations are listed in Table 16. Other environment-specific parameters, such as PPO epochs and mini-batch size, are all the same as [69]. Besides, Table 14 and Table 17 lists some extra hyperparameters for SIPO.
D.3Baselines
We re-implement all baselines with PPO based on the MAPPO [69] project. All algorithms run for the same number of environment frames. Specific hyperparameters for baselines can be found in Appendix D.3.
SMERL SMERL trains a latent-conditioned policy that can robustly adapt to new scenarios. It promotes diversity by maximizing the mutual information between states and the latent variable. We implement SMERL with PPO, where the actor and the critic take as the input the concatenation of observation and a one-hot latent variable. The discriminator is a 2-layer feed-forward network with 64 hidden units. The learning rate of the discriminator is the same as the learning rate of the critic network. The input of the discriminator is the same as the input we use for SIPO-WD. The critic has 2 value heads for an accurate estimation of intrinsic return. Since SMERL trains a
Table 17: SIPO hyperparameters across all environments.
λmax	Discriminator lr	Lagrangian lr
10	4.0e-4	0.1
single latent-conditioned policy, we train SMERL for  $M \times$  more environment steps, such that total environment frames are the same. The scaling factor of intrinsic rewards is 0.1 and the threshold for diversification is [0.81, 0.45, 0.72] (0.9 × [0.9, 0.5, 0.8]) for "3v1", "counterattack", and "corner" respectively.
DvD DvD simultaneously trains a population of policies to maximize the determinant of a kernel matrix based on action difference. We concatenate the one-hot actions along a trajectory as the behavioral embedding. The square of the variance factor, i.e.,  $\sigma^2$  in the RBF kernel, is set to be the length of behavioral embedding. We also use the same Bayesian bandits as proposed in the original paper. Training DvD in "counterattack" and "corner" exceeds the GPU memory and we exclude the results in the main body.
DIPG DIPG iteratively maximizes the maximum mean discrepancy (MMD) distance between the state distribution of the current policy and previously discovered policies. For DIPG, we follow the open-source implementation<sup>3</sup>. We set the same variance factor in the RBF kernel as SIPO-RBF and apply the same state as the input of the RBF kernel. We sweep the coefficient of MMD loss among  $\{0.1, 0.5, 0.9\}$  and find 0.1 the most appropriate (larger value will cause training instability). We use the same method to save archived trajectories as SIPO and the input of the RBF kernel is the same as the input we use for SIPO-RBF. To improve training efficiency, we only back-propagate the MMD loss at the first PPO epoch.
RSPO RSPO iteratively discovers diverse policies by optimizing extrinsic rewards on novel trajectories while optimizing diversity on other trajectories. The diversity measure is defined as the action-cross entropy along the trajectory. For RSPO, we follow the opensource implementation<sup>4</sup> and use the same hyperparameters on the SMAC  $2c\_ vs\_ 64\mathrm{zg}$  map in the original paper for GRF experiments.
TrajDi TrajDi was originally designed for cooperative multi-agent domains to facilitate zero-shot coordination. It defines a generalized Jensen-Shanon divergence objective between policy action distributions. Then this objective and rewards are simultaneously optimized via population-based training. We tried TrajDi in SMAC and GRF. We sweep the action discount factor among  $\{0.1, 0.5, 0.9\}$  and the coefficient of TrajDi loss among  $\{0.1, 0.01, 0.001\}$ . However, TrajDi fails to converge in the "3v1" scenario and exceeds the GPU memory in the "counterattack" and "corner" scenarios. Therefore, we exclude the performance of TrajDi in the main body.
Domino We have meticulously re-implemented Domino within our codebase according to the appendix of Zahavy et al. [71]. We execute the algorithm in the Humanoid locomotion task, employing the robot state (excluding torques) for successor feature computation. Despite our earnest efforts to optimize Domino's performance, our findings reveal its comparable performance to SMERL, illustrated by a minimal diversity score of 0.01. Therefore, we exclude the performance of Domino in the main body.
APT APT maximized the nearest neighbor state-entropy estimation for skill discovery. While we also adopted this metric for diversity evaluation, there is a fundamental distinction in formulation. APT optimizes state entropy within a single policy, whereas our method, SIPO, targets the joint entropy of a population of policies. It is okay for each single policy within the population to have low state entropy. To employ APT's objective of discovering diverse policies, training a population of agents concurrently is required. The algorithm should optimize the estimated entropy over states visited by all policies. Yet, this approach mandates large-scale k-NN computation  $(k = 12)$  over substantial batches, leading to significant computational inefficiency. Despite our dedicated efforts, we didn't finish a single training trial of APT within 48 hours (in contrast to other PBT baselines, e.g. DvD, which completes training in less than 8 hours).
D.4 Ablation Study Details
For the three ablation studies: fix-L, CE, and filter, we list the specific hyperparameters here:
<sup>3</sup>https://github.com/dtak/DIPG-public
4https://github.com/footoredo/rspo-iclr-2022
- fix-L: we set the Lagrange multiplier to be 0.2;
- CE: the threshold is 3.800 and the intrinsic reward scale factor is  $1 / 1000$  of that in the WD setting;
- filter: all the hyperparameters in the setting are the same as those in the WD setting.
E.1 Proof of theorem 4.1
Theorem 4.1. Assume  $D$  is a distance metric. Denote the optimal value of Problem 3 as  $T_{1}$ . Let  $T_{2} = \sum_{i=1}^{M} J(\tilde{\pi}_{i})$  where
$$
\tilde {\pi} _ {i} = \operatorname * {a r g   m a x} _ {\pi_ {i}} J (\pi_ {i})
$$
$$
s. t. \quad D \left(\pi_ {i}, \tilde {\pi} _ {j}\right) \geq \delta / 2, \quad \forall 1 \leq j <   i \tag {3}
$$
Proof. Suppose the optimal solution of Problem 3 is  $\pi_1, \pi_2, \ldots, \pi_M$  satisfying  $J(\pi_1) \geq J(\pi_2) \geq \ldots \geq J(\pi_M)$  and the optimal solution of Problem 6 is  $\tilde{\pi}_1, \tilde{\pi}_2, \ldots, \tilde{\pi}_M$  satisfying  $J(\tilde{\pi}_1) \geq J(\tilde{\pi}_2) \geq \ldots \geq J(\tilde{\pi}_M)$ .
Assume the contrary that Thm. 4.1 is not true, which means  $\sum_{i=1}^{M} J(\pi_i) = T_1 > T_2 = \sum_{i=1}^{M} J(\tilde{\pi}_i)$ . Then we choose the smallest number  $N \leq M$  that satisfies
$$
\sum_ {i = 1} ^ {N} J \left(\pi_ {i}\right) > \sum_ {i = 1} ^ {N} J \left(\tilde {\pi} _ {i}\right).
$$
By  $T_{1} > T_{2}$  we know that  $N$  exists. In addition, because Problem 6 solves unconstrained RL in the first iteration, we know that  $\tilde{\pi}_{1} = \arg \max_{\pi} J(\pi)$  and then  $J(\pi_{1}) \leq J(\tilde{\pi}_{1})$ . Therefore,  $N \geq 2$ .
Suppose  $J(\pi_N) \leq J(\tilde{\pi}_N)$ . Then we have
$$
\sum_ {i = 1} ^ {N - 1} J \left(\pi_ {i}\right) > \sum_ {i = 1} ^ {N - 1} J \left(\tilde {\pi} _ {i}\right).
$$
Contradicting the fact that  $N$  is the smallest number satisfies that equation.
Hence, we know that  $J(\pi_N) > J(\tilde{\pi}_N)$ . Then
$$
J (\pi_ {1}) \geq J (\pi_ {2}) \geq \dots \geq J (\pi_ {N}) > J (\tilde {\pi} _ {N}).
$$
Consider the optimization problem of  $\tilde{\pi}_N$ :
$$
\tilde {\pi} _ {N} = \underset {\pi} {\arg \max } J (\pi)
$$
$$
\begin{array}{l l} \text {s . t .} & D (\pi , \tilde {\pi} _ {j}) \geq \delta / 2, \quad \forall 1 \leq j <   N. \end{array}
$$
This optimization does not find  $\{\pi_1,\dots ,\pi_N\}$  but find  $\tilde{\pi}_N$ , which means that for each  $\pi_i$ ,  $1\leq i\leq N$ , there exists  $1\leq j_{i} < N$  such that  $D(\pi_i,\tilde{\pi}_{j_i}) < \delta /2$ . Otherwise, we will get the solution of the above problem as  $\pi_{i}$  instead of  $\tilde{\pi}_N$ .
By the Pigeonhole Principle, we know that there exist two indexes  $i_1 \in [N]$  and  $i_2 \in [N]$  ( $i_1 \neq i_2$ ) such that  $j_{i_1} = j_{i_2} = \hat{j}$ . Then we have
$$
D \left(\pi_ {i _ {1}}, \pi_ {i _ {2}}\right) \leq D \left(\pi_ {i _ {1}}, \tilde {\pi} _ {\hat {j}}\right) + D \left(\pi_ {i _ {2}}, \tilde {\pi} _ {\hat {j}}\right) <   \delta / 2 + \delta / 2 = \delta ,
$$
where the inequality follows by the triangle inequality of the distance function.
It contradicts with the fact that  $D(\pi_{i_1}, \pi_{i_2}) \geq \delta$  in Problem 3.
Therefore, we prove the theorem  $\sum_{i=1}^{M} J(\pi_i) = T_1 \leq T_2 = \sum_{i=1}^{M} J(\tilde{\pi}_i)$ .
E.2 Proof of Theorem 5.1
In this section, we consider the  $i$ -th iteration of SIPO illustrated in Eq. (4). For the sake of simplicity, we use  $a \leq \lambda \leq b$  for vector  $\lambda$  to denote each component of  $\lambda$  satisfies  $a \leq \lambda_i \leq b$ , where  $a, b \in \mathbb{R}$ . We use  $\pi$  to denote the policy we are optimizing, and  $\pi_j$  ( $1 \leq j < i$ ) to denote a previously obtained policy. We denote the Lagrange function as  $L(\pi, \lambda) = -J(\pi) - \sum_{j=1}^{i-1} \lambda_j(D(\pi, \pi_j) - \delta)$ .
To prove Theorem 5.1, we consider the following two optimization problems:
$$
\left(\pi_ {i}, \boldsymbol {\lambda} ^ {\star}\right) = \arg \min  _ {\pi} \max  _ {\boldsymbol {\lambda} \geq 0} L (\pi , \boldsymbol {\lambda}) \tag {11}
$$
$$
\left(\tilde {\pi} _ {i}, \tilde {\lambda} ^ {\star}\right) = \arg \min  _ {\pi} \max  _ {0 \leq \lambda \leq \Lambda} L (\pi , \lambda), \tag {12}
$$
where  $\Lambda = \frac{1}{\epsilon_0}$  and  $\epsilon_0 > 0$  is sufficiently small.
We make the following assumptions to prove this theorem:
Assumption E.1.  $0 \leq J(\cdot) \leq 1$ .
Assumption E.2.  $\forall \lambda \geq 0$ ,  $L(\cdot, \lambda)$  is  $l$ -smooth and  $\zeta$ -Lipschitz.