SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We may notice that solving the optimization problem (11) is hard because its domain is unbounded. Therefore, we make some approximations and consider the bounded optimization problem (12). First, we prove the following lemma about the value function  $J$ :
Lemma E.3.  $J(\pi_i)\leq J(\tilde{\pi}_i)$
Proof. As the domain of  $\pmb{\lambda}$  in Eq. 12 is smaller than Eq. (11), we have  $L(\pi_i, \pmb{\lambda}) \geq L(\tilde{\pi}_i, \tilde{\pmb{\lambda}})$ .
By the fundamental property of Lagrange duality, we know that  $L$  achieves its optimal value when  $\lambda = 0$  and the optimal value is  $-J(\pi_i)$ .
By the optimality of  $(\tilde{\pi}_i,\tilde{\lambda}^\star)$  , we know that
$$
- \sum_ {j = 1} ^ {i - 1} \tilde {\lambda} _ {j} ^ {\star} \left(D \left(\tilde {\pi} _ {i}, \pi_ {j}\right) - \delta\right) \geq 0. \tag {13}
$$
$$
- J (\pi_ {i}) = L (\pi_ {i}, \boldsymbol {\lambda} ^ {\star}) \geq \tilde {L} (\tilde {\pi} _ {i}, \tilde {\boldsymbol {\lambda}} ^ {\star}) = - J (\tilde {\pi} _ {i}) - \sum_ {j = 1} ^ {i - 1} \tilde {\lambda} _ {j} ^ {\star} (D (\tilde {\pi} _ {i}, \pi_ {j}) - \delta) \geq - J (\tilde {\pi} _ {i}).
$$
Then we prove the distance between optimal policy  $\tilde{\pi}_i$  in problem (12) and optimal policy  $\pi_i$  in problem (11) is very small:
Lemma E.4. Under Assumption E.1,  $D(\tilde{\pi}_i,\pi_j)\geq \delta -\epsilon_0,\forall 1\leq j <   i$
Proof. We prove this by contradiction.
Suppose there exists  $1 \leq j_0 < i$ ,  $D(\tilde{\pi}_i, \pi_{j_0}) < \delta - \epsilon_0$ . Then we choose  $\hat{\lambda}$  such that
$$
\hat {\lambda} _ {j} = \left\{ \begin{array}{l l} \Lambda & j = j _ {0}  , \\ 0 & 1 \leq j <   i, j \neq j _ {0}  . \end{array} \right.
$$
By the Assumption E.1, Eq. (13), and  $\Lambda = \frac{1}{\epsilon_0}$ , we have
$$
0 \geq - J (\pi_ {i}) = L (\pi_ {i}, \boldsymbol {\lambda} ^ {\star}) \geq L (\tilde {\pi} _ {i}, \tilde {\boldsymbol {\lambda}} ^ {\star}) \geq L (\tilde {\pi} _ {i}, \hat {\boldsymbol {\lambda}}) \geq - 1 - \Lambda (D (\tilde {\pi} _ {i}, \pi_ {j _ {0}}) - \delta) > 0.
$$
That is a contradiction. So we have proved that
$$
D \left(\tilde {\pi} _ {i}, \pi_ {j}\right) \geq \delta - \epsilon_ {0}, \quad \forall 1 \leq j <   i.
$$
From the deduction above, we get the following approximation lemma:
Lemma E.5. Denote the optimal solution of Eq. 11 and Eq. 12 as  $(\pi_i, \lambda)$  and  $(\tilde{\pi}_i, \tilde{\lambda})$  respectively. Then we have the following approximation about the optimal value and distance:
$$
J (\pi_ {i}) \leq J (\tilde {\pi} _ {i})
$$
$$
D (\tilde {\pi} _ {i}, \pi_ {j}) \geq \delta - \epsilon_ {0}, \quad \forall 1 \leq j <   i
$$
Proof. This lemma follows directly by Lemma E.3 and Lemma E.4.  $\square$
Therefore, it is reasonable to consider the constrained optimization problem (12) instead of primal problem (11) because we have proved that the optimal value doesn't get smaller and the distance of policy is  $\epsilon_0$ -approximation of the primal problem. Finally we use the conclusion in the paper [31] to analysis the convergence of problem (12):
Lemma E.6. ([31], Theorem 4.8) Under Assumption E.2, solving Eq. (12) via two-timescale GDA with learning rate  $\eta_{\pi} = \Theta (\epsilon^{4} / l^{3}\zeta^{2}\Lambda^{2})$  and  $\eta_{\lambda} = \Theta (1 / l)$  requires
$$
\mathcal {O} \left(\frac {l ^ {3} \zeta^ {2} \Lambda^ {2} C _ {1}}{\epsilon^ {6}} + \frac {l ^ {3} \Lambda^ {2} C _ {2}}{\epsilon^ {4}}\right)
$$
iterations to converge to an  $\epsilon$ -stationary point  $\pi_i^\star$ , where  $C_1$  and  $C_2$  are the constants that depend on the distance between the initial point and the optimal point.
Theorem 5.1. Under assumptions E.1 and E.2 and learning rate with learning rate  $\eta_{\pi} = \Theta (\epsilon^4 /l^3\zeta^2\Lambda^2)$  and  $\eta_{\lambda} = \Theta (1 / l)$ , SIPO converges to an  $\epsilon$ -stationary point with convergence rate  $\mathcal{O}\left(\frac{l^3\zeta^2\Lambda^2C_1}{\epsilon^6} +\frac{l^3\Lambda^2C_2}{\epsilon^4}\right)$ .
Proof. We consider the following constraint nonconvex-concave optimization:
$$
\min  _ {\pi} \max  _ {0 \leq \lambda \leq \Lambda} L (\pi , \lambda). \tag {14}
$$
Following Lemma E.6, we know that the Two-Timescale GDA algorithm converges to an  $\epsilon$ -stationary point  $\pi_i^*$ .
From the above deduction, the Two-Timescale GDA algorithm requires  $\mathcal{O}\left(\frac{l^3\zeta^2\Lambda^2C_1}{\epsilon^6} +\frac{l^3\Lambda^2C_2}{\epsilon^4}\right)$  iterations with learning rate  $\eta_{\pi} = \Theta (\epsilon^{4} / l^{3}\zeta^{2}\Lambda^{2})$  and  $\eta_{\lambda} = \Theta (1 / l)$  to converge to an  $\epsilon$ -stationary point with convergence rate.
F Discussion
F.1 The Failure Case of State-Distance-Based Diversity Measures
A failure case of state-distance-based diversity measures may be when the state space includes many irrelevant features. These features cannot reflect behavioral differences. If we run SIPO in such an environment, the learned strategies may be only diverse w.r.t these features and have little visual distinction. Like the famous noisy TV problem [5], the issue of irrelevant features is intrinsically challenging for general RL applications, which cannot be resolved by using action-based or state-occupancy-based diversity measures either.
Thanks to the advantages we discussed in the paper, we generally find that state-distance-based measures can be preferred in challenging RL problems. Meanwhile, since the state dimension can be much higher than actions, it is possible that RL optimization over states may be accordingly more difficult than actions. In practice, we can design a feature selector for those most relevant features for visual diversity and run diversity learning over the filtered features. In SMAC and GRF, we utilize the agent features (excluding enemies) as the input of diversity constraint without further modifications, as discussed in Appendix D. We remark that even after filtering, the agent features remain high-dimensional while our algorithm still works well. Note that using a feature selector is a common practice in many existing domains, such as novelty search [12], exploration [33], and curriculum learning [7]. There are also works studying how to extract useful low-dimensional features from observations [68, 18], which are orthogonal to our focus.
F.2 The Distance Metric
In Sec. 5, we adopt the two most popular implementations in the machine learning literature, i.e., RBF kernel and Wasserstein distance, while it is totally fine to adopt alternative implementations. For example, we can learn state representations (e.g. auto-encoder, Laplacian, or successor feature) and utilize pair-wise distance or norms as a diversity measure. Similar topics have been extensively discussed in the exploration literature [68, 41]. We leave them as our future directions.
G Pseudocode of SIPO
The pseudocode of SIPO is shown in Algorithm 1.
Algorithm 1 SIPO (red for SIPO-RBF and blue for SIPO-WD)  
Input: Number of Iterations  $M$ , Number of Training Steps within Each Iteration  $T$ .  
Hyperparameter: Learning Rate  $\eta_{\pi}$ , Diversity Threshold  $\delta$ , Intrinsic Scale Factor  $\alpha$ , Lagrange Multiplier Upperbound  $\lambda_{\mathrm{max}}$ , Lagrange Learning rate  $\eta_{\lambda}$ , Wasserstein Critic Learning Rate  $\eta_W$ , RBF Kernel Variance  $\sigma$ .  
1: Archived trajectories  $X \gets \emptyset$  // to store states visited by previous policies  
2: for iteration  $i = 1, \ldots, M$  do  
3: Initialize policy  $\pi_{\theta_i}$  // initialization  
4: Initialize Wasserstein critic  $f_{\phi_i}$   
5: for archive index  $j = 1, \ldots, i - 1$  do  
6: Lagrange multiplier  $\lambda_j \gets 0$   
7: end for  
8: for Training step  $t = 1, \ldots, T$  do  
9: Collect trajectory  $\tau = \{(s_h, a_h, r(s_h, a_h))\}_{h=1}^H$   
10: for archive index  $j = 1, \ldots, i - 1$  do  
11:  $R_{\mathrm{int}}^j \gets 0$   
12: end for  
13: for timestep  $h = 1, \ldots, H$  do  
14:  $r_{\mathrm{int}, h} \gets 0$  // compute intrinsic reward  
15: for archive trajectory  $\chi_j \in X$  do  
16:  $r_{\mathrm{int}, h}^j \gets -\frac{1}{H|\chi_j|} \sum_{s' \in \chi_j} \exp\left(-\frac{\|s_h - s' \|^2}{2\sigma^2}\right)$   
17:  $r_{\mathrm{int}, h}^j \gets \frac{1}{H} [f_{\phi_j}(s_h) - \frac{1}{|\chi_j|} \sum_{s' \in \chi_j} f_{\phi_j}(s')]$   
18:  $r_{\mathrm{int}, h} \gets r_{\mathrm{int}, h} + \lambda_j \cdot r_{\mathrm{int}, h}^j$   
19:  $R_{\mathrm{int}}^j \gets R_{\mathrm{int}, h}^j + r_{\mathrm{int}, h}^j$   
20: end for  
21:  $r_h \gets r(s_h, a_h) + \alpha \cdot r_{\mathrm{int}, h}$   
22: end for  
23: for archive index  $j = 1, \ldots, i - 1$  do  
24:  $\lambda_j \gets \text{clip}(\lambda_j + \eta_\lambda (-R_{\mathrm{int}}^j + \delta), 0, \lambda_{\mathrm{max}})$  // gradient ascent on  $\lambda_j$   
25:  $\phi_j \gets \phi_j + \eta_W \frac{1}{H} \sum_{h=1}^{H} \nabla_{\phi_j} (f_{\phi_j}(s_h) - \frac{1}{|\chi_j|} \sum_{s' \in \chi_j} f_{\phi_j}(s'))$   
26:  $\phi_j \gets \text{clip}(\phi_j, -0.01, 0.01)$   
27: end for  
28: Update  $\pi_{\theta_i}$  with  $\{(s_h, a_h, r_h)\}$  by PPO algorithm // policy gradient on  $\theta_i$   
29: end for  
30: Collect many trajectories  $\chi_i$  // collect trajectories to approximate  $d_{\pi_{\theta_i}}$   
31:  $X \gets X \cup \{\chi_i\}$  // for the use of following iterations  
32: end for