SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The adjacency matrix  $A$  of  $G$  is an  $|S| \times |S|$  matrix whose  $(i,j)$  entry is 1 when  $s_i$  and  $s_j$  are adjacent, and 0 otherwise. The degree matrix  $D$  is a diagonal matrix whose entry  $(i,i)$  equals the number of edges incident to  $s_i$ . The Laplacian matrix of  $G$  is defined as  $L = D - A$ . Its second smallest eigenvalue  $\lambda_2(L)$  is called the algebraic connectivity of the graph  $G$ , and the corresponding normalized eigenvector is called the Fiedler vector [60]. Last, the normalized Laplacian matrix is defined as  $\mathcal{L} = D^{-\frac{1}{2}}LD^{-\frac{1}{2}}$ .
A.2 Proof of Proposition 1
To start with, we can find a lower bound of the second term in Eq. (5) as follows:
$$
\begin{array}{l} - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} [ I (c, \tau | s _ {0}) ] = - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ - \sum_ {\tau} P (\tau | s _ {0}) \log P (\tau | s _ {0}) + \sum_ {c, \tau} P (c, \tau | s _ {0}) \log P _ {\theta} (\tau | s _ {0}, c) \right] \\ = - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ - \sum_ {c, \tau} P (c, \tau | s _ {0}) \log P (\tau | s _ {0}) + \sum_ {c, \tau} P (c, \tau | s _ {0}) \log P _ {\theta} (\tau | s _ {0}, c) \right] \\ = - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | c, s _ {0}) \log \frac {P _ {\theta} (\tau | s _ {0} , c)}{P (\tau | s _ {0})} \right] \\ = - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | c, s _ {0}) \left[ \log \frac {P _ {\theta} (\tau | s _ {0} , c)}{U n i f (\tau | s _ {0})} - \log \frac {P (\tau | s _ {0})}{U n i f (\tau | s _ {0})} \right] \right] \\ = - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \left[ \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | c, s _ {0}) \log \frac {P _ {\theta} (\tau | s _ {0} , c)}{U n i f (\tau | s _ {0})} \right] - D _ {K L} (P (\tau | s _ {0}) | | U n i f (\tau | s _ {0})) \right] \\ \geq - \beta \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | c, s _ {0}) \log \frac {P _ {\theta} (\tau | s _ {0} , c)}{U n i f (\tau | s _ {0})} \right] \\ = - \beta \underset { \begin{array}{c} s _ {0} \sim \mu (\cdot) \\ c \sim P _ {\omega} (\cdot | s _ {0}) \end{array} } {\mathbb {E}} \left[ D _ {K L} \left(P _ {\theta} (\tau | s _ {0}, c) \right\| U n i f (\tau | s _ {0}) \right] \tag {14} \\ \end{array}
$$
where  $D_{KL}(\cdot)$  denotes the Kullback-Leibler (KL) Divergence which is non-negative,  $P_{\theta}(\tau |s_0,c) = \prod_{t = 0}^{T - 1}\pi_{\theta}(a_t|s_t,c)P(s_{t + 1}|s_t,a_t)$  is the probability of the trajectory  $\tau$  given  $s_0$  and  $c$  under the option policy  $\pi_{\theta}$ , and  $Unif(\tau |s_0)$  is the probability of the same trajectory given  $s_0$  under the random walk policy. Instead of explicitly calculating  $P(\tau |s_0)$  which is impractical, we introduce  $Unif(\tau |s_0)$  to convert the second term in Eq. (5) into a regularization term to encourage exploration and diversity.
As for the first term in Eq. (5), we can deal with it as Eq. (15), where we introduce  $P_{\phi}(c|s_0,G)$  as the variational estimation of  $P(c|s_0,G)$  which is hard to acquire. The first inequality in Eq. (15) is based on the fact that KL Divergence is non-negative. While, the second inequality holds because we only keep the trajectory  $\tau$  from which  $G$  is sampled, so the trajectory and its corresponding landmark states form a bijection.
$$
\begin{array}{l} \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} [ I (c, G | s _ {0}) ] = \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ - \sum_ {c} P _ {\omega} (c | s _ {0}) \log P _ {\omega} (c | s _ {0}) + \sum_ {c, G} P _ {\omega} (c | s _ {0}) P _ {\theta} (G | s _ {0}, c) \log P (c | s _ {0}, G) \right] \\ = \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ - \sum_ {c} P _ {\omega} (c | s _ {0}) \log P _ {\omega} (c | s _ {0}) + \sum_ {c, G} P _ {\omega} (c | s _ {0}) P _ {\theta} (G | s _ {0}, c) \log \left[ P _ {\phi} (c | s _ {0}, G) \frac {P (c | s _ {0} , G)}{P _ {\phi} (c | s _ {0} , G)} \right] \right] \\ = \mathcal {H} (\mathcal {C} | \mathcal {S}) + \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, G} P _ {\omega} (c | s _ {0}) P _ {\theta} (G | s _ {0}, c) \log P _ {\phi} (c | s _ {0}, G) \right] + \sum_ {G} P (G | s _ {0}) D _ {K L} (P (c | s _ {0}, G) | | P _ {\phi} (c | s _ {0}, G)) \\ \geq \mathcal {H} (\mathcal {C} | \mathcal {S}) + \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, G} P _ {\omega} (c | s _ {0}) P _ {\theta} (G | s _ {0}, c) \log P _ {\phi} (c | s _ {0}, G) \right] \\ = \mathcal {H} (\mathcal {C} | \mathcal {S}) + \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, G} P _ {\omega} (c | s _ {0}) \left[ \sum_ {\tau^ {\prime}} P _ {\theta} \left(\tau^ {\prime} | s _ {0}, c\right) P ^ {D P P} (G | \tau^ {\prime}) \right] \log P _ {\phi} (c | s _ {0}, G) \right] \\ \geq \mathcal {H} (\mathcal {C} | \mathcal {S}) + \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \left[ \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | s _ {0}, c) P ^ {D P P} (G | \tau) \log P _ {\phi} (c | s _ {0}, G) \right] \tag {15} \\ \end{array}
$$
First, we take the gradient with respect to  $\omega$  and get the following result:
$$
\begin{array}{l} \nabla_ {\omega} \mathcal {L} = \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} [ - \sum_ {c} [ \nabla_ {\omega} P _ {\omega} (c | s _ {0}) \log P _ {\omega} (c | s _ {0}) + \nabla_ {\omega} P _ {\omega} (c | s _ {0}) ] \\ + \sum_ {c, \tau} \nabla_ {\omega} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | s _ {0}, c) P ^ {D P P} (G | \tau) \log P _ {\phi} (c | s _ {0}, G) \\ - \beta \sum_ {c} \nabla_ {\omega} P _ {\omega} (c | s _ {0}) D _ {K L} \left(P _ {\theta} (\tau | s _ {0}, c) | | U n i f (\tau | s _ {0})\right) \\ + \alpha_ {1} \sum_ {c, \tau} \nabla_ {\omega} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | s _ {0}, c) f (\tau) \tag {16} \\ - \alpha_ {2} \sum_ {c} \nabla_ {\omega} P _ {\omega} (c | s _ {0}) \sum_ {\vec {\tau} (s _ {0}, c)} P _ {\theta} (\vec {\tau} | s _ {0}, c) g (\vec {\tau} (s _ {0}, c)) \\ + \alpha_ {3} \sum_ {c} \nabla_ {\omega} P _ {\omega} (c | s _ {0}) \sum_ {\vec {\tau} (s _ {0}, c)} P _ {\theta} (\vec {\tau} | s _ {0}, c) h (\cup_ {c ^ {\prime}} \vec {\tau} (s _ {0}, c ^ {\prime})) ] \\ \end{array}
$$
Given that  $\nabla_{\omega}P_{\omega}(c|s_0) = P_{\omega}(c|s_0)\nabla_{\omega}\log P_{\omega}(c|s_0)$  and the definition of KL Divergence, i.e.,  $D_{KL}(P_{\theta}(\tau |s_0,c)||Unif(\tau |s_0)) = \sum_{\tau}P_{\theta}(\tau |s_0,c)\sum_{t = 0}^{T - 1}[\log \pi_{\theta}(a_t|s_t,c) - \log \pi_{unif}(a_t|s_t)]$ , we can simplify Eq. (16) as:
$$
\nabla_ {\omega} \mathcal {L} = \underset { \begin{array}{c} s _ {0} \sim \mu (\cdot) \\ c \sim P _ {\omega} (\cdot | s _ {0}) \end{array} } {\mathbb {E}} \left[ \nabla_ {\omega} \log P _ {\omega} (c | s _ {0}) Q ^ {P _ {\omega}} (c, s _ {0}) \right] \tag {17}
$$
where the related Q-function  $Q^{P_{\omega}}(c,s_0)$  is defined as:
$$
\begin{array}{l} Q ^ {P _ {\omega}} (c, s _ {0}) = - \log P _ {\omega} (c | s _ {0}) + \underset {\vec {\tau} _ {(s _ {0}, c)} \sim P _ {\theta} (\cdot | s _ {0}, c)} {\mathbb {E}} \left[ - \alpha_ {2} g (\overrightarrow {\tau} _ {(s _ {0}, c)}) + \alpha_ {3} h (\bigcup_ {c ^ {\prime}} \overrightarrow {\tau} _ {(s _ {0}, c ^ {\prime})} \right] \\ + \underset {\tau \sim P _ {\theta} (\cdot | s _ {0}, c)} {\mathbb {E}} \left[ P ^ {D P P} (G | \tau) \log P _ {\phi} (c | s _ {0}, G) - \beta \sum_ {t = 0} ^ {T - 1} \log \pi_ {\theta} \left(a _ {t} | s _ {t}, c\right) + \alpha_ {1} f (\tau) \right] \tag {18} \\ \end{array}
$$
Next, we calculate the gradient with respect to  $\theta$  as follows:
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} = \underset {s _ {0} \sim \mu (\cdot)} {\mathbb {E}} \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) \nabla_ {\theta} P _ {\theta} (\tau | s _ {0}, c) P ^ {D P P} (G | \tau) \log P _ {\phi} (c | s _ {0}, G) \\ - \beta \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) \nabla_ {\theta} P _ {\theta} (\tau | s _ {0}, c) \sum_ {t = 0} ^ {T - 1} \left[ \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}, c\right) - \log \pi_ {u n i f} \left(a _ {t} \mid s _ {t}\right) \right] \\ - \beta \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | s _ {0}, c) \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} | s _ {t}, c\right) \\ + \alpha_ {1} \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) \nabla_ {\theta} P _ {\theta} (\tau | s _ {0}, c) f (\tau) \\ + \sum_ {c} P _ {\omega} (c | s _ {0}) \sum_ {\vec {\tau} (s _ {0}, c)} \nabla_ {\theta} P _ {\theta} (\vec {\tau} | s _ {0}, c) \left[ - \alpha_ {2} g (\vec {\tau} (s _ {0}, c)) + \alpha_ {3} h \left(\cup_ {c ^ {\prime}} \vec {\tau} (s _ {0}, c ^ {\prime})\right) \right] \tag {19} \\ \end{array}
$$
With  $\nabla_{\theta}P_{\theta}(\tau |s_0,c) = P_{\theta}(\tau |s_0,c)\nabla_{\theta}\log P_{\theta}(\tau |s_0,c) = P_{\theta}(\tau |s_0,c)\sum_{t = 0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_t|s_t,c)$ , and  $\nabla_{\theta}P_{\theta}(\vec{\tau}|s_0,c) = P_{\theta}(\vec{\tau}|s_0,c)\sum_{m = 1}^{M}\sum_{t = 0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_t^m |s_t^m,c)$  where  $s_t^m (a_t^m)$  is the state (action) at step  $t$  in trajectory  $m$ , Eq. (19) can be written as follows:
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} = \underset {s _ {0}, c, \tau} {\mathbb {E}} \left[ \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} (a _ {t} | s _ {t}, c) \left[ P ^ {D P P} (G | \tau) \log P _ {\phi} (c | s _ {0}, G) - \beta \sum_ {t = 0} ^ {T - 1} \log \pi_ {\theta} (a _ {t} | s _ {t}, c) + \alpha_ {1} f (\tau) \right] \right] \\ + \underset {s _ {0}, c, \vec {\tau}} {\mathbb {E}} \left[ \sum_ {m = 1} ^ {M} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} ^ {m} \mid s _ {t} ^ {m}, c\right) \left[ - \alpha_ {2} g \left(\overrightarrow {\tau} _ {(s _ {0}, c)}\right) + \alpha_ {3} h \left(\cup_ {c ^ {\prime}} \overrightarrow {\tau} _ {(s _ {0}, c ^ {\prime})}\right) \right] \right] \\ = \underset {s _ {0}, c, \vec {\tau}} {\mathbb {E}} \left[ \sum_ {m = 1} ^ {M} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} ^ {m} \mid s _ {t} ^ {m}, c\right) Q _ {m} ^ {\pi_ {\theta}} (\vec {\tau}, s _ {0}, c) \right] \tag {20} \\ \end{array}
$$
where the advantage term is as Eq. (21),  $\vec{\tau} = \{\tau_1,\dots ,\tau_M\}$ $\tau_{m} = (s_{0}^{m},a_{0}^{m},\dots ,s_{T - 1}^{m},a_{T - 1}^{m},s_{T}^{m})$
$$
\begin{array}{l} Q _ {m} ^ {\pi_ {\theta}} (\vec {7}, s _ {0}, c) = \frac {P ^ {D P P} \left(G _ {m} \mid \tau_ {m}\right) \log P _ {\phi} (c \mid s _ {0} , G _ {m})}{M} - \frac {\beta}{M} \sum_ {t = 0} ^ {T - 1} \log \pi_ {\theta} \left(a _ {t} ^ {m} \mid s _ {t} ^ {m}, c\right) \tag {21} \\ + \frac {\alpha_ {1}}{M} f (\tau_ {m}) - \alpha_ {2} g (\overrightarrow {\tau} _ {(s _ {0}, c)}) + \alpha_ {3} h \left(\bigcup_ {c ^ {\prime}} \overrightarrow {\tau} _ {(s _ {0}, c ^ {\prime})}\right) \\ \end{array}
$$
Then, it's not hard to see the relationship between  $Q^{P_{\omega}}$  and  $Q_{m}^{\pi_{\theta}}$  as:
$$
Q ^ {P _ {\omega}} (c, s _ {0}) = - \log P _ {\omega} (c | s _ {0}) + \underset {\vec {\tau}} {\mathbb {E}} \left[ \sum_ {m = 1} ^ {M} Q _ {m} ^ {\pi_ {\theta}} (\vec {\tau}, s _ {0}, c) \right] \tag {22}
$$
B Comparisons with Recent Variational Option Discovery Algorithms
Here, we provide comparisons of our algorithm with more recent variational option discovery methods [30, 31, 32, 33, 34, 35].
In [30], the authors focus on alternative approaches to leverage options learned during the unsupervised phase, rather than new option discovery algorithms. They still adopt objectives based on Mutual Information (MI) as previous works without solving the exploration issue.
The authors of [31] propose a slightly-modified version of VIC to improve usefulness of the discovered options by introducing an extra posterior. Still, their options are not explicitly trained for better coverage/exploration like ours.
The authors of [32] propose to replace the fixed prior distribution  $P(c)$  with a fixed dynamics model over the option latent codes  $P(c_{t}|c_{t - 1})$ . Each latent code corresponds to a sub-trajectory. By concatenating sub-trajectories, the agent can reach much further states. They only rely on MI-based objectives, so they cannot model the coverage of each option (like ours) and instead choose to chain options for better overall coverage. The fixed  $P(c_{t}|c_{t - 1})$  can result in inflexibility when applying options in downstream tasks.
In [33], they employ a multi-step protocol to generate options organized in a tree-like structure. Heuristics and structural limits are involved in each step, which may hinder its generality. Also, they propose to optimize the local coverage around the final state rather than the overall trajectory coverage like ours.
The authors of [34] note that, with variational objectives proposed in VIC and DIAYN, the agent can be discouraged from seeking out new states, since the variational posterior  $P_{\phi}(c|s)$  is likely to make poor predictions when presented with trajectories containing previously unseen states, resulting in low rewards for the policy. Thus, an exploration bonus should be introduced as an extra reward term. In [34], they choose to train ensembles of discriminators and adopt their disagreement as a measure of state uncertainty. States of higher uncertainty are assigned with higher exploration bonus. Similar to theirs, our algorithm introduces additional bonus as a reimbursement of the unnecessary pessimistic exploration. The difference lies that they adopt an implicit exploration measure (i.e., the disagreement among ensembles) while we use explicit ones based on DPP.
In [35], they optimize the MI  $I(s, c) = \mathcal{H}(s) - \mathcal{H}(s|c)$ , where  $\mathcal{H}(s)$  is for improving exploration of the learned options. They adopt a variational posterior  $P_{\phi}(s|c)$  to estimate  $\mathcal{H}(s|c)$ . Learning  $P_{\phi}(s|c)$  can be challenging when the state space is high-dimensional, compared with  $P_{\phi}(c|s)$  used in our paper, which can hinder the optimization. Further, in DADS, they categorize algorithms utilizing  $I(s, c) = \mathcal{H}(s) - \mathcal{H}(s|c)$  as the forward form of MI-based option discovery, and they empirically and theoretically show the limited capability of these algorithms for exploration even with  $\mathcal{H}(s)$  in the objective.
C Implementation Details and Analysis of ODPP
C.1 The Choice of Diversity Measure
The expected cardinality is a better choice for the diversity measure than the likelihood shown as Eq. (1). Using the log-likelihood based on the Determinant of the DPP kernel matrix directly would heavily penalize repeated items in the sampled set  $\mathcal{W}$  in Eq. (1). For example, if there are very similar points in  $\mathcal{W}$ , the corresponding rows in the kernel matrix will be almost identical and lead to a zero determinant, which will cause numerical issues for the logarithm function. Take our work as an example: at the beginning of the training stage, the moving range of the Mujoco agent is very limited, then we always include very close states in a trajectory, which will always lead to a zero determinant and thus cannot provide training signals. However, the expected cardinality only counts the number of diverse states in a trajectory that will not be heavily influenced by the repeated items. Therefore, we select the expected cardinality as the diversity measure in this paper.
C.2 Fast Greedy MAP Inference for DPP
The maximum a posteriori (MAP) inference of DPP aims at finding the subset of items with the highest possibility under the DPP measure, which is NP-hard [48]. The log-probability function in
DPP, i.e.,  $l(W) = \log \det(L_W)$ , is submodular, which means:
$$
\forall i \in \mathcal {W}, W _ {1} \subseteq W _ {2} \subseteq \mathcal {W} \backslash \{i \}, l (W _ {1} \cup \{i \}) - l (W _ {1}) \geq l (W _ {2} \cup \{i \}) - l (W _ {2}) \tag {23}
$$
Thus, the MAP inference for DPP can be converted to a submodular maximization problem, where greedy algorithms have shown promising empirical success. Recently, the authors of [39] propose a fast greedy method for MAP inference in DPP with time complexity  $\mathcal{O}(S^2 N)$  to return  $S$  items out of a sample space of size  $N$ . The key step of their algorithm is that for each iteration, the item which maximizes the marginal gain:
$$
j = \underset {i \in \mathcal {W} \backslash W _ {m a p}} {\arg \max } l \left(W _ {m a p} \cup \{i \}\right) - l \left(W _ {m a p}\right) \tag {24}
$$
is added to  $W_{map}$  starting from an initial set  $\emptyset$ , until the maximal marginal gain becomes negative or the target sample number is reached (i.e., stopping criteria). This part is not our contribution. We provide its detailed pseudo code as Algorithm 2. For the derivation, please refer to the original paper [39]. We also provide its implementation code as a part of the complete code of ODPP.
Algorithm 2 Fast Greedy MAP Inference for DPP
1: Input: The set of items  $\mathcal{W}$  and its kernel matrix  $\mathbf{L}$ , stopping criteria
2: Initialize: For  $i \in \mathcal{W}$ ,  $\mathbf{c}_i = \left\lfloor \right\rfloor, d_i^2 = \mathbf{L}_{ii}$ ;  $W_{map} = \{j\}$ , where  $j = \arg \max_{i \in \mathcal{W}} \log(d_i^2)$
3: while stopping criteria not satisfied do
4: for  $i\in \mathcal{W}\backslash W_{map}$  do
5:  $e_i = (\mathbf{L}_{ji} - \mathbf{\bar{c}}_j,\mathbf{c}_i > ) / d_j$
6:  $\mathbf{c}_i = [\mathbf{c}_ie_i],d_i^2 = d_i^2 -e_i^2$
7: end for
8:  $j = \arg \max_{i\in \mathcal{W}\backslash W_{map}}\log (d_i^2)$ ,  $W_{map} = W_{map}\cup \{j\}$
9: end while
10: Return  $W_{\text{map}}$
C.3 Computation of the Laplacian Spectrum for the Infinite-scale State Spaces
As mentioned in Section 3.2, the feature vector of each state  $\overrightarrow{b_i}$  is defined with the eigenvectors corresponding to the  $D$ -smallest eigenvalues of the Laplacian matrix of the state transition graph. However, for the infinite-scale state spaces, we cannot obtain this Laplacian spectrum through matrix-based methods, so we adopt the NN-based method proposed in [50] for estimating the Laplacian spectrum, which has been proved to be scalable for infinite-scale state spaces and sufficiently accurate compared with the groundtruth. Since this algorithm is not our contribution, we only provide the take-away messages here for implementation.
According to [50], the  $k$  smallest eigenvalues  $\lambda_{1:k}$  and corresponding eigenvectors  $v_{1:k}$  of the Laplacian  $L$  can be estimated by:  $(k = D$  for our case)
$$
\min  _ {v _ {1}, \dots , v _ {k}} \sum_ {i = 1} ^ {k} (k - i + 1) v _ {i} ^ {T} L v _ {i}, \text {s . t .} v _ {i} ^ {T} v _ {j} = \delta_ {i j}, \forall i, j = 1, \dots , k \tag {25}
$$
For the large-scale state space, the eigenvectors can be represented as a neural network that takes a state  $s$  as input and outputs a  $k$ -dimension vector  $[f_1(s), \dots, f_k(s)]$  as an estimation of  $[v_1(s), \dots, v_k(s)]$ . Accordingly, the objective in Equation (25) can be expressed as: (please refer to [50] for details)
$$
G \left(f _ {1}, \dots , f _ {k}\right) = \frac {1}{2} \mathbb {E} _ {\left(s, s ^ {\prime}\right) \sim \mathcal {T}} \left[ \sum_ {l = 1} ^ {k} \sum_ {i = 1} ^ {l} \left(f _ {i} (s) - f _ {i} \left(s ^ {\prime}\right)\right) ^ {2} \right] \tag {26}
$$
where  $\mathcal{T}$  is a set of state-transitions collected by interacting with the environment through a random policy. Further, the orthonormal constraints in Equation (25) are implemented as a penalty term:
$$
P \left(f _ {1}, \dots , f _ {k}\right) = \alpha \mathbb {E} _ {s \sim \rho , s ^ {\prime} \sim \rho} \left[ \sum_ {l = 1} ^ {k} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {l} \left(f _ {i} (s) f _ {j} (s) - \delta_ {i j}\right) \left(f _ {i} \left(s ^ {\prime}\right) f _ {j} \left(s ^ {\prime}\right) - \delta_ {i j}\right) \right] \tag {27}
$$
where  $\alpha$  is the weight term and  $\rho$  is the distribution of states in  $\mathcal{T}$ . To sum up, the eigenfunctions  $f$  can be trained as an NN by minimizing the loss function:
$$
L \left(f _ {1}, \dots , f _ {k}\right) = G \left(f _ {1}, \dots , f _ {k}\right) + P \left(f _ {1}, \dots , f _ {k}\right) \tag {28}
$$
C.4 Analysis on Computation Complexity
The learning target of ODPP is an intra-option policy  $\pi_{\theta}(a|s,c)$  conditioned on the option choice  $c$ . As in Section 3.3, this policy is learned with an Actor-Critic algorithm for which the Q-function is defined as Eq. (13). This Q-function contains variational and DPP-based objectives. Compared with previous variational option discovery algorithms (e.g., DIAYN, VIC, VALOR), we additionally need to (a) sample landmark states from each trajectory and (b) calculate the DPP-related terms:  $f(\cdot), g(\cdot), h(\cdot)$ .
For (a), we adopt a fast greedy MAP inference algorithm for DPP. As mentioned in Appendix C.2, it takes  $\mathcal{O}(S^2 N)$  to sample  $S$  landmark states from an option trajectory of length  $N$ . In our setting,  $N = 50$ ,  $S = 10$ , so the process can be done in real-time.
For (b), we need to build DPP kernel matrices, and then compute  $f(\cdot)$ ,  $g(\cdot)$ ,  $h(\cdot)$  based on eigenvalues of the corresponding kernel matrix as in Eq. (7)-(9). The time complexity for eigen decomposition is  $\mathcal{O}(N^3)$ , where  $N$  is the size of the matrix. For the state kernel matrix,  $N$  is the number of states in an option trajectory (i.e., the option horizon) which we set as 50. For the trajectory kernel matrix,  $N$  corresponds to the number of trajectories collected in each training iteration, which is set as 100. Thus,  $f(\cdot)$ ,  $g(\cdot)$ ,  $h(\cdot)$  can be computed in real-time.
To build the kernel matrix, we need feature vectors for each state. As introduced in Appendix C.3, the feature vector is the output of a pre-trained neural network which takes the state as input. The training of this feature function is based on state transitions in the replay buffer and only needs to be done for once or twice in the whole option discovery process, of which the time cost is within 30 minutes.
To sum up, compared with previous variational methods, we additionally introduce three DPP items to explicitly model the option diversity and coverage, of which the involvement only slightly increases the time complexity.
C.5 Analysis on Scalability
ODPP can be adapted to more intricate setups encompassing longer option horizons, a greater number of skills, or visual domains. We elaborate on the scalability of ODPP as follows.
(a) The skill horizon is constrained by the MAP inference and eigen decomposition operations previously described. Given their time complexity, the skill horizon could readily be expanded from 50 to 100 or even 500. This augmentation would necessitate an additional time of  $\mathcal{O}(10^{-3})$  or  $\mathcal{O}(10^{-2})$  seconds per training iteration, compared with previous variational methods. These estimations are based on computations on a machine with a single Intel i7 CPU and four GeForce RTX 2060 GPUs. Note that a skill horizon longer than 100 is rarely necessary. Employing a skill with an excessively long horizon may compromise flexibility in decision-making.
(b) Compared with variational methods, our algorithm does not introduce extra limitations on the number of learned skills. Moreover, in Figure 5(c), we show that even when learning a large number of options at the same time (as much as 60), we can still get options with high quality (mean) and diversity (standard deviation) which increase during the training process.
(c) ODPP needs to learn the Laplacian feature embeddings. For visual domains, this process can incorporate a pretrained CNN model as a feature extractor, which serves to convert visual input into feature vectors. Subsequently, the original algorithm can be applied. Applications in visual domains could pose a common challenge for all option discovery algorithms and present an exciting avenue for future research.