SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As for the kernel matrix, according to Section 2.2, we can construct  $L(\mathcal{X})$  by defining the quality measure  $\overrightarrow{q}$  and normalized feature vector for each state in  $\mathcal{X}$ . States with higher expected returns should be visited more frequently and thus be assigned with higher quality values. However, given that there is no prior knowledge on the reward function, we assign equal quality measure to each state as 1. As for the features of each state, we define them using the Laplacian spectrum, i.e., eigenvectors corresponding to the  $D$ -smallest eigenvalues of the Laplacian matrix of the state transition graph, denoted as  $[\overrightarrow{v_1},\dots ,\overrightarrow{v_D}]$ . To be specific, for each state  $s$ , its normalized feature is defined as:  $\overrightarrow{b}(s) = [\overrightarrow{v_1}(s),\dots ,\overrightarrow{v_D}(s)] / \sqrt{\sum_{j=1}^{D}(\overrightarrow{v_j}(s))^2}$ . The reasons for this feature design are as follows: (1) As shown in Spectral Clustering [49], states with high similarity in this feature embedding fall in
Algorithm 1 Unsupervised Option Discovery based on DPP (ODPP)
1: Initialize the prior network  $P_{\omega}$ , policy network  $\pi_{\theta}$ , variational decoder  $P_{\phi}$ , and trajectory set  $\vec{\tau}$   
2: for each training episode do  
3: for  $i = 1,2,\ldots,N$  do  
4: Sample  $s_0^i \sim \mu(\cdot)$  and  $c \sim P_{\omega}(\cdot|s_0^i)$  or use the ones from the previous episode to collect trajectories subject to the same option and starting point  
5: Collect a trajectory  $\tau_i = (s_0^i, a_0^i, \dots, s_{T-1}^i, a_{T-1}^i, s_T^i)$ , where  $a_t^i \sim \pi_{\theta}(\cdot|s_t^i, c)$   
6:  $\vec{\tau} \gets \vec{\tau} \cup \{\tau_i\}$   
7: end for  
8: Update  $P_{\omega}, \pi_{\theta}$  with PPO and  $P_{\phi}$  with SGD, based on  $\vec{\tau}$  and Eq. (11)-(13);  $\vec{\tau} \gets \{\}$   
9: end for
the same cluster. Under a DPP with this feature embedding, the sampled states with distinct features should belong to different clusters, i.e., the landmarks. Then, by maximizing  $\mathcal{L}_1^{DPP}$  (i.e., the expected number of landmarks covered in  $\mathcal{X}$ ), the agent is encouraged to traverse multiple clusters within the state space. (2) With this feature design, our algorithm generalizes of the Laplacian-based option discovery [12]. In [12], they set a threshold to partition the state space into two parts - the set of states with higher values in the Fiedler vector (i.e.,  $\vec{v_2}$ ) than the threshold is used as the initiation set and the other states are used as the termination set, then the option policy is trained to connect states within these two areas. We note that, as a special case of our algorithm, when setting the feature dimension  $D = 2$ , we can get similar options with the Laplacian-based methods through maximizing  $\mathcal{L}_1^{DPP}$ . Given that the eigenvector corresponding to the smallest eigenvalue of a Laplacian matrix is  $\vec{v_1} = \vec{1}$ , states with diverse feature embeddings encoded by  $[\vec{v_1},\vec{v_2} ]$  (i.e.,  $D = 2$ ) tend to differ in  $\vec{v_2}$ . By maximizing  $\mathcal{L}_1^{DPP}$ , the options are trained to visit the states that are as distinct in  $\vec{v_2}$  as possible in a trajectory, which is similar with the ones learned in [12]. We empirically demonstrate this in Section 4. Note that the Laplacian spectrum  $[\vec{v_1},\dots ,\vec{v_D} ]$  for infinite-scale state space can be learned as a neural network through representation learning [50] which is introduced in Appendix C.3. Thus, our algorithm can be applied to tasks with large-scale state spaces.
Next, we expect the set of sampled trajectories related to the same option  $c$  and starting from the same state  $s_0$ , i.e.,  $\overrightarrow{\tau}_{(s_0,c)}$ , to be consistent and thus hard to be distinguished by a DPP, which is important given the stochasticity of the policy output. This is realized by minimizing the expected mode number in each  $\overrightarrow{\tau}_{(s_0,c)}$ :  $(s_0 \sim \mu(\cdot), c \sim P_{\omega}(\cdot | s_0)$ ,  $\overrightarrow{\tau}_{(s_0,c)} \sim P_{\theta}(\cdot | s_0, c))$
$$
\mathcal {L} _ {2} ^ {D P P} (\omega , \theta) = \underset {s _ {0}, c, \vec {\tau} _ {(s _ {0}, c)}} {\mathbb {E}} \left[ g \left(\vec {\tau} _ {(s _ {0}, c)}\right) \right], g \left(\vec {\tau} _ {(s _ {0}, c)}\right) = \underset {\mathbf {Y} \sim P _ {L (y)}} {\mathbb {E}} [ | \mathbf {Y} | ] = \sum_ {i = 1} ^ {M} \frac {\lambda_ {i} ^ {y}}{\lambda_ {i} ^ {y} + 1} \tag {8}
$$
where  $\mathcal{V}$  is the set of  $M$  trajectories related to  $c$  starting at  $s_0$  (i.e.,  $\{\tau_1,\dots ,\tau_M\} = \overrightarrow{\tau}_{(s_0,c)})$ ,  $L(\mathcal{V})$  is the kernel matrix built with the feature vectors of each trajectory, i.e.,  $\overrightarrow{b(\tau_i)}$ . The feature vector of a trajectory can be obtained based on the ones of its landmark states  $G_{i}$  through the Structured DPP framework [51]:  $\overrightarrow{b(\tau_i)} = \sum_{s\in G_i}\overrightarrow{b(s)}$ .  $(\overrightarrow{b(s)}$  is defined above.) Alternatively, we can use the hidden layer output of the decoder  $P_{\phi}(c|G,s_0)$ , which embeds the information contained in  $G$ . This design is commonly adopted in DPP-related works [52, 53]. We will compare these two choices in Section 4. Notably, our kernel matrix design, i.e.,  $L(\mathcal{X})$  and  $L(\mathcal{Y})$ , is task-irrelevant and domain-knowledge-free.
Last, the set of sampled trajectories subject to different options should be diverse. This is achieved by maximizing its expected mode number (i.e., Eq. (9)). Here,  $\mathcal{Z}$  is the union set of  $\mathcal{V}$  related to different options, i.e.,  $\cup_{c'} \vec{\tau}_{(s_0, c')}$ , and  $L(\mathcal{Z})$  is the kernel matrix corresponding to trajectories in  $\mathcal{Z}$ .
$$
\mathcal {L} _ {3} ^ {D P P} (\omega , \theta) = \underset {s _ {0}, c, \overrightarrow {\vec {\tau}} _ {(s _ {0}, c)}} {\mathbb {E}} \left[ h \left(\bigcup_ {c ^ {\prime}} \overrightarrow {\vec {\tau}} _ {(s _ {0}, c ^ {\prime})}\right) \right], h \left(\bigcup_ {c ^ {\prime}} \overrightarrow {\vec {\tau}} _ {(s _ {0}, c ^ {\prime})}\right) = \underset {\mathbf {Z} \sim P _ {L (\mathcal {Z})}} {\mathbb {E}} [ | \mathbf {Z} | ] = \sum_ {i = 1} ^ {K} \frac {\lambda_ {i} ^ {\mathcal {Z}}}{\lambda_ {i} ^ {\mathcal {Z}} + 1} \tag {9}
$$
3.3 Overall Algorithm Framework
The overall objective is to maximize Eq. (10). The hyperparameters  $\alpha_{1:3} \geq 0$  (provided in Appendix C.6) are the weights for each DPP term and can be fine-tuned to enable a tradeoff between coverage and diversity of the learned options.
$$
\mathcal {L} (\omega , \theta , \phi) = \mathcal {L} ^ {I B} (\omega , \theta , \phi) + \alpha_ {1} \mathcal {L} _ {1} ^ {D P P} (\omega , \theta) - \alpha_ {2} \mathcal {L} _ {2} ^ {D P P} (\omega , \theta) + \alpha_ {3} \mathcal {L} _ {3} ^ {D P P} (\omega , \theta) \tag {10}
$$
Based on Eq. (10), we can calculate the gradients with respect to  $\omega, \theta, \phi$ , i.e., the parameters of the prior network, policy network and variational decoder, and then apply corresponding algorithms for optimization. First,  $\nabla_{\phi}\mathcal{L} = \nabla_{\phi}\mathcal{L}^{IB}$ , so  $P_{\phi}$  can be optimized as a standard likelihood maximization problem with SGD [54]. Next, regarding  $\omega$  and  $\theta$ , we have Proposition 2, which is proved in Appendix A.3. Notably, we select PPO [55] to update  $\omega$  and  $\theta$ . The overall algorithm (ODPP) is summarized as Algorithm 1, where the main learning outcome is the intra-option policy  $\pi_{\theta}$ . When applied to downstream tasks,  $\pi_{\theta}(a|s,c)$  can be fixed and we only need to learn a high-level policy  $P_{\psi}(c|s)$  to select among options. We note that our algorithm is highly salable and only slightly increases the time complexity compared with previous algorithms in this field. Detailed discussion on the complexity and scalability are provided in Appendix C.4 and C.5.
Proposition 2. The gradients of the overall objective  $\mathcal{L}(\omega, \theta, \phi)$  with respect to  $\omega$  and  $\theta$  can be unbiasedly estimated using Eq. (11). Here,  $Q^{P_{\omega}}$  and  $Q_{m}^{\pi_{\theta}}$  are the  $Q$ -functions for the prior and policy networks, respectively. Consequently, both networks can be trained using reinforcement learning.
$$
\nabla_ {\omega} \mathcal {L} = \underset {s _ {0}, c} {\mathbb {E}} \left[ \nabla_ {\omega} \log P _ {\omega} (c | s _ {0}) Q ^ {P _ {\omega}} \right], \nabla_ {\theta} \mathcal {L} = \underset {s _ {0}, c, \overrightarrow {\tau}} {\mathbb {E}} \left[ \sum_ {m = 1} ^ {M} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} ^ {m} | s _ {t} ^ {m}, c\right) Q _ {m} ^ {\pi_ {\theta}} \right] \tag {11}
$$
$$
\begin{array}{l} Q ^ {P _ {\omega}} (c, s _ {0}) = - \log P _ {\omega} (c | s _ {0}) + \underset {\vec {\tau}} {\mathbb {E}} \left[ \sum_ {m = 1} ^ {M} Q _ {m} ^ {\pi_ {\theta}} (\vec {\tau}, s _ {0}, c) \right] (12) \\ Q _ {m} ^ {\pi_ {\theta}} (\vec {\tau}, c, s _ {0}) = \frac {P ^ {D P P} \left(G _ {m} \mid \tau_ {m}\right) \log P _ {\phi} (c \mid s _ {0} , G _ {m})}{M} - \frac {\beta}{M} \sum_ {t = 0} ^ {T - 1} \log \pi_ {\theta} \left(a _ {t} ^ {m} \mid s _ {t} ^ {m}, c\right) (13) \\ + \frac {\alpha_ {1}}{M} f (\tau_ {m}) - \alpha_ {2} g (\overrightarrow {\tau} _ {(s _ {0}, c)}) + \alpha_ {3} h \left(\bigcup_ {c ^ {\prime}} \overrightarrow {\tau} _ {(s _ {0}, c ^ {\prime})}\right) \\ \end{array}
$$
4 Evaluation and Main Results
In this section, we compare ODPP with SOTA baselines on a series of RL tasks. (1) For intuitive visualization, we test these algorithms on maze tasks built with Mujoco [56]. In particular, we select the Point and Ant as training agents, and put them in complex 3D Mujoco Maze environments (Figure 4(a) and 4(d)). We evaluate the diversity and coverage of the options learned with different algorithms, through visualizing corresponding trajectories. Then, we provide a quantitative study to see if these options can aid learning in downstream tasks – goal-achieving or exploration tasks in the maze environments. Both tasks are long-horizon with an episode length of 500. (2) To show the applicability of our algorithm on a wide range of tasks, we test it on 3D Ant locomotion tasks (Figure 5(a)) and Atari video games. In this part, we focus on evaluating if the agent can learn effective skills without supervision of task-specific rewards.
As mentioned in Section 2.1, our algorithm follows the option-first variational option discovery, which starts from a random policy rather than an efficient exploration policy like the trajectory-first methods. To keep it fair, we compare our algorithm with several SOTA option-first methods: VIC [7], DIAYN [8], VALOR [27], DADS [9], and APS [57]. Further, our algorithm integrates the variational and Laplacian-based option discovery, so we compare it with a SOTA Laplacian-based method as well: DCO [12]. The codes are available at https://github.com/LucasCJYSDL/ODPP.
In Figure 2, we visualize trajectories of options learned with different algorithms in the Point Room task, which start from the same initial state and have the same horizon (i.e., 50 steps). Note that the visualizations in Figure 1-3 are aerial views of the 3D Mujoco Room/Corridor. As mentioned in Section 3.2, the objective of DCO is to train options that can connect states with high and low values in the Fielder vector of the state transition graph. Due to its algorithm design [12], we can only learn one option with DCO at a time. While, with the others, we can learn multiple options simultaneously. (1) From (a)-(c), we can see that variational option discovery methods can discover diverse options, but these options can hardly approach the "bottleneck" states in the environment which restricts their coverage of the state space. On the contrary, in (d), options trained with DCO can go through two "bottleneck" states but lack diversity, since they stick to the direction given by the Fiedler vector. While, as shown in (h), options learnt with our algorithm have both superior diversity
Figure 2: (a) VIC, (b) DIAYN, (c) VALOR, (d) DCO, (e) ODPP  $(\mathcal{L}^{IB})$ , (f) ODPP  $(\mathcal{L}^{IB},\mathcal{L}_1^{DPP})$ , (g) ODPP using trajectory features defined by hidden layer output, (h) ODPP using trajectory features defined with the Structured DPP framework.
and coverage. (2) In (e), we can already get significant better options than the baselines by only using  $\mathcal{L}^{IB}$  as the objective. From (e) to (f), it can be observed that additionally introducing the objective term  $\mathcal{L}_1^{DPP}$  can encourage options to cover more landmark states. From (f) to (h), we further add  $\mathcal{L}_2^{DPP}$  and  $\mathcal{L}_3^{DPP}$ , which makes the option trajectories more distinguishable from the view of DPP. Also, in (g) and (h), we adopt different designs of the trajectory feature. It shows that using trajectory features defined with the Structured DPP framework (introduced in Section 3.2) is better than using the hidden layer output of the decoder  $P_{\phi}$ . These results show the effectiveness of each component in our algorithm design and demonstrate that our algorithm can construct options with higher diversity and coverage than baselines. In Appendix D.3, we provide quantitative results to further show the improvement brought by each objective term of ODPP.
Next, as claimed in Section 3.2, if setting the feature dimension  $D = 2$ , ODPP is expected to learn similar options with DCO, thus ODPP generalizes the Laplacian-based method. In Figure 3, we visualize the value of each state in the Fiedler vector as the background color (the darker the higher), and the options learned with DCO and ODPP in the Point Corridor task starting from different locations. We can observe from (a) and (b) that most of the learned options are similar both in direction and
length. Further, if we adopt the normal setting, where  $D = 30$  and the number of options to learn at a time is 10, we can get diverse options shown as (c), which can be beneficial for various downstream tasks. In this case, ODPP can be viewed as a generalization and extension of DCO through variational tools (e.g.,  $\mathcal{L}^{IB}$ ) to learn multiple diverse options at a time.
(a) DCO
(b) ODPP  $(D = 2)$
(c) ODPP
Figure 3: (a) Options learned with DCO; (b) Options learned with ODPP, when setting the feature dimension as 2; (c) Options learned with ODPP in the normal setting.
4.2 Evaluation in Downstream Tasks
In Figure 4, we evaluate the options learned with different algorithms on a series of downstream tasks. These options are trained without task-specific rewards, and thus potentially applicable to different downstream tasks in the environment. Firstly, we test the options in Point Room/Corridor goal-achieving tasks where a point agent is trained to achieve a certain goal (i.e., red points in Figure 4(a) and 4(d)). This task is quite challenging since: (1) The location of the goal is not included in the observation. (2) The agent will get a positive reward only when it reaches the goal area; otherwise, it will receive a penalty. Hence, the reward setting is highly sparse and delayed, and the agent needs to fully explore the environment for the goal state. In Figure 4(b)-4(c), we compare the mean and standard deviation of the performance (i.e., mean return of a decision step) of different algorithms
(a) Mujoco Room
(b) Point Room Task
(c) Point Corridor Task
(d) Mujoco Corridor
(e) Ant Room Task
(f) Ant Corridor Task
Figure 4: (a)(d) Mujoco Maze tasks. (b)(c) Applying the options to goal-achieving tasks in the Point Room/Corridor where the agent needs to achieve one of the four goals (red points). (e)(f) Applying options to exploration tasks in the Ant Room/Corridor where the agent needs to explore as far as possible to get higher reward.
in the training process, which is repeated four times (each time with a different goal). It can be observed that, with options learned by ODPP, the convergence speed and return value can be much higher. Note that the pretrained options are fixed for downstream tasks and we only need to learn an option selector  $P_{\psi}(c|s)$  which gives out option choice  $c$  at state  $s$ . In this way, we can simplify a continuous control task to a discrete task, and the advantage can be shown through the comparison with using PPO (i.e., "Base RL") directly on the task. To keep it fair, the PPO agent is pretrained for the same number of episodes as the option learning. Moreover, we evaluate these algorithms in Ant Room/Corridor exploration tasks where an Ant agent is trained to explore the areas as far from the start point (center) as possible. The reward for a trajectory is defined with the largest distance that the agent has ever reached during this training episode. In Figure 4(e)-4(f), we present the change of the trajectory return during the training process of these algorithms (repeated five times with different random seeds). The options trained with ODPP provide a good initialization of the policy for this exploration task. With this policy, the Ant agent can explore a much larger area in the state space.
As mentioned in Section 3, we learn a prior network  $P_{\omega}$  together with the option policy network  $\pi_{\theta}$ . This design is different from previous algorithms which choose to fix the prior distribution. In Appendix D.1, we provide a detailed discussion on this and empirically show that we can get a further performance improvement in the downstream task by initializing the option selector  $P_{\psi}$  with  $P_{\omega}$ .
4.3 Performance of Unsupervised Skill Discovery Across Various Benchmarks
In the 3D Locomotion task, an Ant agent needs to coordinate its four legs to move. In Figure 5(a), we visualize two of the controlling behaviors, learned by ODPP without supervision of any extrinsic rewards. Please refer to Appendix D.2 for more visualizations. The picture above shows that the Ant rolls to the right by running on Leg 1 first and then Leg 4, which is a more speedy way to move ahead. While, the picture below shows that it learns how to walk to the right by stepping on its front legs (2&3) and back legs (1&4) by turn. These complex behaviors are learned with only intrinsic objectives on diversity and coverage, which is quite meaningful. Further, we show numeric results to evaluate the learned behaviors. We use the reward defined in [58] as the metric, which is designed to encourage the Ant agent to move as fast as possible at the least control cost. In Figure 5(b), we take the average of five different random seeds, and the result shows that options trained by ODPP outperform the baselines. The reward drops during training for some baselines, which is reasonable since this is not the reward function used for option learning. Finally, to see whether we can learn a large number of options in the meantime with ODPP, we test the performance of the discovered options when setting the number of options to learn as 10, 20, 40, 60. From Figure 5(c), it can be observed that even when learning a large number of options at the same time, we can still get options with high quality (mean) and diversity (standard deviation) which increase during the training.
Figure 5: (a) Visualization of the controlling behaviors learned by ODPP. (b) The change of the mean and standard deviation of the trajectory rewards subject to different options in the training process. (c) The performance change as the number of options to learn at the same time goes up.
(b) RiverRaid
(c) AirRaid
Figure 6: Evaluation results on OpenAI Gym and Atari games to show the superiority of our algorithm on more general tasks. Our algorithm performs the best in all the three tasks, and the performance improvement becomes more significant as the task difficulty increases.
To further demonstrate the applicability of ODPP, we compare it with SOTA baselines on more general OpenAI Gym and Atari tasks [59]. Notably, we adopt two more advanced skill discovery algorithms as baselines: DADS and APS. Comparisons among these two and previous baselines are in Appendix D.4. Skills discovered with different algorithms are evaluated with reward functions carefully-crafted for each task, provided by OpenAI Gym. For each algorithm, we learn 10 skills, of which the average cumulative rewards (i.e., sum of rewards within a skill duration) in the training process are shown in Figure 6. The skill duration is set as 100 for CartPole and 50 for the other two. Note that the complete episode horizon is 200 for CartPole and 10000 for AirRaid and RiverRaid. Thus, it would be unfair to compare the cumulative reward of a skill with the one of a whole episode. Our algorithm performs the best in all the three tasks, and the improvement becomes more significant as the task difficulty increases. When relying solely on MI-based objectives, the agent tends to reinforce already discovered behaviors for improved diversity rather than exploration. The explicit incorporation of coverage and diversity objectives in our algorithm proves beneficial in this case.
ODPP is a novel unsupervised option discovery framework based on DPP, which unifies variational and Laplacian-based option discovery methods. Building upon the information-theoretic objectives in prior variational research, we propose three DPP-related measures to explicitly quantify and optimize diversity and coverage of the discovered options. Through a novel design of the DPP kernel matrix based on the Laplacian spectrum of the state transition graph, ODPP generalizes SOTA Laplacian-based option discovery algorithms. We demonstrate the superior performance of ODPP over SOTA baselines using variational and Laplacian-based methods on a series of challenging benchmarks. Regarding limitations of ODPP, the primary one is to assign suitable weights for each objective term:  $\alpha_{1:3}$  in Eq. (10) and  $\beta$  in Eq. (6). These hyperparameters are crucial, yet conducting a grid search for their joint selection would be exhaustive. Instead, we employ a sequential, greedy method to select hyperparameters in accordance with our ablation study's procedure, further detailed in Appendix C.6.
This project is supported by ONR award N00014-23-1-2850 and CISCO research award 76934189.
[59] Hosu, I., T. Rebedea. Playing atari games with deep reinforcement learning and human checkpoint replay. CoRR, abs/1607.05077, 2016.
[60] Fiedler, M. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal, 23(2):298-305, 1973.
A Notations and Proof
A.1 Basic Concepts and Notations
Markov Decision Process (MDP): The reinforcement learning problem can be described with an MDP, denoted by  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0,1]$  is the state transition function,  $\mathcal{R}: \mathcal{S} \times \mathcal{A} \to R^1$  is the reward function, and  $\gamma \in (0,1]$  is the discount factor.
State transition graph in an MDP: The state transitions in  $\mathcal{M}$  can be modelled as a state transition graph  $G = (V_G, E_G)$ , where  $V_G$  is a set of vertices representing the states in  $\mathcal{S}$ , and  $E_G$  is a set of undirected edges representing state adjacency in  $\mathcal{M}$ . We note that:
Remark. There is an edge between state  $s$  and  $s'$  (i.e.,  $s$  and  $s'$  are adjacent) if and only if  $\exists a \in \mathcal{A}$ , s.t.  $\mathcal{P}(s, a, s') > 0 \lor \mathcal{P}(s', a, s) > 0$ .