SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiayu Chen
Purdue University  
st Lafayette, IN 47907  
en3686@purdue.edu
Vaneet Aggarwal
Purdue University  
West Lafayette, IN 47907  
vaneet@purdue.edu
Tian Lan
The George Washington University Washington, DC 20052 tlan@gwu.edu
Learning rich skills under the option framework without supervision of external rewards is at the frontier of reinforcement learning research. Existing works mainly fall into two distinctive categories: variational option discovery that maximizes the diversity of the options through a mutual information loss (while ignoring coverage) and Laplacian-based methods that focus on improving the coverage of options by increasing connectivity of the state space (while ignoring diversity). In this paper, we show that diversity and coverage in unsupervised option discovery can indeed be unified under the same mathematical framework. To be specific, we explicitly quantify the diversity and coverage of the learned options through a novel use of Determinantal Point Process (DPP) and optimize these objectives to discover options with both superior diversity and coverage. Our proposed algorithm, ODPP, has undergone extensive evaluation on challenging tasks created with Mujoco and Atari. The results demonstrate that our algorithm outperforms state-of-the-art baselines in both diversity- and coverage-driven categories.
Reinforcement Learning (RL) has achieved impressive performance in a variety of scenarios, such as games [1, 2], robotic control [3, 4], and transportation [5, 6]. However, most of its applications rely on carefully-crafted, task-specific rewards to drive exploration and learning, limiting its use in real-life scenarios often with sparse or no rewards. To this end, utilizing unsupervised option discovery – acquiring rich skills without supervision of environmental rewards by building temporal action-sequence abstractions (denoted as options), to support efficient learning can be essential. The acquired skills are not specific to a single task and thus can be utilized to solve multiple downstream tasks by implementing a corresponding meta-controller that operates hierarchically on these skills.
Existing unsupervised option discovery approaches broadly fall into two categories: (1) Variational option discovery, e.g., [7, 8, 9], which aims to improve diversity of discovered options by maximizing the mutual information [10] between the options and trajectories they generate. It tends to reinforce already discovered behaviors for improved diversity rather than exploring (e.g., visiting poorly-connected states) to discover new ones, so the learned options may have limited coverage of the state space. (2) Laplacian-based option discovery, e.g., [11, 12], which clusters the state space using a Laplacian spectrum embedding of the state transition graph, and then learns options to connect different clusters. This approach is shown to improve the algebraic connectivity of the state space [13] and reduce expected covering time during exploration. However, the discovered options focus on improving connectivity between certain clusters and thus could be homogeneous and lack diversity. Note that coverage in this paper is defined as a property with respect to a single option, which can be measured as the number of state clusters traversed by an option trajectory. By maximizing coverage
Figure 1: Illustrative example. (a) Options from variational methods have good diversity but poor (single-option) coverage of the corridors. (b) Each option from Laplacian-based methods aims to improve its coverage by visiting poorly-connected corner states but tends to be homogeneous as the others. (c) Options from our proposed method have both superior diversity and coverage. (d) The coverage and diversity measure of the options in (a)-(c) defined with DPP (i.e., Eq.(7) and (9)).
of each single option and diversity among different options in the meanwhile, the overall span of all options can be maximized. However, diversity and coverage may not go hand-in-hand in option discovery, as visualized in Figure 1(a)(b). Attempts such as [14, 15] have been made to address this gap, but they rely on expert datasets that contain diverse trajectories spanning the whole state spaces and lack an analytical framework for diversity and coverage of the discovered options.
This paper introduces a novel framework for unsupervised option discovery by utilizing Determinantal Point Process (DPP) to quantify and optimize both diversity and (single-option) coverage of the learned options. A DPP establishes a probability measure for subsets of a set of items. The expected cardinality, representing the average size of random subsets drawn according to the DPP, serves as a diversity measure, as it reflects the likelihood of sampling diverse items and the number of distinct items in the set. First, to enhance option diversity, we apply a DPP to the set of trajectories for different options. Maximizing the expected cardinality under this DPP encourages agents to explore diverse trajectories under various options. Second, we create another DPP for the set of visited states in a trajectory and maximize its expected cardinality. This prompts the agent to visit distant states from distinct state clusters through each trajectory, leading to higher single-option coverage. As shown in Figure 1(d), these objectives effectively measure diversity and coverage of the options. Lastly, to establish the option-policy mapping and so learn multiple options simultaneously, we maximize the mutual information between the option choice and related trajectories, as in variational option discovery. Rather than using the whole trajectory [16] or only the goal state [7], our solution extract critical landmark states from a trajectory via a maximum a posteriori (MAP) inference of a DPP, allowing noise mitigation while retaining vital information. In conclusion, our proposed framework unifies diversity and coverage in option discovery using a mathematical modeling tool—DPP. Our key contributions are as follows. (1) To the best of our knowledge, this is the first work to adopt DPP for option discovery. (2) The proposed unified framework enables explicit maximization of option diversity and coverage, capturing advantages of both variational and Laplacian-based methods. (3) Empirical results on a series of challenging RL tasks demonstrate the superiority of our algorithm over state-of-the-art (SOTA) baselines.
2.1 Unsupervised Option Discovery
As proposed in [17], the option framework consists of three components: an intra-option policy  $\pi : S \times \mathcal{A} \to [0,1]$ , a termination condition  $\beta : S \to \{0,1\}$ , and an initiation set  $I \subseteq S$ . An option  $< I, \pi, \beta >$  is available in state  $s$  if and only if  $s \in I$ . If the option is taken, actions are selected according to  $\pi$  until it terminates according to  $\beta$  (i.e.,  $\beta = 1$ ). The option framework enables learning and planning at multiple temporal levels and has been widely adopted in RL. Multiple research areas centered on this framework have been developed. Unsupervised Option Discovery aims at discovering skills that are diverse and efficient for downstream task learning without supervision from rewards, for which algorithms have been proposed for both single-agent and multi-agent scenarios [18, 19, 20, 21]. Hierarchical Reinforcement Learning [22, 23] and Hierarchical Imitation Learning [24, 25, 26], on the other hand, aim at directly learning a hierarchical policy incorporated with skills, either from interactions with the environment or expert demonstrations.
We focus on single-agent unsupervised option discovery. One primary research direction in this field is variational option discovery, which aims to learn a latent-conditioned option policy  $\pi(a|s,c)$ , where  $c$  represents the option latent. Diverse options can be learned by maximizing the mutual information between the option choice  $c$  and the trajectory generated by the policy conditioned on  $c$ . As shown in [27], this objective is equivalent to a Variational Autoencoder (VAE) [28]. Variational option discovery can be categorized into two groups based on the structure of the VAE used: (1) Trajectory-first methods, such as EDL [14] and OPAL [15], follow the structure  $\tau \xrightarrow{E} c \xrightarrow{D} \tau$ . Here,  $\tau$  represents the agent's trajectory, while  $D$  and  $E$  denote the VAE's decoder and encoder, respectively. In this case, the policy network  $\pi(a|s,c)$  serves as the decoder  $D$ . The effectiveness of these methods rely on the quality of the expert trajectory set used as the encoder's input. For instance, OPAL employed for Offline RL [29] assumes access to a trajectory dataset generated by a mix of diverse policies starting from various initial states. Similarly, in EDL, highly-efficient exploration strategies such as State Marginal Matching [16] are utilized to simulate perfect exploration and obtain a set of trajectories ( $\tau$ ) with good state space coverage for option learning. (2) Option-first methods, such as VIC [7], DIAYN [8], VALOR [27], and DADS [9], follow the structure  $c \xrightarrow{E} \tau \xrightarrow{D} c$  and learn  $\pi(a|s,c)$  as the encoder  $E$ . Unlike EDL and OPAL learning from "exploration experts", these methods start from a random policy to discover diverse options. However, due to challenges in exploration, they fail to expand the set of states visited by the random policy, leading to poor state space coverage. For instance, in Figure 1(a), though trajectories of different options are diverse, none of them explore the corridor. There have been more notable advancements in variational option discovery methods, as cited in [30, 31, 32, 33, 34, 35]. A detailed comparison between these methods and our algorithm, highlighting our contributions, can be found in Appendix B.
In this paper, we tackle a more challenging scenario where the agent must learn to identify diverse options and thoroughly explore the environment, starting from a random policy. This approach supports self-driven learning and exploration, rather than relying on expert policy/trajectories like trajectory-first methods. We achieve this by integrating the variational method with another key branch in unsupervised option discovery—Laplacian-based option discovery [11, 12]. This approach is based on the Laplacian spectrum of the state transition graph, which can be estimated using state transitions in the replay buffer. The state transition graph and its Laplacian matrix are formally defined in Appendix A.1. Specifically, the approach presented in [12] first estimates the Fiedler vector – the eigenvector associated with the second smallest eigenvalue of the Laplacian. Options are then trained to connect states with high and low values in the Fiedler vector. These states are loosely connected by "bottleneck" states. By connecting them with options, the algebraic connectivity of the state space is enhanced, and thus the exploration can be accelerated [13]. However, these options may be homogeneous. For instance, in Figure 1(b), multiple options are found to explore the right corridor, but they all follow the same direction given by the Fiedler vector. Inspired by variational and Laplacian-based methods, we propose a new option discovery framework that unifies the optimization of diversity and coverage, through a novel use of DPP. As shown in Figure 1(c), multiple diverse options are discovered, most of which traverse the "bottleneck" states to enhance coverage.
2.2 Determinantal Point Process
According to [36], given a set of items  $\mathcal{W} = \{w_1, \dots, w_N\}$ , a point process  $\mathcal{P}$  on  $\mathcal{W}$  is a probability measure on the set of all the subsets of  $\mathcal{W}$ .  $\mathcal{P}$  is called a Determinantal Point Process (DPP) if a random subset  $\mathbf{W}$  drawn according to  $\mathcal{P}$  has probability:
$$
\mathcal {P} _ {L (\mathcal {W})} (\mathbf {W} = W) = \frac {\det (L _ {W})}{\sum_ {W ^ {\prime} \subseteq \mathcal {W}} \det (L _ {W ^ {\prime}})} = \frac {\det (L _ {W})}{\det (L + I)} \tag {1}
$$
$I \in \mathbb{R}^{N \times N}$  is the identity matrix.  $L = L(\mathcal{W}) \in \mathbb{R}^{N \times N}$  is the DPP kernel matrix, which should be symmetric and positive semidefinite.  $L_{W} \in \mathbb{R}^{|W| \times |W|}$  is the sub-matrix of  $L$  indexed by elements in  $W$ . Specifically,  $P_{L}(\mathbf{W} = \{w_{i}\}) \propto L_{ii}$  and  $P_{L}(\mathbf{W} = \{w_{i}, w_{j}\}) \propto L_{ii}L_{jj} - L_{ij}^{2}$  where  $L_{ij}$  measures the similarity between item  $i$  and  $j$ . Since the inclusion of one item reduces the probability of including similar items, sets consisting of diverse items are more likely to be sampled by a DPP.
The DPP kernel matrix  $L$  can be constructed as a Gram Matrix [37]:  $L = \widetilde{B}^T\widetilde{B} = \text{Diag}(\overrightarrow{q})\cdot B^T B\cdot \text{Diag}(\overrightarrow{q})$ .  $\overrightarrow{q} = [q_1,\dots ,q_N]\in \mathbb{R}^N$  with  $q_{i}\geq 0$  denotes the quality measure.  $B = \left[\vec{b}_{1}\dots \vec{b}_{N}\right]\in \mathbb{R}^{D\times N}$  is the stacked feature matrix where  $\vec{b_i}\in \mathbb{R}^D$  is the feature vector corresponding to  $w_{i}$ . The
inner product of feature vectors, e.g.,  $\overrightarrow{b_i}^T\overrightarrow{b_j}$ , is used as the similarity measure between items in  $\mathcal{W}$ . From Eq. (1), we can see that  $\mathcal{P}_L(\mathbf{W} = W)$  is proportional to the squared  $|W|$ -dimension volume of the parallelepiped spanned by the columns of  $\widetilde{B}$  corresponding to the elements in  $W$ . Diverse sets have feature vectors that are more orthogonal and span larger volumes, making them more probable.
The expected cardinality of samples from a DPP is an effective measure of the diversity of  $\mathcal{W}$  and reflects the number of modes in  $\mathcal{W}$  [38]. We provide detailed reasons why we choose the expected cardinality instead of the likelihood in Eq. (1) as the diversity measure in Appendix C.1. According to [36], the expected cardinality of a set sampled from a DPP can be calculated with Eq. (2), where  $\lambda_i^{\mathcal{W}}$  is the  $i$ -th eigenvalue of  $L(\mathcal{W})$ . DPPs have found applications across a wide array of domains to promote diversity due to their unique ability to model diverse subsets, such as information retrieval [39, 40], computer vision [38], natural language processing [41, 42], and reinforcement learning [43, 44, 45]. This motivates us to employ DPPs in skill discovery for diversity enhancement.
$$
\underset {\mathbf {W} \sim P _ {L (\mathcal {W})}} {\mathbb {E}} [ | \mathbf {W} | ] = \sum_ {i = 1} ^ {N} \frac {\lambda_ {i} ^ {\mathcal {W}}}{\lambda_ {i} ^ {\mathcal {W}} + 1} \tag {2}
$$
3 Proposed Approach
In this section, we propose a DPP-based framework that unifies the variational and Laplacian-based option discovery to get options with both superior diversity and coverage. As discussed in Section 2.1, defining an option requires relating it to its initial states  $s_0$ , specifying its termination condition, and learning the intra-option policy. Our algorithm learns a prior network  $P_{\omega}(c|s_0)$  to determine the option choice  $c \in \mathcal{C}$  at an initial state, and an intra-option policy network  $\pi_{\theta}(a|s,c)$  to interact with the environment using a certain option for a fixed number of steps (i.e., the termination condition). With  $P_{\omega}$  and  $\pi_{\theta}$ , we can collect trajectories  $\tau = (s_0,a_0,\dots ,s_T)$  corresponding to different options.
In Section 3.1, we propose  $\mathcal{L}^{IB}$ , a lower bound for the mutual information between the option and the landmark states in its trajectory, inspired by variational option discovery. By introducing the conditional variable  $c$  and maximizing  $\mathcal{L}^{IB}$ , we can establish the option-policy mapping and learn multiple options simultaneously. Each option can generate specific trajectories. However, the mutual information objective only implicitly measures option diversity as the difficulty of distinguishing them via the variational decoder, and does not model the coverage. Thus, in Section 3.2, we additionally introduce explicit coverage and diversity measures based on DPP as objectives. In particular,  $\mathcal{L}_1^{DPP}$  measures single-agent coverage as the number of landmark states traversed by an option trajectory. Maximizing this metric encourages each option to cover a broader area in the state space. Notably,  $\mathcal{L}_1^{DPP}$  generalizes the Laplacian-based option discovery objectives by employing the Laplacian spectrum as the feature vector to construct the DPP kernel matrix.  $\mathcal{L}_2^{DPP}$  measures the diversity among trajectories of the same option. By minimizing it, the consistency these trajectories can be enhanced.  $\mathcal{L}_3^{DPP}$  measures the diversity among trajectories corresponding to different options, which should be maximized to improve the diversity among various options. The rationale behind introducing  $\mathcal{L}_{1:3}^{DPP}$  is to maximize both the coverage of each individual option and the diversity among various options. By doing so, we can maximize the overall span of all options, thereby fostering the formation of diverse skills that fully explore the state space.
3.1 Landmark-based Mutual Information Maximization
Previous works tried to improve the diversity of learned options by maximizing the mutual information between the option choice  $c$  and trajectory  $\tau$  [27] or goal state  $s_{T}$  [46] generated by the corresponding intra-option policy. However, the whole trajectory contains noise and the goal state only cannot sufficiently represent the option policy. In this paper, we propose to maximize the mutual information between  $c$  and landmark states  $G$  in  $\tau$  instead.  $G$  is a set of distinct, representative states. Specifically, after clustering all states according to their features, a diverse set of landmarks with varied feature embeddings is identified to represent each different cluster. Notably,  $G$  can be extracted from  $\tau$  through the maximum a posteriori (MAP) inference of a DPP [36], shown as Eq. (3) where  $\mathcal{X}$  denotes the set of states in  $\tau$ . The intuition is that these landmarks should constitute the most diverse subset of  $\mathcal{X}$  and thus should be the most probable under this DPP.
$$
G = \underset {X \subseteq \mathcal {X}} {\arg \max } \mathcal {P} _ {L (\mathcal {X})} (\mathbf {X} = X) = \underset {X \subseteq \mathcal {X}} {\arg \max } \mathcal {P} (\mathbf {X} = X | \mathbf {L} = L (\mathcal {X})) = \underset {X \subseteq \mathcal {X}} {\arg \max } \frac {\det  (L _ {X})}{\det  (L + I)} \tag {3}
$$
In order to maximize the mutual information between  $G$  and  $c$  while filtering out the redundant information for option discovery in  $\tau$ . According to the Information Bottleneck framework [47], this can be realized through Eq. (4), where  $\mu(\cdot)$  is the initial state distribution,  $I(\cdot)$  denotes the mutual information, and  $I_{ct}$  is the information constraint.
$$
\max  _ {\theta , \omega} \mathbb {E} _ {s _ {0} \sim \mu (\cdot)} I (c, G | s _ {0}; \theta , \omega), \text {s . t .}, I (c, \tau | s _ {0}; \theta , \omega) \leq I _ {c t} \tag {4}
$$
Equivalently, with the introduction of a Lagrange multiplier  $\beta \geq 0$ , we can optimize:
$$
\max  _ {\theta , \omega} \mathbb {E} _ {s _ {0} \sim \mu (\cdot)} [ I (c, G | s _ {0}; \theta , \omega) - \beta I (c, \tau | s _ {0}; \theta , \omega) ] \tag {5}
$$
It is infeasible to directly compute and optimize Eq. (5), so we have the following proposition.
Proposition 1. The optimization problem as Eq. (5) can be solved by maximizing  $\mathcal{L}^{IB}(\omega, \theta, \phi)$ :
$$
\mathcal {H} (\mathcal {C} | \mathcal {S}) + \underset {s _ {0}, c, \tau} {\mathbb {E}} \left[ P ^ {D P P} (G | \tau) \log P _ {\phi} (c | s _ {0}, G) \right] - \beta \underset {s _ {0}, c} {\mathbb {E}} \left[ D _ {K L} \left(P _ {\theta} (\tau | s _ {0}, c) \mid \mid U n i f (\tau | s _ {0})\right) \right] \tag {6}
$$
Here,  $s_0 \sim \mu(\cdot), c \sim P_{\omega}(\cdot | s_0), \tau \sim P_{\theta}(\cdot | s_0, c)$ .  $\mathcal{H}(\mathcal{C}|\mathcal{S})$  represents the entropy associated with the option choice distribution at a given state.  $P^{DPP}(G|\tau) = \operatorname*{det}(L_G) / \operatorname*{det}(L + I)$  is the probability of extracting  $G$  from  $\tau$ , under a DPP on the set of states in  $\tau$ .  $P_{\phi}(c|s_0, G)$  serves as a variational estimation of the posterior term  $P(c|s_0, G)$  in  $I(c, G|s_0)$ . Unif(τ|s₀) denotes the probability of sampling trajectory  $\tau$  given  $s_0$  under a uniformly random walk policy.
The derivation can be found in Appendix A.2. (1) The first two terms of Eq. (6) constitute a lower bound of the first term in Eq. (5).  $\mathcal{H}(\mathcal{C}|S)$  and  $P_{\phi}(c|s_0,G)$  can be estimated using the output from the prior network and variational posterior network, respectively. (2) The third term in Eq. (6) corresponds to a lower bound of the second term in Eq. (5). Instead of directly calculating  $-\beta I(c,\tau |s_0)$  which is implausible, we introduce Unif  $(\tau |s_0)$  to convert it to a regularization term as in Eq. (6). Specifically, by minimizing  $D_{KL}(P_{\theta}(\tau |s_0,c)||Unif(\tau |s_0))$  , which is the KL Divergence [10] between the distribution of trajectories under our policy and a random walk policy, exploration of the trained policy  $\pi_{\theta}$  can be improved. Note that  $P_{\theta}(\tau |s_0,c) = \prod_{t = 0}^{T - 1}\pi_{\theta}(a_t|s_t,c)P(s_{t + 1}|s_t,a_t)$  where  $P(s_{t + 1}|s_t,a_t)$  is the transition function in the MDP.
Another challenge in calculating  $\mathcal{L}^{IB}$  is to infer the landmarks  $G$  with Eq. (3). ( $L$  is the kernel matrix of the DPP, of which the construction is further introduced in the next section.) The MAP inference problem related to a DPP is NP-hard [48], and greedy algorithms have shown to be promising as a solution. In this work, we adopt the fast greedy method proposed in [39] for the MAP inference, which is further introduced in Appendix C.2 with its pseudo code and complexity analysis.
3.2 Quantifying Diversity and Coverage via DPP
In this section, we propose three optimization terms defined with DPP to explicitly model the coverage and diversity of the learned options. Jointly optimizing these terms with Eq. (6), the discovered options are expected to exhibit improved state space coverage and enhanced diversity.
The first term relates to improving the coverage of each option by maximizing:
$$
\mathcal {L} _ {1} ^ {D P P} (\omega , \theta) = \underset {s _ {0}} {\mathbb {E}} \left[ \sum_ {c, \tau} P _ {\omega} (c | s _ {0}) P _ {\theta} (\tau | s _ {0}, c) f (\tau) \right], f (\tau) = \underset {\mathbf {X} \sim P _ {L (\mathcal {X})}} {\mathbb {E}} [ | \mathbf {X} | ] = \sum_ {i = 1} ^ {T + 1} \frac {\lambda_ {i} ^ {\mathcal {X}}}{\lambda_ {i} ^ {\mathcal {X}} + 1} \tag {7}
$$
where  $\mathcal{X}$  is the set of states in  $\tau$ ,  $L(\mathcal{X})$  is the kernel matrix built with feature vectors of states in  $\mathcal{X}$ ,  $f(\tau)$  is the expected number of modes (landmark states) covered in a trajectory  $\tau$ . Thus,  $\mathcal{L}_1^{DPP}$  measures single-option coverage, by maximizing which we can enhance exploration of each option.